Data types

View as Markdown

When defining data types and schemas in a connector, use the functions from the types package:

"github.com/krenalis/krenalis/tools/types"

This package provides functions to construct data types to use in connectors. For example, types.Boolean() returns a Type value representing the boolean type, and types.Array(types.Int(32)) returns a Type value representing an array of 32-bit signed integers.

In connectors, data types need to be defined for:

How to construct data types

Below are the data types and how to construct them using the types package functions.

string

Represents UTF-8 encoded string.

types.String()

Strings can use one of the following semantics: country, email, phone, or URL.

Strings can be limited by allowed values, a pattern, or maximum lengths in characters and bytes. These constraints cannot be combined with country or phone semantics.

For example:

// String limited to specific values.
types.String().WithValues("Hearts", "Diamonds", "Clubs", "Spades")

// String matching a pattern.
types.String().WithPattern(regexp.MustCompile(`\d+`))

// String with a maximum length of 2 characters.
types.String().WithMaxLength(2)

// String with a maximum length of 1000 bytes.
types.String().WithMaxBytes(1000)

You can combine maximum character and byte lengths:

// String with a maximum of 20 characters and 25 bytes.
types.String().WithMaxLength(20).WithMaxBytes(25)

These constraints are not freely composable: allowed values, a pattern, and length limits are mutually exclusive. Only WithMaxLength and WithMaxBytes can be combined with each other, and each constraint can be set only once. Any other combination panics.

boolean

Represents a boolean value.

types.Boolean()

int(n)

An integer with n bytes, where n can be 8, 16, 24, 32, or 64.

types.Int(n)

Integers can use one of the following semantics: duration or measurement.

You can also declare the integer as unsigned, meaning it can only hold non-negative values, for example:

// 32-bit unsigned integer.
types.Int(32).Unsigned()

You can limit integers to a minimum and maximum range, for example:

// 8-bit integer with range [-20, 20].
types.Int(8).WithIntRange(-20, 20)

// 32-bit unsigned integer with range [100, 1000].
types.Int(32).Unsigned().WithUnsignedRange(100, 1000)

float(n)

A floating-point number with n bytes, n can be 32, or 64. Includes +Inf, -Inf, and NaN values.

types.Float(n)

A float configured with Real() can use one of the following semantics: duration or measurement.

Floats can have a minimum and maximum value and can be restricted to real numbers only, for example:

// 32-bit float with range [-20.5, 8].
types.Float(32).WithFloatRange(-20.5, 8)

// 64-bit float with range [0, 56.481782].
types.Float(64).WithFloatRange(0, 56.481782)

// 64-bit float, excluding +Inf, -Inf, and NaN.
types.Float(64).Real()

decimal(p,s)

A decimal number with precision p and scale s, where p ranges from 1 to 76, s from 0 to 37, and s is less than or equal to p.

types.Decimal(p, s)

Decimals can use one of the following semantics: duration, measurement, money, or percentage.

Decimals can also have a minimum and maximum value range, for example:

import "github.com/krenalis/krenalis/core/decimal"

...

min := decimal.MustInt(-10.5)
max := decimal.MustInt(8.25)

// Decimal with precision 5 and scale 2, range [-10.5, 8.25].
types.Decimal(5, 2).WithDecimalRange(min, max)

datetime

Represents a date and time within the range [1, 9999] with nanosecond precision and no timezone.

types.DateTime()

date

Represents a date within the range [1, 9999].

types.Date()

time

Represents a time of day with nanosecond precision and no timezone.

types.Time()

year

Represents a year within the range [1, 9999].

types.Year()

uuid

Represents a UUID.

types.UUID()

json

Represents JSON data.

types.JSON()

ip

Represents an IPv4 or IPv6 address.

types.IP()

array(T)

Represents an array of elements of type T.

types.Array(T)

For example:

// Array of strings.
types.Array(types.String())

// Array of 16-bit signed integers.
types.Array(types.Int(16))

// Array of UUID arrays.
types.Array(types.Array(types.UUID()))

Arrays can be limited in the minimum and maximum number of elements:

// Array of 32-bit unsigned integers with at least 1 element.
types.Array(types.Int(32)).WithMinElements(1)

// Array with a maximum of 10 decimal values.
types.Array(types.Decimal(10, 2)).WithMaxElements(10)

// Array with 5 to 15 string values.
types.Array(types.String()).WithMinElements(5).WithMaxElements(15)

Arrays can also be constrained to have unique values for their elements, except for arrays of json, array, map, and object:

// Array of 64-bit unsigned integers with unique values.
types.Array(types.Int(64).Unsigned()).WithUnique()

// Array of UUIDs with unique values.
types.Array(types.UUID()).WithUnique()

object

Represents an object with specified properties.

types.Object([]types.Property{...})

For example:

types.Object([]types.Property{
    {Name: "first_name", Type: types.String().WithMaxLength(30)},
    {Name: "last_name", Type: types.String().WithMaxLength(30)},
    {Name: "birth_date", Type: types.Year()},
})

You can also use the types.ObjectOf function to construct an object. Unlike types.Object, it does not panic if a property is invalid but returns an error instead:

typ, err := types.ObjectOf([]types.Property{...})
if err != nil {
    ...
}

Use types.ObjectOf whenever the properties are not written in the connector but come from the application, for example when building a schema from a list of custom fields. See Types built from application data.

Properties

An object property is defined as follows:

type Property struct {
    Name           string
    Prefilled      string
    Type           Type
    CreateRequired bool
    UpdateRequired bool
    ReadOptional   bool
    Nullable       bool
    DisplayName    string
    Description    string
}
  • Name: The name of the property. It must start with a letter [A-Za-z_] and can only contain alphanumeric characters and underscores [A-Za-z0-9_] after that. To check if a name is valid, use the types.IsValidPropertyName function. When the name comes from the application, see Turn application field names into property names.
  • Prefilled: A prefilled value to use in transformation mappings for events sent to applications. It pre-fills the input with the expression that evaluates to the property's value.
  • Type: The type of the property, which can be any data type.
  • CreateRequired: Indicates whether the property is required for creation.
  • UpdateRequired: Indicates whether the property is required for the update.
  • ReadOptional: Indicates whether the property may not be present when reading.
  • Nullable: Indicates whether the property can be null. In Go, this means it can be nil. In JavaScript, it can be null, and in Python, it can be None.
  • DisplayName: A human-readable name for the property, shown in user interfaces. When the application exposes its own display name for a field, use it instead of deriving one from Name.
  • Description: A description providing additional information about the property's usage. Use it only when it adds something beyond DisplayName; do not repeat the display name in it. When the application exposes its own description for a field, use it instead of writing one yourself.

map(T)

Represents a map with string keys and values of type T.

types.Map(T)

For example:

// Map with 64-bit signed integers as values.
types.Map(types.Int(64))

// Map with arrays of strings as values.
types.Map(types.Array(types.String()))

Semantics

Semantics describe what a type's values represent, such as an email address, a duration, or a monetary amount. Apply them with the corresponding As... method:

Semantics Method Supported by
country AsCountry() string without string constraints
duration AsDuration() int, decimal, or real float
email AsEmail() string
measurement AsMeasurement() int, decimal, or real float
money AsMoney() decimal
percentage AsPercentage() decimal
phone AsPhone() string without string constraints
URL AsURL() string

Each type can have only one semantics. Country and phone semantics cannot be combined with string constraints such as allowed values, a pattern, or maximum lengths. These methods panic if the type, its existing constraints or semantics, or an option is invalid.

Country

Country semantics represent uppercase country codes in ISO 3166-1 alpha-2 (types.ISO3166Alpha2) or alpha-3 (types.ISO3166Alpha3) format. The format is required.

For example, a country code in ISO 3166-1 alpha-2 format:

types.String().AsCountry(types.ISO3166Alpha2)

Examples of values are "US" and "IT" for alpha-2, or "USA" and "ITA" for alpha-3.

Duration

Duration semantics represent numeric durations measured in types.Millisecond, types.Second, types.Minute, types.Hour, types.Day, or types.Week. The unit is required. Durations can use int and decimal types, or float types configured with Real().

For example, a duration expressed in seconds:

types.Decimal(7, 3).AsDuration(types.Second)

You can also use types.Int(32).AsDuration(types.Second) or types.Float(64).Real().AsDuration(types.Second). Examples of values are 5, 3027.306, and -891.

Email

Email semantics identify a string as an email address but do not validate its format.

types.String().AsEmail()

Examples of values are "info@acme.com" and "john@example.com".

Measurement

Measurement semantics represent numeric values expressed in a unit. The unit is required. Measurements can use int and decimal types, or float types configured with Real().

The available units are:

  • types.Gram (g), types.Kilogram (kg)
  • types.Millimeter (mm), types.Centimeter (cm), types.Meter (m), types.Kilometer (km)
  • types.Milliliter (mL), types.Liter (L)
  • types.Byte (B), types.Kilobyte (kB), types.Megabyte (MB), types.Gigabyte (GB)
  • types.Celsius (°C), types.Fahrenheit (°F)
  • types.Ounce (oz), types.Pound (lb)
  • types.Inch (in), types.Foot (ft), types.Yard (yd), types.Mile (mi)

For example, a measurement expressed in kilograms:

types.Float(64).Real().AsMeasurement(types.Kilogram)

Examples of values are 68, -250, and 23.09.

Money

Money semantics represent monetary amounts and can only be used with decimal types. The currency is optional; when set, it must be an ISO 4217 currency code.

For example, a monetary value in US dollars:

types.Decimal(12, 4).AsMoney().WithCurrency("USD")

Examples of values are 22.57, 1600, and -515.3104.

Percentage

Percentage semantics represent percentages in decimal form and can only be used with decimal types. For example, 0.9 represents 90%, and 21 represents 2,100%.

types.Decimal(4, 2).AsPercentage()

Examples of values are 0.25 (25%), -0.50 (-50%), and 13.0 (1,300%).

Phone

Phone semantics describe phone numbers in canonical E.164 format, including their country calling codes.

types.String().AsPhone()

Examples of values are "+12025550123" and "+14155550123".

The types package provides functions to check and normalize phone values before using them in a connector. These functions check whether a number is structurally possible; they do not verify that it is assigned or reachable.

IsPhone

types.IsPhone(s) returns true if s is already in canonical E.164 form. Spaces and formatting characters are not allowed in a canonical value.

types.IsPhone("+12025550123")       // true
types.IsPhone("+1 (202) 555-0123")  // false
types.IsPhone("2025550123")         // false

NormalizePhone

types.NormalizePhone(s) returns the canonical E.164 value and true if normalization succeeds, or an empty string and false otherwise. The input must be a complete phone number in international format, beginning with + after trimming leading and trailing whitespace.

phone, ok := types.NormalizePhone("+1 (202) 555-0123")
// phone == "+12025550123", ok == true

The input must contain a single phone number, without extensions or surrounding text, and must not exceed 250 bytes before trimming whitespace.

NormalizePhoneInRegion

types.NormalizePhoneInRegion(s, region) also accepts national numbers and international dialing prefixes, interpreting them using region as the parsing context. It has the same input limits and return values as NormalizePhone.

The region must be an uppercase two-letter CLDR region code recognized by the phone number parser, such as "IT", "US", or "GB". See Unicode's territory information table for region names and codes.

A recognized region is required even for a phone number in international format. It provides parsing context and does not restrict the number's country.

phone, ok := types.NormalizePhoneInRegion("(202) 555-0123", "US")
// phone == "+12025550123", ok == true

phone, ok = types.NormalizePhoneInRegion("+1 (202) 555-0123", "GB")
// phone == "+12025550123", ok == true

URL

URL semantics identify a string as a URL but do not validate its format.

types.String().AsURL()

Examples of values are "https://blog.example.com/john-smith" and "https://john.example.org/".

Types built from application data

Most functions and modifiers of the types package panic when given an invalid argument. For example, types.Decimal(80, 2) panics because the precision is out of range, and types.String().WithValues() panics because no value is provided.

This is not a problem when the arguments are static, that is, written in the connector or fully determined by its code: a panic means the connector has a bug, and you find it while developing or testing it.

It is a problem when the arguments are dynamic, that is, read from an API response, from the application's field metadata, or from any other external source. A value that the application changes, or that is simply unexpected, would make the connector panic at runtime and crash Krenalis. Some connectors build their schemas this way, for example from the list of custom fields defined by the user in the application.

Before passing a dynamic value to a function that panics, we suggest one of the following:

  • Prefer the variant that returns an error, if one exists, and propagate the error. Currently the only such variant is types.ObjectOf, the counterpart of types.Object.
  • Otherwise, validate the value first. For example, you can check that a length limit is in the range [1, types.MaxStringLen] before calling WithMaxLength, and that a list of allowed values is not empty before calling WithValues.

When validation fails, we recommend returning an error from RecordSchema or EventTypeSchema. Silently skipping the offending property and returning the rest of the schema produces a schema that looks complete, while Krenalis and its users have no way to know that something is missing.

The one case where leaving a property out is the better choice is an application field whose name cannot become a property name; see below.

Turn application field names into property names

A property name must start with an ASCII letter or an underscore [A-Za-z_] and can contain only ASCII letters, digits, and underscores [A-Za-z0-9_] after that; types.IsValidPropertyName checks exactly this. Applications rarely use the same syntax for their own field names, so a connector that builds schemas from application fields has to bridge the two.

You can follow this procedure to decide how. Start from the application's documentation and find out what its field names can contain; if it is not documented, the safest assumption is that a field name can be any sequence of Unicode characters. Then, depending on how the two syntaxes relate:

  • Every application field name is already a valid property name. There is no need to call types.IsValidPropertyName, since it can never fail here. It is more useful to check the name against the application's own syntax: a name that does not match tells you that the connector's assumption about the application is wrong.
  • The two syntaxes overlap only in part, but a name can be converted back and forth. Check the name against the application's syntax as above, and convert it: into a valid property name when building the schema and reading values with Records, and back into the original application name when writing values with Upsert or SendEvents.
  • In any other case. Check the name with types.IsValidPropertyName and, if it does not pass, leave that property out of the schema.