Data types

View as Markdown

When defining data types and schemas in a connector, use the functions from the types package:

"github.com/krenalis/krenalis/tools/types"

This package provides functions to construct data types to use in connectors. For example, types.Boolean() returns a Type value representing the boolean type, and types.Array(types.Int(32)) returns a Type value representing an array of 32-bit signed integers.

In connectors, data types need to be defined for:

How to construct data types

Below are the data types and how to construct them using the types package functions.

string

Represents UTF-8 encoded string.

types.String()

String can be limited by allowed values, a pattern, or maximum lengths characters and bytes, for example:

// String limited to specific values.
types.String().WithValues("Hearts", "Diamonds", "Clubs", "Spades")

// String matching a pattern.
types.String().WithPattern(regexp.MustCompile(`\d+`))

// String with a maximum length of 2 characters.
types.String().WithMaxLength(2)

// String with a maximum length of 1000 bytes.
types.String().WithMaxBytes(1000)

You can combine maximum character and byte lengths:

// String with a maximum of 20 characters and 25 bytes.
types.String().WithMaxLength(20).WithMaxBytes(25)

These constraints are not freely composable: allowed values, a pattern, and length limits are mutually exclusive. Only WithMaxLength and WithMaxBytes can be combined with each other, and each constraint can be set only once. Any other combination panics.

boolean

Represents a boolean value.

types.Boolean()

int(n)

An integer with n bytes, where n can be 8, 16, 24, 32, or 64.

types.Int(n)

You can also declare the integer as unsigned, meaning it can only hold non-negative values, for example:

// 32-bit unsigned integer.
types.Int(32).Unsigned()

You can limit integers to a minimum and maximum range, for example:

// 8-bit integer with range [-20, 20].
types.Int(8).WithIntRange(-20, 20)

// 32-bit unsigned integer with range [100, 1000].
types.Int(32).Unsigned().WithUnsignedRange(100, 1000)

float(n)

A floating-point number with n bytes, n can be 32, or 64. Includes +Inf, -Inf, and NaN values.

types.Float(n)

Floats can have a minimum and maximum value and can be restricted to real numbers only, for example:

// 32-bit float with range [-20.5, 8].
types.Float(32).WithFloatRange(-20.5, 8)

// 64-bit float with range [0, 56.481782].
types.Float(64).WithFloatRange(0, 56.481782)

// 64-bit float, excluding +Inf, -Inf, and NaN.
types.Float(64).Real()

decimal(p,s)

A decimal number with precision p and scale s, where p ranges from 1 to 76, s from 0 to 37, and s is less than or equal to p.

types.Decimal(p, s)

Decimals can also have a minimum and maximum value range, for example:

import "github.com/krenalis/krenalis/core/decimal"

...

min := decimal.MustInt(-10.5)
max := decimal.MustInt(8.25)

// Decimal with precision 5 and scale 2, range [-10.5, 8.25].
types.Decimal(5, 2).WithDecimalRange(min, max)

datetime

Represents a date and time within the range [1, 9999] with nanosecond precision and no timezone.

types.DateTime()

date

Represents a date within the range [1, 9999].

types.Date()

time

Represents a time of day with nanosecond precision and no timezone.

types.Time()

year

Represents a year within the range [1, 9999].

types.Year()

uuid

Represents a UUID.

types.UUID()

json

Represents JSON data.

types.JSON()

ip

Represents an IPv4 or IPv6 address.

types.IP()

array(T)

Represents an array of elements of type T.

types.Array(T)

For example:

// Array of strings.
types.Array(types.String())

// Array of 16-bit signed integers.
types.Array(types.Int(16))

// Array of UUID arrays.
types.Array(types.Array(types.UUID()))

Arrays can be limited in the minimum and maximum number of elements:

// Array of 32-bit unsigned integers with at least 1 element.
types.Array(types.Int(32)).WithMinElements(1)

// Array with a maximum of 10 decimal values.
types.Array(types.Decimal(10, 2)).WithMaxElements(10)

// Array with 5 to 15 string values.
types.Array(types.String()).WithMinElements(5).WithMaxElements(15)

Arrays can also be constrained to have unique values for their elements, except for arrays of json, array, map, and object:

// Array of 64-bit unsigned integers with unique values.
types.Array(types.Int(64).Unsigned()).WithUnique()

// Array of UUIDs with unique values.
types.Array(types.UUID()).WithUnique()

object

Represents an object with specified properties.

types.Object([]types.Property{...})

For example:

types.Object([]types.Property{
    {Name: "first_name", Type: types.String().WithMaxLength(30)},
    {Name: "last_name", Type: types.String().WithMaxLength(30)},
    {Name: "birth_date", Type: types.Year()},
})

You can also use the types.ObjectOf function to construct an object. Unlike types.Object, it does not panic if a property is invalid but returns an error instead:

typ, err := types.ObjectOf([]types.Property{...})
if err != nil {
    ...
}

Use types.ObjectOf whenever the properties are not written in the connector but come from the application, for example when building a schema from a list of custom fields. See Types built from application data.

Properties

An object property is defined as follows:

type Property struct {
    Name           string
    Prefilled      string
    Type           Type
    CreateRequired bool
    UpdateRequired bool
    ReadOptional   bool
    Nullable       bool
    DisplayName    string
    Description    string
}
  • Name: The name of the property. It must start with a letter [A-Za-z_] and can only contain alphanumeric characters and underscores [A-Za-z0-9_] after that. To check if a name is valid, use the types.IsValidPropertyName function. When the name comes from the application, see Turn application field names into property names.
  • Prefilled: A prefilled value to use in transformation mappings for events sent to applications. It pre-fills the input with the expression that evaluates to the property's value.
  • Type: The type of the property, which can be any data type.
  • CreateRequired: Indicates whether the property is required for creation.
  • UpdateRequired: Indicates whether the property is required for the update.
  • ReadOptional: Indicates whether the property may not be present when reading.
  • Nullable: Indicates whether the property can be null. In Go, this means it can be nil. In JavaScript, it can be null, and in Python, it can be None.
  • DisplayName: A human-readable name for the property, shown in user interfaces. When the application exposes its own display name for a field, use it instead of deriving one from Name.
  • Description: A description providing additional information about the property's usage. Use it only when it adds something beyond DisplayName; do not repeat the display name in it. When the application exposes its own description for a field, use it instead of writing one yourself.

map(T)

Represents a map with string keys and values of type T.

types.Map(T)

For example:

// Map with 64-bit signed integers as values.
types.Map(types.Int(64))

// Map with arrays of strings as values.
types.Map(types.Array(types.String()))

Types built from application data

Most functions and modifiers of the types package panic when given an invalid argument. For example, types.Decimal(80, 2) panics because the precision is out of range, and types.String().WithValues() panics because no value is provided.

This is not a problem when the arguments are static, that is, written in the connector or fully determined by its code: a panic means the connector has a bug, and you find it while developing or testing it.

It is a problem when the arguments are dynamic, that is, read from an API response, from the application's field metadata, or from any other external source. A value that the application changes, or that is simply unexpected, would make the connector panic at runtime and crash Krenalis. Some connectors build their schemas this way, for example from the list of custom fields defined by the user in the application.

Before passing a dynamic value to a function that panics, we suggest one of the following:

  • Prefer the variant that returns an error, if one exists, and propagate the error. Currently the only such variant is types.ObjectOf, the counterpart of types.Object.
  • Otherwise, validate the value first. For example, you can check that a length limit is in the range [1, types.MaxStringLen] before calling WithMaxLength, and that a list of allowed values is not empty before calling WithValues.

When validation fails, we recommend returning an error from RecordSchema or EventTypeSchema. Silently skipping the offending property and returning the rest of the schema produces a schema that looks complete, while Krenalis and its users have no way to know that something is missing.

The one case where leaving a property out is the better choice is an application field whose name cannot become a property name; see below.

Turn application field names into property names

A property name must start with an ASCII letter or an underscore [A-Za-z_] and can contain only ASCII letters, digits, and underscores [A-Za-z0-9_] after that; types.IsValidPropertyName checks exactly this. Applications rarely use the same syntax for their own field names, so a connector that builds schemas from application fields has to bridge the two.

You can follow this procedure to decide how. Start from the application's documentation and find out what its field names can contain; if it is not documented, the safest assumption is that a field name can be any sequence of Unicode characters. Then, depending on how the two syntaxes relate:

  • Every application field name is already a valid property name. There is no need to call types.IsValidPropertyName, since it can never fail here. It is more useful to check the name against the application's own syntax: a name that does not match tells you that the connector's assumption about the application is wrong.
  • The two syntaxes overlap only in part, but a name can be converted back and forth. Check the name against the application's syntax as above, and convert it: into a valid property name when building the schema and reading values with Records, and back into the original application name when writing values with Upsert or SendEvents.
  • In any other case. Check the name with types.IsValidPropertyName and, if it does not pass, leave that property out of the schema.