Connector basics

View as Markdown

In the creation of a new Go module, for your application connector, you can utilize the following template by pasting it into a Go file. Not all methods in the file need to be implemented; see below for descriptions of individual methods. Customize the template with your desired package name, type name, and pertinent connector information:

// Package klaviyo provides a connector for Klaviyo.
package klaviyo

import (
    "context"
    "net/http"
    "time"

    "github.com/krenalis/krenalis/connectors"
    "github.com/krenalis/krenalis/tools/types"
)

func init() {
    connectors.RegisterApplication(connectors.ApplicationSpec{
        Code:       "klaviyo",
        Label:      "Klaviyo",
        Categories: connectors.CategorySaaS,
        AsSource: &connectors.AsApplicationSource{
            Targets:       connectors.TargetUser,
            HasSettings:   true,
            Documentation: connectors.RoleDocumentation{
                Summary: "Import profiles as users from Klaviyo",
            },
        },
        AsDestination: &connectors.AsApplicationDestination{
            Targets:       connectors.TargetEvent | connectors.TargetUser,
            HasSettings:   true,
            SendingMode:   connectors.Server,
            Documentation: connectors.RoleDocumentation{
                Summary: "Export users as profiles and send events to Klaviyo",
            },
        },
        Terms: connectors.ApplicationTerms{
            User:   "Profile",
            Users:  "Profiles",
            UserID: "Unique ID",
        },
        EndpointGroups: []connectors.EndpointGroup{
            {
                Patterns:    []string{"/api/event-bulk-create-jobs"},
                RateLimit:   connectors.RateLimit{RequestsPerSecond: 2.5, Burst: 10},
                RetryPolicy: retryPolicy,
            },
            {
                Patterns:    []string{"/api/profiles/"},
                RateLimit:   connectors.RateLimit{RequestsPerSecond: 11.6, Burst: 75},
                RetryPolicy: retryPolicy,
            },
        },
    }, New)
}

var retryPolicy = connectors.RetryPolicy{
    "429":     connectors.RetryAfterStrategy(),
    "500 503": connectors.ExponentialStrategy(connectors.NetFailure, 100*time.Millisecond),
}

type Klaviyo struct {
    // Your connector fields.
}

// New returns a new connector instance for Klaviyo.
func New(env *connectors.ApplicationEnv) (*Klaviyo, error) {
    // ...
}

// EventTypeSchema returns the schema of the specified event type.
func (ky *Klaviyo) EventTypeSchema(ctx context.Context, eventType string) (types.Type, error) {
    // ...
}

// EventTypes returns the event types of the connector's instance.
func (ky *Klaviyo) EventTypes(ctx context.Context) ([]*connectors.EventType, error) {
    // ...
}

// PreviewSendEvents builds and returns the HTTP request that would be used to
// send the given events to the application, without actually sending it.
func (ky *Klaviyo) PreviewSendEvents(ctx context.Context, events connectors.Events) (*http.Request, error) {
    // ...
}

// RecordSchema returns the schema of the specified target and role.
func (ky *Klaviyo) RecordSchema(ctx context.Context, target connectors.Targets, role connectors.Role) (types.Type, error) {
    // ...
}

// Records returns the records of the specified target.
func (ky *Klaviyo) Records(ctx context.Context, target connectors.Targets, updatedAt time.Time, cursor string, schema types.Type) ([]connectors.Record, string, error) {
    // ...
}

// SendEvents sends events to the application. events is a non-empty sequence of
// events to send.
func (ky *Klaviyo) SendEvents(ctx context.Context, events connectors.Events) error {
    // ...
}

// Upsert updates or creates records in the application for the specified
// target.
func (ky *Klaviyo) Upsert(ctx context.Context, target connectors.Targets, records connectors.Records, schema types.Type) error {
    // ...
}

Implementation

Let's explore how to implement an application connector, for example for Klaviyo.

First create a Go module:

mkdir klaviyo
cd klaviyo
go mod init klaviyo

Then add a Go file to the new directory. For example copy the previous template file.

Later on, you can build an executable with your connector.

About the connector

The ApplicationSpec type describes the specification of the application connector:

  • Code: unique identifier in kebab-case (a-z0-9-), e.g. "hubspot", "google-analytics", "salesforce".
  • Label: display label in the Admin console, typically the application's name (e.g. "HubSpot", "Google Analytics", "Salesforce").
  • Categories: the categories the connector belongs to. There must be at least one category, but for an application connector it should be only connectors.CategorySaaS.
  • AsSource: information about the application connector when it used as source. This should be set only when the application connector can be used as a source, otherwise should be nil.
    • Targets: targets supported by the application connector when it is used as source. Can only contain TargetUser.
    • HasSettings: indicates whether the connection has settings when used as a source
    • Description: description of the connector when it is used as a source.
  • AsDestination: information about the application connector when it used as destination. This should be set only when the application connector can be used as a destination, otherwise should be nil.
    • Targets: targets supported by the application connector when it is used as a destination. Can contain TargetEvent and TargetUser.
    • HasSettings: indicates whether the connection has settings when used as destination
    • SendingMode: mode used to send events to the application, if the application supports events. Possible values are Client, Server, or ClientAndServer.
    • Description: description of the connector when it is used as a destination.
    • Terms: singular and plural names the application uses to refer to users (e.g., "Client"/"Clients", "Customer"/"Customers", "User"/"Users"). Also includes the term used for the user identifier (e.g., "ID", "User ID", "Unique ID"). If omitted, the defaults are "User", "Users", and "User ID".
  • OAuth: OAuth 2.0 configuration. To be filled in only if OAuth is required. See OAuth documentation.
  • EndpointGroups: rate limiting and retry policies per endpoint group. See Rate limits and retry documentation.
  • Layouts: layouts for the datetime, date, and time values when they are represented as strings. See Time layouts in Data values for more details.

This information is passed to the RegisterApplication function that, executed during package initialization, registers the application connector:

func init() {
    connectors.RegisterApplication(connectors.ApplicationSpec{
        Code:  "klaviyo",
        label: "Klaviyo",
        Categories: connectors.CategorySaaS,
        AsSource: &connectors.AsApplicationSource{
            Targets:       connectors.TargetUser,
            HasSettings:   true,
            Documentation: connectors.RoleDocumentation{
                Summary: "Import profiles as users from Klaviyo",
            },
        },
        AsDestination: &connectors.AsApplicationDestination{
            Targets:       connectors.TargetEvent | connectors.TargetUser,
            HasSettings:   true,
            SendingMode:   connectors.Server,
            Documentation: connectors.RoleDocumentation{
                Summary: "Export users as profiles and send events to Klaviyo",
            },
        },
        Terms: connectors.ApplicationTerms{
            User:   "Profile",
            Users:  "Profiles",
            UserID: "Unique ID",
        },
        EndpointGroups: []connectors.EndpointGroup{
            {
                Patterns:    []string{"/api/event-bulk-create-jobs"},
                RateLimit:   connectors.RateLimit{RequestsPerSecond: 2.5, Burst: 10},
                RetryPolicy: retryPolicy,
            },
            {
                Patterns:    []string{"/api/profiles/"},
                RateLimit:   connectors.RateLimit{RequestsPerSecond: 11.6, Burst: 75},
                RetryPolicy: retryPolicy,
            },
        },
    }, New)
}

Constructor

The second argument supplied to the RegisterApplication function is the function utilized for creating an application instance:

func New(env *connectors.ApplicationEnv) (*Klaviyo, error)

This function accepts an application environment and yields a value representing your custom type.

The structure of ApplicationEnv is defined as follows:

// ApplicationEnv is the environment for an application connector.
type ApplicationEnv struct {

    // Settings holds the raw settings data.
    Settings json.Value

    // SetSettings is the function used to update the settings.
    SetSettings SetSettingsFunc

    // OAuthAccount is the OAuth account identifier for authentication.
    OAuthAccount string

    // HTTPClient is the HTTP client to use for all requests.
    HTTPClient HTTPClient
}
  • Settings: Contains the instance settings in JSON format. Further details on how the connector defines its settings will be discussed later.
  • SetSettings: A function that enables the connector to update its settings as necessary.
  • OAuthAccount: The application's account associated with the OAuth authorization.
  • HTTPClient: The HTTP client used by the connector to make requests to the application. It seamlessly implements OAuth authorization if required and retries idempotent requests as specified.