Import users from applications

This type of pipeline imports users from an application into the workspace's data warehouse. It is available only for source application connections that support users.

Create pipeline

Create a source pipeline to import users from an application.

Request

  • name

    string Required

    The pipeline's name.

    Must be a non-empty string with a maximum of 60 characters.

  • connection

    int Required

    The ID of the connection from which to read the users. It must be a source application that can import users.

  • target

    string Required

    The entity on which the pipeline operates, which must be "User" in order to create a pipeline that imports users.

    Possible values: "User".

  • enabled

    boolean

    Indicates if the pipeline is enabled once created.

  • filter

    nullable object

    The filter applied to the application users. If it's not null, only the application users that match the filter will be included, otherwise all users will be included.

    See the filters documentation for more details.

    • filter.logical

      string Required Possible values: "and" or "or".

    • filter.conditions

      array of object Required

      A filter's condition.

      • property

        string Required

        The name or path of the property. If the property has a json type, it can include a json path.

      • operator

        string Required

        The condition's operator. The allowed values depend on the property's type.

        Possible values: "is", "is not", "is less than", "is less than or equal to", "is greater than", "is greater than or equal to", "is between", "is not between", "contains", "does not contain", "is one of", "is not one of", "starts with", "ends with", "is before", "is on or before", "is after", "is on or after", "is true", "is false", "is empty", "is not empty", "is null", "is not null", "exists" or "does not exist".

      • values

        array of string

        The values the operator applies to, if any. These depend on both the operator and the property's type, including whether they're present and how many there are.

  • incremental

    boolean

    Determines whether users are imported incrementally:

    • true: imports only users who were created or updated since the last import.
    • false: imports all users, regardless of previous imports.

  • transformation

    object Required

    The mapping or function responsible for transforming application users into identities linked to the pipeline. Once the identity resolution process is complete, the identities associated with all pipelines are merged into profiles.

    One of either a mapping or a function must be provided, but not both. The one that is not provided can be either missing or set to null.

    • transformation.mapping

      nullable object with string values Conditionally Required

      The transformation mapping. A key represents a property path in the profile schema, and its corresponding value is an expression. This expression can reference property paths from the source schema of the application.

    • transformation.function

      nullable object Conditionally Required

      The transformation function. A JavaScript or Python function that given an application user, returns an identity.

      • transformation.function.source

        string Required

        The source code of the JavaScript or Python function.

        Must be a non-empty string with a maximum of 50000 characters.

      • transformation.function.language

        string Required

        The language of the function.

        Possible values: "JavaScript" or "Python".

      • transformation.function.preserveJSON

        boolean

        Specifies whether JSON values are passed to and returned from the function as strings, keeping their original format without any encoding or decoding.

      • transformation.function.inPaths

        array of string Required

        The paths of the properties that will be passed to the function. At least one path must be present.

      • transformation.function.outPaths

        array of string Required

        The paths of the properties that may be returned by the function. At least one path must be present.

  • inSchema

    schema Required

    The schema for the properties used in the filter, as well as the input properties for the transformation.

    When importing users from apps, this should be a subset of the application's destination schema.

  • outSchema

    schema Required

    The schema for the output properties of the transformation.

    When importing users from apps, this should be a subset of the profile schema.

Response

  • id

    int

    The ID of the pipeline.
POST /v1/pipelines
curl https://example.com/v1/pipelines \
-H "Authorization: Bearer api_xxxxxxx" \
--json '{
"name": "Import newsletter subscribers",
"connection": 230527183,
"target": "User",
"enabled": true,
"filter": {
"logical": "and",
"conditions": [
{
"property": "country",
"operator": "is",
"values": [
"US"
]
}
]
},
"incremental": true,
"transformation": {
"function": {
"source": "const transform = (user) => { ... }",
"language": "JavaScript",
"preserveJSON": false,
"inPaths": [
"email",
"firstName",
"lastName"
],
"outPaths": [
"email",
"first_name",
"last_name"
]
}
},
"inSchema": {
"kind": "object",
"properties": [
{
"name": "email",
"type": {
"kind": "string"
}
},
{
"name": "firstName",
"type": {
"kind": "string"
},
"nullable": true
},
{
"name": "lastName",
"type": {
"kind": "string"
},
"nullable": true
},
{
"name": "country",
"type": {
"kind": "string",
"maxLength": 2
},
"nullable": true
}
]
},
"outSchema": {
"kind": "object",
"properties": [
{
"name": "first_name",
"type": {
"kind": "string",
"maxLength": 100
},
"readOptional": true,
"description": "First name"
},
{
"name": "last_name",
"type": {
"kind": "string",
"maxLength": 100
},
"readOptional": true,
"description": "Last name"
},
{
"name": "email",
"type": {
"kind": "string",
"maxLength": 254
},
"readOptional": true,
"description": "Email"
}
]
}
}'
Response
{
  "id": 705981339
}
Errors
404
workspace does not exist
422
connection does not exist
422
connector does not exist
422
transformation language is not supported

Update pipeline

Update a source pipeline that imports users from an application.

Request

  • :id

    int Required

    The ID of the source application pipeline to update.

  • name

    string Required

    The pipeline's name.

    Must be a non-empty string with a maximum of 60 characters.

  • enabled

    boolean

    Indicates if the pipeline is enabled. Use the Set status endpoint to change only the pipeline's status.

  • filter

    nullable object

    The filter applied to the application users. If it's not null, only the application users that match the filter will be included, otherwise all users will be included.

    See the filters documentation for more details.

    • filter.logical

      string Required Possible values: "and" or "or".

    • filter.conditions

      array of object Required

      A filter's condition.

      • property

        string Required

        The name or path of the property. If the property has a json type, it can include a json path.

      • operator

        string Required

        The condition's operator. The allowed values depend on the property's type.

        Possible values: "is", "is not", "is less than", "is less than or equal to", "is greater than", "is greater than or equal to", "is between", "is not between", "contains", "does not contain", "is one of", "is not one of", "starts with", "ends with", "is before", "is on or before", "is after", "is on or after", "is true", "is false", "is empty", "is not empty", "is null", "is not null", "exists" or "does not exist".

      • values

        array of string

        The values the operator applies to, if any. These depend on both the operator and the property's type, including whether they're present and how many there are.

  • incremental

    boolean

    Determines whether users are imported incrementally:

    • true: imports only users who were created or updated since the last import.
    • false: imports all users, regardless of previous imports.

  • transformation

    object Required

    The mapping or function responsible for transforming application users into identities linked to the pipeline. Once the identity resolution process is complete, the identities associated with all pipelines are merged into profiles.

    One of either a mapping or a function must be provided, but not both. The one that is not provided can be either missing or set to null.

    • transformation.mapping

      nullable object with string values Conditionally Required

      The transformation mapping. A key represents a property path in the profile schema, and its corresponding value is an expression. This expression can reference property paths from the source schema of the application.

    • transformation.function

      nullable object Conditionally Required

      The transformation function. A JavaScript or Python function that given an application user, returns an identity.

      • transformation.function.source

        string Required

        The source code of the JavaScript or Python function.

        Must be a non-empty string with a maximum of 50000 characters.

      • transformation.function.language

        string Required

        The language of the function.

        Possible values: "JavaScript" or "Python".

      • transformation.function.preserveJSON

        boolean

        Specifies whether JSON values are passed to and returned from the function as strings, keeping their original format without any encoding or decoding.

      • transformation.function.inPaths

        array of string Required

        The paths of the properties that will be passed to the function. At least one path must be present.

      • transformation.function.outPaths

        array of string Required

        The paths of the properties that may be returned by the function. At least one path must be present.

  • inSchema

    schema Required

    The schema for the properties used in the filter, as well as the input properties for the transformation.

    When importing users from apps, this should be a subset of the application's destination schema.

  • outSchema

    schema Required

    The schema for the output properties of the transformation.

    When importing users from apps, this should be a subset of the profile schema.

Response

No response.
PUT /v1/pipelines/:id
curl -X PUT https://example.com/v1/pipelines/705981339 \
-H "Authorization: Bearer api_xxxxxxx" \
--json '{
"name": "Import newsletter subscribers",
"enabled": true,
"filter": {
"logical": "and",
"conditions": [
{
"property": "country",
"operator": "is",
"values": [
"US"
]
}
]
},
"incremental": true,
"transformation": {
"function": {
"source": "const transform = (user) => { ... }",
"language": "JavaScript",
"preserveJSON": false,
"inPaths": [
"email",
"firstName",
"lastName"
],
"outPaths": [
"email",
"first_name",
"last_name"
]
}
},
"inSchema": {
"kind": "object",
"properties": [
{
"name": "email",
"type": {
"kind": "string"
}
},
{
"name": "firstName",
"type": {
"kind": "string"
},
"nullable": true
},
{
"name": "lastName",
"type": {
"kind": "string"
},
"nullable": true
},
{
"name": "country",
"type": {
"kind": "string",
"maxLength": 2
},
"nullable": true
}
]
},
"outSchema": {
"kind": "object",
"properties": [
{
"name": "first_name",
"type": {
"kind": "string",
"maxLength": 100
},
"readOptional": true,
"description": "First name"
},
{
"name": "last_name",
"type": {
"kind": "string",
"maxLength": 100
},
"readOptional": true,
"description": "Last name"
},
{
"name": "email",
"type": {
"kind": "string",
"maxLength": 254
},
"readOptional": true,
"description": "Email"
}
]
}
}'
Errors
404
workspace does not exist
404
pipeline does not exist
422
transformation language is not supported

Get pipeline

Get a source pipeline that imports users from an application application.

Request

  • :id

    int Required

    The ID of the source application pipeline that imports users.

Response

  • id

    int

    The ID of the source application pipeline that imports users.

  • name

    string

    The pipeline's name.

    It is not longer than 60 characters.

  • connector

    string

    The code of the connection's connector.

  • connectorType

    string

    The type of the connection's connector. It is always "Application" when the pipeline imports users from an application.

    Possible values: "Application", "Database", "FileStorage", "MessageBroker", "SDK" or "Webhook".

  • connection

    int

    The ID of the connection from which users are read. It is a source application that supports users.

  • connectionRole

    string

    The role of the pipeline's connection. It is always "Source" when the pipeline imports users from an application.

    Possible values: "Source" or "Destination".

  • target

    string

    The entity on which the pipeline operates. It is always "User" when the pipeline imports users from an application.

    Possible values: "User" or "Event".

  • enabled

    boolean

    Indicates if the pipeline is enabled.

  • filter

    nullable object

    The filter applied to the application users. If it's not null, only the application users that match the filter will be included, otherwise all users will be included.

    See the filters documentation for more details.

    • filter.logical

      string Possible values: "and" or "or".

    • filter.conditions

      array of object

      A filter's condition.

      • property

        string

        The name or path of the property. If the property has a json type, it can include a json path.

      • operator

        string

        The condition's operator. The allowed values depend on the property's type.

        Possible values: "is", "is not", "is less than", "is less than or equal to", "is greater than", "is greater than or equal to", "is between", "is not between", "contains", "does not contain", "is one of", "is not one of", "starts with", "ends with", "is before", "is on or before", "is after", "is on or after", "is true", "is false", "is empty", "is not empty", "is null", "is not null", "exists" or "does not exist".

      • values

        array of string

        The values the operator applies to, if any. These depend on both the operator and the property's type, including whether they're present and how many there are.

  • incremental

    boolean

    Indicates whether users are imported incrementally:

    • true: imports only users who were created or updated since the last import.
    • false: imports all users, regardless of previous imports.

  • transformation

    object

    The mapping or function responsible for transforming application users into user identities linked to the pipeline. Once identity resolution is completed, the user identities associated to all pipelines are merged into unified users.

    One of either a mapping or a function is present, but not both. The one that is not present is null.

    • transformation.mapping

      nullable object with string values

      The transformation mapping. A key represents a property path in the profile schema, and its corresponding value is an expression. This expression can reference property paths from the source schema of the application.

    • transformation.function

      nullable object

      The transformation function. A JavaScript or Python function that given an application user, returns an identity.

      • transformation.function.source

        string

        The source code of the JavaScript or Python function.

        It is not longer than 50000 characters.

      • transformation.function.language

        string

        The language of the function.

        Possible values: "JavaScript" or "Python".

      • transformation.function.preserveJSON

        boolean

        Specifies whether JSON values are passed to and returned from the function as strings, keeping their original format without any encoding or decoding.

      • transformation.function.inPaths

        array of string

        The paths of the properties that will be passed to the function. It contains at least one property path.

      • transformation.function.outPaths

        array of string

        The paths of the properties that may be returned by the function. It contains at least one property path.

  • inSchema

    schema

    The schema for the properties used in the filter, as well as the input properties for the transformation.

  • outSchema

    schema

    The schema for the output properties of the transformation.

  • running

    boolean

    Indicates if the pipeline is running.

  • scheduleStart

    nullable int

    The start time of the schedule in minutes, counting from 00:00. It specifies the minute when the first scheduled run of the day begins. Subsequent runs occur based on the interval defined by the scheduler period. If the scheduler is disabled, this value is null.

  • schedulePeriod

    nullable string

    The schedule period, which determines how often the import runs automatically. If it is null, the scheduler is disabled, and no automatic run will occur.

    To change the schedule period, use the Set schedule period endpoint.

    Possible values: "5m", "15m", "30m", "1h", "2h", "3h", "6h", "8h", "12h" or "24h".
GET /v1/pipelines/:id
curl https://example.com/v1/pipelines/705981339 \
-H "Authorization: Bearer api_xxxxxxx"
Response
{
  "id": 705981339,
  "name": "Import newsletter subscribers",
  "connector": "hubspot",
  "connectorType": "Application",
  "connection": 1371036433,
  "connectionRole": "Source",
  "target": "User",
  "enabled": true,
  "filter": {
    "logical": "and",
    "conditions": [
      {
        "property": "country",
        "operator": "is",
        "values": [
          "US"
        ]
      }
    ]
  },
  "incremental": true,
  "transformation": {
    "mapping": {
      "first_name": "contact.firstname",
      "last_name": "contact.lastname",
      "email": "contact.email"
    }
  },
  "inSchema": {
    "kind": "object",
    "properties": [
      {
        "name": "email",
        "type": {
          "kind": "string"
        }
      },
      {
        "name": "firstName",
        "type": {
          "kind": "string"
        },
        "nullable": true
      },
      {
        "name": "lastName",
        "type": {
          "kind": "string"
        },
        "nullable": true
      },
      {
        "name": "country",
        "type": {
          "kind": "string",
          "maxLength": 2
        },
        "nullable": true
      }
    ]
  },
  "outSchema": {
    "kind": "object",
    "properties": [
      {
        "name": "first_name",
        "type": {
          "kind": "string",
          "maxLength": 100
        },
        "readOptional": true,
        "description": "First name"
      },
      {
        "name": "last_name",
        "type": {
          "kind": "string",
          "maxLength": 100
        },
        "readOptional": true,
        "description": "Last name"
      },
      {
        "name": "email",
        "type": {
          "kind": "string",
          "maxLength": 254
        },
        "readOptional": true,
        "description": "Email"
      }
    ]
  },
  "running": false,
  "scheduleStart": 15,
  "schedulePeriod": "1h"
}
Errors
404
workspace does not exist
404
pipeline does not exist