Cloud Data Ingestion for Databricks

This page covers everything you need to sync data from Databricks to Braze using Cloud Data Ingestion (CDI): the resources to create in Databricks, how to connect the source and create a sync in Braze, the source table for each data type, and Databricks-specific troubleshooting.

Setup overview

1. In your Databricks account, set up the tables or views you want to sync to Braze.
2. Create credentials for Braze with one of the supported authentication methods, and allow access to the Databricks catalogs and schemas that contain the data you want to sync.
3. Create a new source and sync in the Braze dashboard.
4. Validate the integration and start the sync.

important

There may be two to five minutes of warm-up time when Braze connects to Classic and Pro SQL instances, which can lead to delays during connection setup and testing, as well as at the beginning of scheduled syncs. Using a serverless SQL instance minimizes warmup time and improves query throughput, but may result in slightly higher integration costs.

Step 1: Set up your source table

The steps in this section create a table with a PAYLOAD column. If you plan to map columns with Visual or write a query with SQL on the Data definition step, you can use an existing table or view instead. Skip the table creation step, and grant Braze read access to the tables or views you want to sync. For a comparison of options, see Choose a data definition option.

You can name the schema and table as you’d like, but the column names should match the definitions in the following tabs. You may also use a view or a materialized view instead of a table.

Optionally, set up a new catalog or schema to hold your source table:

1

CREATE SCHEMA BRAZE-CLOUD-PRODUCTION.INGESTION;

note

If you use a PAYLOAD column, your source table or view can include columns that aren’t listed in the following tabs (for example, auditing or hashing). Braze reads only the columns described in these tabs; other columns are not used during Cloud Data Ingestion syncs.

Select the tab for the data type you want to sync.

• user attributes
• custom events
• purchase events
• catalog items
• canvas triggers
• user deletions

User attributes, custom events, and purchase events all use the same source table shape. Only the contents of PAYLOAD differ.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

CREATE TABLE `BRAZE-CLOUD-PRODUCTION.INGESTION.USERS_ATTRIBUTES_SYNC`
(
  updated_at TIMESTAMP DEFAULT current_timestamp(),
  --At least one of external_id, alias_name and alias_label, or braze_id is required
  external_id STRING,
  --If using user alias, both alias_name and alias_label are required
  alias_name STRING,
  alias_label STRING,
  --braze_id can only be used to update existing users created through the Braze SDK
  braze_id STRING,
  --If you include both email and phone, email is used as the primary identifier
  email STRING,
  phone STRING,
  payload STRING, STRUCT, or MAP
);

Field Name	Type	Mode
UPDATED_AT	TIMESTAMP	REQUIRED
PAYLOAD	STRING, STRUCT, or MAP	REQUIRED
EXTERNAL_ID	STRING	NULLABLE
ALIAS_NAME	STRING	NULLABLE
ALIAS_LABEL	STRING	NULLABLE
BRAZE_ID	STRING	NULLABLE
EMAIL	STRING	NULLABLE
PHONE	STRING	NULLABLE

• UPDATED_AT - The time this row was updated in or added to the table. Braze syncs rows where UPDATED_AT is later than the last synced value. Rows at the exact boundary timestamp may be re-synced if new rows share that same timestamp.
• User identifier columns - Your table may contain one or more user identifier columns. Each row should only contain one identifier (either external_id, the combination of alias_name and alias_label, braze_id, email or phone). A source table may have columns for one, two, three, four, or all five identifier types.
  • EXTERNAL_ID - This identifies the user you want to update. This should match the external_id value used in Braze.
  • ALIAS_NAME and ALIAS_LABEL - These two columns create a user alias object. alias_name should be a unique identifier, and alias_label specifies the type of alias. Users may have multiple aliases with different labels but only one alias_name per alias_label.
  • BRAZE_ID - The Braze user identifier. This is generated by the Braze SDK, and new users cannot be created using a Braze ID through Cloud Data Ingestion. To create new users, specify an external user ID or user alias.
  • EMAIL - The user’s email address. If multiple profiles with the same email address exist, the most recently updated profile is prioritized for updates. If you include both email and phone, email is used as the primary identifier.
  • PHONE - The user’s phone number. If multiple profiles with the same phone number exist, the most recently updated profile is prioritized for updates.
• PAYLOAD - This is a string or struct of the fields you want to sync to the user in Braze.

For attribute syncs, PAYLOAD follows the user attributes object format. You can include nested custom attributes, arrays of objects, and subscription group statuses. For payload examples, see Table setup.

Custom events use the same source table as user attributes. Create the table shown in the User attributes tab, then populate PAYLOAD with an events object.

To sync events, an event name is required. Format the time field as an ISO 8601 string or in yyyy-MM-dd'T'HH:mm:ss:SSSZ format. If the time field is not present, Braze uses the UPDATED_AT column value as the event time. Other fields, including app_id and properties, are optional.

You can sync one event per row.

1
2
3
4
5
6
7
8
9

{
    "app_id" : "your-app-id",
    "name" : "rented_movie",
    "time" : "2013-07-16T19:20:45+01:00",
    "properties": {
        "movie": "The Sad Egg",
        "director": "Alex Smith"
    }
}

Purchase events use the same source table as user attributes. Create the table shown in the User attributes tab, then populate PAYLOAD with a purchases object.

To sync purchase events, product_id, currency, and price are required. Format the optional time field as an ISO 8601 string or in yyyy-MM-dd'T'HH:mm:ss:SSSZ format. If the time field is not present, Braze uses the UPDATED_AT column value as the event time. Other fields, including app_id, quantity, and properties, are optional.

You can sync one purchase event per row.

1
2
3
4
5
6
7
8
9
10
11
12

{
    "app_id" : "11ae5b4b-2445-4440-a04f-bf537764c9ad",
    "product_id" : "Completed Order",
    "currency" : "USD",
    "price" : 219.98,
    "time" : "2013-07-16T19:20:30+01:00",
    "properties" : {
        "products" : [ { "name": "Monitor", "category": "Gaming", "product_amount": 19.99 },
        { "name": "Gaming Keyboard", "category": "Gaming ", "product_amount": 199.99 }
        ]
    }
}

Before you create a catalog sync, create the target catalog in Braze. For the full catalog workflow, see Sync and delete catalog data.

1
2
3
4
5
6
7

CREATE TABLE `BRAZE-CLOUD-PRODUCTION.INGESTION.CATALOGS_SYNC`
(
  updated_at TIMESTAMP DEFAULT current_timestamp(),
  id STRING,
  deleted BOOLEAN,
  payload STRING, STRUCT, or MAP
);

Field Name	Type	Mode
UPDATED_AT	TIMESTAMP	REQUIRED
PAYLOAD	STRING, STRUCT, or MAP	REQUIRED
ID	STRING	REQUIRED
DELETED	BOOLEAN	NULLABLE

The name of the integration must exactly match the name of the catalog you created in Braze.

Braze recommends creating a view from your catalog data so the source table fully refreshes each time a sync runs. For example, if you have a table of product data (product_catalog_1) with product_id and three additional attributes, you could sync the following view:

1
2
3
4
5
6
7
8
9
10
11

CREATE view IF NOT EXISTS BRAZE_CLOUD_PRODUCTION.INGESTION.CATALOGS_SYNC AS (SELECT
    last_updated as UPDATED_AT,
    product_id as ID,
    TO_JSON(
      STRUCT(
      attribute_1,
      attribute_2,
      attribute_3,
      )
    ) as PAYLOAD
  FROM `BRAZE_CLOUD_PRODUCTION.INGESTION.product_catalog_1`);

Canvas triggers power zero-copy personalization. For the destination Canvas setup and considerations, see Zero-copy personalization.

1
2
3
4
5
6
7
8
9
10

CREATE TABLE `BRAZE-CLOUD-PRODUCTION.INGESTION.CANVAS_TRIGGERS_SYNC`
(
  updated_at TIMESTAMP DEFAULT current_timestamp(),
  --At least one of external_id or alias_name and alias_label is required
  external_id STRING,
  --If using user alias, both alias_name and alias_label are required
  alias_name STRING,
  alias_label STRING,
  properties STRING, STRUCT, or MAP
);

Field name	Type	Required
UPDATED_AT	TIMESTAMP	Yes
PROPERTIES	STRING, STRUCT, or MAP	Yes
EXTERNAL_ID	STRING	NULLABLE
ALIAS_NAME	STRING	NULLABLE
ALIAS_LABEL	STRING	NULLABLE

Canvas trigger syncs differ from other user data syncs in three ways:

• Only external ID or user alias identifiers are supported. Email and phone numbers are not supported identifiers.
• Only existing Braze users can be synced. New users cannot be created.
• PROPERTIES replaces the PAYLOAD column. This is a JSON string of the fields you want to use as Canvas entry properties for personalization.

Your Canvas triggers table needs the following columns:

• UPDATED_AT - The time this row was updated or added to the table. Braze syncs rows where UPDATED_AT is later than the last synced value. Rows at the exact boundary timestamp may be re-synced if new rows share that same timestamp.
• Either EXTERNAL_ID or ALIAS_NAME and ALIAS_LABEL as the user identifier column. These identify the users for whom you want to trigger Canvas messaging.
  • EXTERNAL_ID - Identifies the user to enter into the Canvas. This should match the external_id value used in Braze.
  • ALIAS_NAME and ALIAS_LABEL - These columns create a user alias object. alias_name should be a unique identifier, and alias_label specifies the alias type. Users may have multiple aliases with different labels, but only one alias_name per alias_label.
• PROPERTIES - A JSON string of fields to make available as personalization properties in your Canvas. This should contain user-specific information.

note

Properties are not required for every row or user. However, properties values must be a valid JSON string. Input an empty {} string if there are no properties for the row.

To get started, select the Canvas Triggers data type when creating a new sync.

Source tables for user deletes should include one or more user identifier types and an UPDATED_AT timestamp. Payload columns aren’t supported for user delete data.

• UPDATED_AT - The time this row was updated or added to the table. Braze syncs rows where UPDATED_AT is later than the last synced value. Rows at the exact boundary timestamp may be re-synced if new rows share that same timestamp.
• User identifier columns - Each row should only contain one identifier: either external_id, the combination of alias_name and alias_label, or braze_id. A source table may contain columns for one, two, or all three identifier types.
  • EXTERNAL_ID - This identifies the user you want to update. This should match the external_id value used in Braze.
  • ALIAS_NAME and ALIAS_LABEL - These two columns create a user alias object. alias_name should be a unique identifier, and alias_label specifies the type of alias. Users may have multiple aliases with different labels but only one alias_name per alias_label.
  • BRAZE_ID - The Braze user identifier. This is generated by the Braze SDK and new users cannot be created using a Braze ID through Cloud Data Ingestion. To create new users, specify an external user ID or user alias.

important

Do not include a PAYLOAD column in your table for user removal. To prevent accidental, permanent removal of users, a sync fails if a payload column is provided in the source table. Any other columns are allowed but are ignored by Braze.

Create a table with the following fields:

Field Name	Type	Mode
UPDATED_AT	TIMESTAMP	REQUIRED
EXTERNAL_ID	STRING	NULLABLE
ALIAS_NAME	STRING	NULLABLE
ALIAS_LABEL	STRING	NULLABLE
BRAZE_ID	STRING	NULLABLE

On the Create import sync page, set the Data Type to Delete Users so that the proper actions are taken during the integration run. For more information, see Delete users with Cloud Data Ingestion.

warning

Deleting user profiles cannot be undone. The delete action permanently removes users, which may cause discrepancies in your data. For details, see Effects of deleting user profiles.

Step 2: Set up authentication

Braze can connect to Databricks with any of the following authentication methods:

OAuth M2M

Braze connects to your Databricks workspace as a service principal with OAuth machine-to-machine (M2M) authentication. Databricks recommends OAuth M2M for unattended connections like Cloud Data Ingestion. Create a new service principal for Braze to use, and grant it access to your SQL warehouse and source tables. Braze needs the following details to connect:

• Client ID (also called application ID) for the service principal
• Client secret for Braze to authenticate

Create the service principal

1. In your Databricks workspace, select your username in the top bar, and then select Settings.
2. Go to Identity and access, and then select Manage next to Service principals.
3. Select Add service principal > Add new.
4. Enter a name that helps you identify the service principal, such as “Braze CDI”, and then select Add.
5. Select the service principal that you created. On the Configurations tab, make sure Databricks SQL access is selected.

Generate a client secret

1. On the service principal’s page, select the Secrets tab, and then select Generate secret.
2. Enter a lifetime for the secret, in days. Databricks allows a maximum lifetime of 730 days.
3. Select Generate.
4. Copy the Secret and Client ID values, and then select Done.

Keep the client ID and secret in a safe place until you need to enter them on the Braze dashboard during the credential creation step. Databricks shows the secret only once.

note

Databricks doesn’t allow unlimited lifetimes on service principal secrets. Remember to replace the secret before it expires to maintain the flow of data to Braze. For steps, see Update the client secret.

Grant access to the SQL warehouse

Braze needs permission to run queries on your SQL warehouse, but doesn’t need workspace admin access.

1. In the sidebar, select SQL Warehouses, and then select the warehouse you want Braze to use.
2. Select Permissions.
3. Search for and select the service principal you created.
4. In the permission dropdown, select Can use, and then select Add.

Grant access to the source tables

Grant the service principal read access to the catalog, schema, and tables that you set up in Step 1. Use the service principal’s application ID (the same value as the client ID) as the grantee.

1
2
3

GRANT USE CATALOG ON CATALOG `BRAZE-CLOUD-PRODUCTION` TO `APPLICATION_ID`;
GRANT USE SCHEMA ON SCHEMA `BRAZE-CLOUD-PRODUCTION`.INGESTION TO `APPLICATION_ID`;
GRANT SELECT ON TABLE `BRAZE-CLOUD-PRODUCTION`.INGESTION.USERS_ATTRIBUTES_SYNC TO `APPLICATION_ID`;

Replace APPLICATION_ID with the application ID of your service principal. Update the other names as needed, but the permissions should match the preceding example.

Update the client secret

Braze stores one client secret per credential. When you update it, every source and sync that uses the credential switches to the new secret. To replace the secret:

1. In Databricks, generate a new secret for the service principal by following the steps in Generate a client secret. Keep the old secret until step 6.
2. In the Braze dashboard, go to Data Settings > Cloud Data Ingestion > Credentials, and then select your OAuth M2M credential. The Used by section lists every source and sync that uses this credential.
3. Pause the syncs that use this credential, or plan the update for a time when none of them are scheduled to run. This prevents a sync from starting while you update the credential.
4. Select Edit credential, enter the new client secret, and then select Save. The change applies to every source and sync that uses this credential.
5. Resume any syncs you paused.
6. After Braze saves the new secret, delete the old secret in Databricks.

Personal access token

Braze authenticates with a personal access token generated in your Databricks workspace. Existing personal access token credentials keep working, and you don’t need to migrate them. However, personal access tokens often expire on a schedule set by your workspace admin, and syncs stop when the token expires. To move an existing source to OAuth M2M, create an OAuth M2M credential and select it when you edit the source.

To create a personal access token:

1. In your Databricks workspace, select your Databricks username in the top bar, and then select User Settings from the dropdown.
2. On the Access tokens tab, select Generate new token.
3. Enter a comment that helps you to identify this token, such as “Braze CDI”, and change the token’s lifetime to no lifetime by leaving the Lifetime (days) box empty.
4. Select Generate.
5. Copy the displayed token, and then select Done.

Keep the token in a safe place until you need to enter it on the Braze dashboard during the credential creation step.

Step 3: Allow access to Braze IPs

If you have network policies in place, you must give Braze network access to your Databricks instance. Allow access from the following IPs corresponding to your Braze dashboard’s region.

• united states (us)
• european union (eu)
• australia (au)
• indonesia (id)
• japan (jp)
• south korea (kr)

Step 4: Create a new source in the Braze dashboard

In the Braze dashboard, go to Data Settings > Cloud Data Ingestion > Sources, select Add data source, and then select Databricks.

Step 4.1: Add Databricks connection information and source table

Enter a name for your source, and then enter your Databricks credentials and configuration:

• For Credential name, enter a label that helps you identify these credentials in Braze.
• For Authentication method, select the method you set up in Step 2, and enter its credentials. For OAuth M2M, enter Client ID and Client secret. For Personal access token, enter Access token.

Then, proceed to the next step.

Step 4.2: Test connection and connect to source

Next, select Test connection. Once successful, finalize remaining settings and click Connect to Source. If the connection fails, an error message appears to help troubleshoot the issue.

note

You must successfully test a source before it can be created. If you close the creation page, your source is not saved.

Step 5: Create a new sync in the Braze dashboard

Go to Data Settings > Cloud Data Ingestion > Syncs, and select Create data sync.

Configure sync details and define your data

Enter a name for your sync. Then, select an active source and a sync data type. Next, select Visual or Table and enter your source table, or select SQL to write a custom SQL query. Finally, select Preview and validate.

After validation succeeds, a preview of the data appears. Select Next: Notifications to continue. If validation fails, an error message appears to help troubleshoot the issue.

note

You must successfully validate a sync before progressing to next steps. If you need to close out of the sync creation page, select Save as draft to keep your work in progress.

Add notification preferences

Input contact email(s) for sync error notifications. Braze uses this contact information to send notifications about integration errors, such as unexpected loss of table access.

Contact emails only receive notifications of global or sync-level errors such as missing tables, permissions, and others. They do not receive row-level issues. Global errors indicate critical problems with the connection that prevent syncs from running.

Such problems can include the following:

• Connectivity issues
• Lack of resources
• Permissions issues
• Catalog tier is out of space (catalog syncs only)

Set the schedule

Lastly, configure your sync as non-recurring or recurring.

Non-recurring syncs run only when you select Sync Now in the dashboard or call the Trigger a sync endpoint.

Recurring syncs can run as often as every 5 minutes or as rarely as once per month. By default, the shortest interval you can set in the dashboard is 15 minutes, which helps manage your warehouse compute costs and request volume. To sync as often as every 5 minutes, contact Braze Support or your customer success manager. Braze schedules recurring syncs in UTC.

note

You must successfully validate an integration before it can move from Draft to Active state. If you close the creation page, your integration is saved, and you can revisit the details page to make changes and validate.

Run the sync

When activated, your sync runs on the schedule configured during setup. If you want to run the sync outside its schedule or to fetch the most recent data, select Sync Now. This run does not impact regularly scheduled future syncs. To start a sync automatically when your data updates, call the Trigger a sync endpoint from your pipeline.

Set up additional integrations or users (optional)

You can set up multiple integrations with Braze. When creating additional syncs, you can reuse existing credentials if connecting to the same Databricks account.

If you reuse the same user across integrations, you cannot delete the user in the Braze dashboard until it’s removed from all active syncs.

Use Databricks as a connected source

A connected source is a zero-copy alternative to syncing data into Braze. Braze queries your Databricks instance directly to build CDI segments, without copying the underlying data. For concepts, limitations, and cost implications, see Connected sources.

important

There may be two to five minutes of warm-up time when Braze connects to Classic and Pro SQL instances, which leads to delays during connection setup and testing, as well as during CDI Segment Extension creation and refresh. Using a serverless SQL instance minimizes warmup time and improves query throughput, but may result in slightly higher integration costs.

Step 1: Set up authentication and grant permissions

Set up credentials for Braze with one of the methods in Step 2: Set up authentication. Whichever method you choose, Braze uses these credentials to generate the list of tables available in your CDI Segment Extensions, and to query source tables to create new segments. After the connected source is created, Braze discovers the names and descriptions of all tables available to the user in the source schema.

You can grant access to all tables in a schema, or grant privileges only to specific tables. Whichever tables the Braze role has access to are available to query in the CDI Segment Extension.

The CREATE TABLE permission is required so Braze can create a table with your CDI Segment Extension query results before updating the segment in Braze. Braze creates a temporary table per segment, which persists only while Braze updates the segment.

• OAuth M2M: Grant the service principal access to the schema used for the connected source, including CREATE TABLE. Use the service principal’s application ID (the same value as the client ID) as the grantee.

  
  1
  2
  3
  
    GRANT USE CATALOG ON CATALOG `BRAZE-CLOUD-PRODUCTION` TO `APPLICATION_ID`;
    GRANT USE SCHEMA, CREATE TABLE ON SCHEMA `BRAZE-CLOUD-PRODUCTION`.INGESTION TO `APPLICATION_ID`;
    GRANT SELECT ON SCHEMA `BRAZE-CLOUD-PRODUCTION`.INGESTION TO `APPLICATION_ID`;
  
  

  Replace APPLICATION_ID with the application ID of your service principal.

• Personal access token: Make sure the service account has CREATE TABLE privileges on the schema used for the connected source.

Network access setup matches Step 3.

Step 2: Create the connected source in Braze

Go to Data Settings > Cloud Data Ingestion > Connected Sources, select Add data source, and then select Databricks.

In Setup source, enter the following:

• Credentials: Credential name, Authentication method, Hostname, HTTP path, and the fields for your authentication method. For OAuth M2M, enter Client ID and Client secret. For Personal access token, enter Access token.

• Configuration: Catalog and Schema

Choose a name for the connected source. This name is used in the list of available sources when you create a new CDI Segment Extension.

Configure a maximum runtime for this source. Braze automatically aborts any queries that exceed the maximum runtime. The maximum runtime allowed is 60 minutes; a lower runtime reduces costs incurred on your Databricks account. This setting applies to queries run through this source, including syncs and CDI Segment Extensions that use it.

Select Test Connection to verify that the list of tables visible to the user is what you expect, then select Done. Your connected source is now created and ready to use in CDI Segment Extensions.

You may set up multiple connected sources with Braze, but each source should be configured to connect a different schema. When creating additional sources, you may reuse existing credentials if connecting to the same Databricks account.

Format a payload from existing columns

If you prefer to store each attribute in its own column internally, you need to convert those columns to a JSON string to populate the sync with Braze. Use this query in Databricks to format source columns into CDI fields.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

CREATE OR REPLACE TABLE BRAZE.EXAMPLE_USER_DATA (
    attribute_1 string,
    attribute_2 STRING,
    attribute_3 NUMERIC,
    my_user_id STRING
);

SELECT
    CURRENT_TIMESTAMP as UPDATED_AT,
    my_user_id as EXTERNAL_ID,
    TO_JSON(
      STRUCT(
        attribute_1,
        attribute_2,
        attribute_3
      )
    ) as PAYLOAD
  FROM BRAZE.EXAMPLE_USER_DATA;

For more optimization guidance, see Cloud Data Ingestion best practices.

Troubleshooting

Test Connection runs slow

Test Connection is running on your data warehouse, so increasing warehouse capacity may improve its speed. For Databricks, there may be two to five minutes of warm-up time when Braze connects to Classic and Pro SQL instances, which leads to delays during connection setup and testing, as well as at the beginning of scheduled syncs. Using a serverless SQL instance minimizes warmup time and improves query throughput, but may result in slightly higher integration costs.

Command failed because warehouse was stopped

If you receive this error, ensure the Databricks warehouse is running.

Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden

If you receive this error, see Databricks: Forbidden error while accessing S3 data.

Where to run SQL checks

Run validation checks directly in the Databricks SQL editor on a SQL warehouse, against the same table or view used by your CDI integration. For more information, see Databricks SQL editor.

Related pages

• Table setup for required columns and payload formatting
• Best practices for sync scheduling and data point efficiency
• Sync logs and observability for monitoring sync runs
• FAQ for common questions across all sources