Skip to main content

Databricks Connection Guide

Preview

This feature is currently available in a private preview and may not be accessible to all customers.

This guide will take you through connecting your Databricks account to Coalesce.

Pre-Requisites​

There are different guidelines for Databricks and Coalesce.

Databricks​

  • Your workspace must be using the Unity catalog.
  • Must be a Databricks account admin create to OAuth applications.
  • Databricks users must be given the permission to create a Personal Access Token (PAT) for Token authentication.

Coalesce​

Gather Your Databricks Information​

You'll need your Databricks account URL and path for both OAuth and token authentication.

Get Your Databricks Path​

Your Databricks path is your SQL server HTTP path. You’ll need it to authenticate Databricks in Coalesce.

Serverless

We recommend using a Serverless Compute resource.

  1. In Databricks, go to SQL Warehouses.

  2. Click on the name of the Warehouse you want to connect. You can also create a new SQL Warehouse.

    This screenshot shows the Databricks SQL Warehouses page listing multiple SQL warehouses with details such as name, size, and type. One warehouse named avwcdock_wh is highlighted, indicating it is a serverless configuration.
    Compute

  3. Go to Connection details.

  4. Copy the value in the HTTP path.

    This screenshot shows the Connection Details tab for a Databricks SQL Warehouse named awoodcock_wh. It displays fields such as Server hostname, HTTP path, JDBC URL, and ODBC URL, which are used to configure connections to the warehouse from different clients and tools.
    Connection Details

Get Your Databricks Account URL​

Your account URL is the URL after logging in. You only need the information in the brackets. No backslashes or HTTPS is needed.

https://<abc-11111111-2222.cloud.databricks.com>/settings/workspace/

OAuth User-to-Machine​

User-to-Machine is used for your daily work access. If you want to set up any automation, use Machine-to-Machine. You will need to create a App Connection and get the client ID and client secret along with the URL and path.

  1. Go to the Account console. If you don't have access, you'll need to reach out to your Databricks administrator.

  2. Then go to Settings > App Connections and click Add connection.

    This screenshot shows the App Connections section under Settings, where various external applications (like Tableau or Power BI) are listed along with their creation details and client IDs. An Add connection button allows users to create new integrations with additional applications.
    Databricks - Settings > App connections

  3. Give the app a Name:

    1. Redirect URL is your Coalesce account URL at oauthredirect. This is usually the URL you use to log into Coalesce. For example, https://<COALESCE_APP_DOMAIN>/oauthredirect, https://app.coalesce.io/oauthredirect.

      This screenshot shows the interface for adding a new app connection to Databricks. Users can specify the application name, redirect URLs, access scopes, and token time-to-live values before finalizing the setup with the “Add” button.
      Databricks - Add Connection

    2. Change the Access Scopes to All APIs.

    3. Generate client secret is checked.

    4. You can leave TTL as the default or change it.

  4. Click Add.

  5. A Client ID and Client Secret will pop-up. Save them for later.

User-to-Machine Coalesce Configuration​

You should have your Databricks client ID, client secret, your Databricks path, and Databricks account URL before continuing.

  1. In Coalesce, go to a Project and Workspace that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.

    Mixing Data Platforms

    A Project can only be associated with one data platform.

  2. Open the Build Settings, then Workspace, and click on the cog next to the Workspace name to open Workspace Settings.

  3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or https://.

    Coalesce - This screenshot displays a Workspace Settings panel where a Databricks Account URL can be specified, allowing integration with Databricks services. Other configuration options such as name, platform, and description are also shown, enabling users to tailor their workspace setup.
    Workspace Settings > User Credentials

  4. Click on User Credentials.

  5. Set Authentication Type to OAuth (User-to-Machine).

  6. Add your Databricks path and click Save.

    This screenshot displays the Connection tab of a workspace’s user credentials settings, where an OAuth (User-to-Machine) authentication type is selected. The user can provide a Databricks account URL, click Authenticate, specify a path, and then test the connection to verify proper configuration.
    Workspace Settings > User Credentials

  7. Go to OAuth Settings and toggle Enable OAuth.

  8. Click Edit and enter the Client ID and Client Secret from Databricks.

    This screenshot displays the OAuth Settings section of the Workspace Settings panel, where OAuth can be enabled. The user can configure the Client ID, Client Secret, and then save or cancel the changes.
    Workspace Settings > Add Client ID and Client Secret

  9. Click Save.

  10. Go back to User Credentials and click Authenticate.

  11. You'll be taken to the Build Page and see a testing, then success message.

This screenshot shows two notification messages indicating that OAuth authentication was confirmed and a connection test was performed. The messages confirm that the authentication process succeeded and the connection was established successfully.
Testing and Success Message

OAuth Machine-to-Machine​

Machine-to-Machine is used for automation. You'll need to create a Service Principal.

  1. On the Databricks Account Console, go to User Management and select Service Principals.

    This screenshot shows the user management page in Databricks, focusing on service principals. It lists existing service principals with their application IDs and roles, and provides an option to add new ones.
    Databricks User management

  2. After giving the service principal a name, you'll grant the right permissions, then create the client ID and client secret.

    1. The service principal should have Workspace, Warehouse, and Catalog permissions.

  3. Click on the service principal and go to the Permissions tab and click Grant Access. Add the service principal you just created as a User.

    This screenshot shows the Permissions tab for a service principal, where administrators can grant access to a user, group, or service principal with a selected role. Once roles and recipients are defined, saving these settings applies the new permissions.
    Add the new service principal as user

  4. Go to the Databricks Catalog and select the catalog the service principal should have access to.

  5. Click Grant.

    This screenshot shows the Permissions tab in the Databricks Catalog Explorer, listing principals and their associated privileges. A "Grant" button is available, allowing administrators to assign new privileges to specific principals on selected objects.
    Click Grant on the catalog

  6. Choose the service principal and then select Data Editor.

    This screenshot shows the “Grant” dialog in Databricks, where a principal named “Docs III” is given privileges on the catalog “coalesce_aws_dbx.” A Data Editor privilege preset is selected, revealing a series of checkboxes for read, write, and create actions that can be toggled before finalizing the grant.
    Grant the Data Editor Role

  7. Then go to SQL Warehouses, and select the warehouse the service principal should have access to.

    This screenshot shows the overview page for a Databricks SQL warehouse. The warehouse is currently stopped, configured as serverless on a 2X-Small cluster, and includes buttons in the top right for permissions and editing.
    SQL Warehouse Permissions

  8. Click Permissions, and select the service principal from the list. Set it to Can Use.

    This screenshot shows the Manage permissions dialog for a Databricks SQL warehouse. The user can assign specific permission levels to different principals, such as Can use, before confirming changes with the Add button.
    Permission Level is Can Use
    Service Principal Missing From List

    If your service principal is missing from the list, go to your user Settings > Identity and access > Service principals, and add it there too. Then go back to the SQL Warehouse step.

  9. Go to the Account console in Databricks.

  10. Select User management > Service principals.

  11. Click the one you created, then click Generate secret.

    This screenshot shows a user management page displaying details for a service principal named Docs III. A Generate secret button is available to create new OAuth secrets for secure authentication to the Databricks API.
    Generate Your Client ID and Client Secret

  12. Choose how long the token should be valid for, then click Generate.

  13. Save the Client ID and Client Secret for configuring in Coalesce.

Machine-to-Machine Coalesce Configuration​

You should have your Databricks client ID, client secret, your Databricks path, and Databricks account URL before continuing.

  1. In Coalesce, go to a Project and Workspace that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.

    Mixing Data Platforms

    A Project can only be associated with one data platform.

  2. Open the Build Settings, then Workspace, and click on the cog next to the Workspace name to open Workspace Settings.

  3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or https://.

    Coalesce - This screenshot displays a Workspace Settings panel where a Databricks Account URL can be specified, allowing integration with Databricks services. Other configuration options such as name, platform, and description are also shown, enabling users to tailor their workspace setup.
    Workspace Settings > User Credentials

  4. Click on User Credentials.

  5. Set Authentication Type to OAuth (Machine-to-MACHINE).

  6. Add your Databricks path and click Save.

  7. Click Edit and add enter the Client ID and Client Secret from Databricks.

  8. You must click Save first.

  9. Then click Test Connection.

    This screenshot shows the Workspace Settings panel with the User Credentials section. The user can select an OAuth (Machine to Machine) authentication type, fill in the Databricks Account URL, path, client ID, client secret, then test the connection or save the configuration changes.
    Workspace Settings

  10. You'll be taken to the Build Page and see a testing, then success message.

    This screenshot shows two notification messages indicating that OAuth authentication was confirmed and a connection test was performed. The messages confirm that the authentication process succeeded and the connection was established successfully.
    Testing and Success Message

Token​

OAuth

We recommend using OAuth for authentication.

Tokens or Personal Access Tokens (PAT) are used in Databricks to authenticate at the workspace level.

Get Your Databricks Access Token​

  1. Log into your Databricks account.
  2. Go to your username at the top and select Settings.
  3. Under User, select Developer.
  4. Next to Access Tokens, click Manage.
  5. Click Generate new token.
  6. Write a description for the token and choose the lifetime.
  7. Make sure to save the access token somewhere secure. It can’t be recovered or shown again.
This screenshot shows a Databricks page for managing personal access tokens. It includes a button to generate a new token, allowing users to securely interact with the Databricks API without using passwords.
Databricks Access Token Page
Databricks PAT

Read the Databricks documentation on Databricks personal access token authentication to learn more about managing your access tokens in Databricks.

Add Your Connection Details in Coalesce​

You should have your Databricks access token, your Databricks path, and Databricks account URL before continuing.

  1. In Coalesce, go to a Project and Workspace that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.

    Mixing Data Platforms

    A Project can only be associated with one data platform.

  2. Open the Build Settings, then Workspace, and click on the cog next to the Workspace name to open Workspace Settings.

  3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or https://.

    Coalesce - This screenshot displays a Workspace Settings panel where a Databricks Account URL can be specified, allowing integration with Databricks services. Other configuration options such as name, platform, and description are also shown, enabling users to tailor their workspace setup.
    Workspace Settings

  4. Click on User Credentials.

  5. Set Authentication Type to Token (Cloud).

  6. Token is your Databricks access token.

  7. Path is the Databricks HTTP path.

  8. Click Test Connection.

    This screenshot displays the User Credentials tab of the Workspace Settings panel, where users can enter a Databricks account URL and authentication token. A Path field and a Test Connection button are also provided, allowing users to validate their Databricks integration.
    Workspace Settings > User Credentials
tip

Congratulations, you’ve connected Databricks to Coalesce.

What's Next?​