Skip to main content

Databricks

Unity Catalog and Hive

This integration covers both Unity Catalog and non-Unity Catalog (Hive) Databricks.

Per Workspace

This integration is to be replicated for each workspace you want to integrate with Catalog.

Requirements​

  • You must be a Databricks administrator and Metastore Admin of the workspace to integrate.
  • OAuth M2M authentication requires a service principal with appropriate permissions.

To create a service principal, follow the instructions for your cloud provider:

1. Generate OAuth Credentials​

Once you have created a service principal:

  1. Click on the service principal.
  2. Navigate to the OAuth secrets tab.
  3. Click Generate secret.
  4. Important: Copy both the Client ID and Client Secret immediately - the secret will only be shown once.

2. Retrieve Your host​

Your host or instance name can be found in your Databricks URL: https://<instance-name>.cloud.databricks.com or https://<instance-name>/.

3. Retrieve Your HTTP path​

Your http_path identifies the compute resource that Coalesce will use to query metadata. The location differs between SQL warehouses and clusters. For more details, see the Databricks compute details documentation.

For Clusters:

  1. Log in to your Databricks workspace.
  2. In the sidebar, click Compute.
  3. Select your cluster.
  4. On the Configuration tab, expand Advanced options.
  5. Click the JDBC/ODBC tab and copy the HTTP path.

For SQL Warehouses:

  1. Log in to your Databricks workspace.
  2. In the sidebar, click SQL Warehouses.
  3. Select your SQL warehouse.
  4. On the Connection details tab, copy the HTTP path.

4. Enable System Tables​

To enable system tables, follow the Databricks system tables documentation.

After enabling system tables, grant your service principal access to the system catalog:

  1. In your Databricks workspace, go to Catalog and navigate to the system catalog.
  2. Click Permissions tab → Grant.
  3. Select your service principal and grant: USE CATALOG, USE SCHEMA, and SELECT.

5. Add Connection to Coalesce Catalog​

Now that you have your OAuth credentials, add the Databricks integration to Coalesce:

  1. Go to your integration page in Coalesce.

  2. Click Add Integration and select Databricks.

    Add Databricks integration

  3. Select Catalog Managed and name your integration.

  4. Fill in the credential fields:

    • Host: Your Databricks workspace hostname (for example, https://dbc-abc12345-6789.cloud.databricks.com)
    • HTTP Path: Your SQL warehouse path (for example, /sql/1.0/warehouses/xxxxx)
    • Client ID: The OAuth client ID from your service principal
    • Client Secret: The OAuth client secret from your service principal
Add Databricks integration

For your first sync, it will take up to 48 hours and we will notify you when it is complete.