Databricks

Unity Catalog and Hive

This integration covers both Unity Catalog and non-Unity Catalog (Hive) Databricks.

Per Workspace

This integration is to be replicated for each workspace you want to integrate with Catalog.

AI/BI Dashboards and Apps

To catalog AI/BI dashboards and Databricks Apps, add the separate Databricks BI integration. See Databricks BI.

Requirements

• You must be a Databricks administrator and Metastore Admin of the workspace to integrate.
• OAuth M2M authentication requires a service principal with appropriate permissions.
• For identity-federated workspaces, the service principal must also be granted the account admin role to extract account-level users.

To create a service principal, follow the instructions for your cloud provider:

• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure

1. Generate OAuth Credentials

Once you have created a service principal:

1. Click on the service principal.
2. Navigate to the OAuth secrets tab.
3. Click Generate secret.
4. Important: Copy both the Client ID and Client Secret immediately - the secret will only be shown once.

2. Retrieve Your host

Your host or instance name can be found in your Databricks URL: https://<instance-name>.cloud.databricks.com or https://<instance-name>/.

3. Retrieve Your HTTP path

Your http_path identifies the compute resource that Coalesce will use to query metadata. The location differs between SQL warehouses and clusters. For more details, see the Databricks compute details documentation.

For Clusters:

1. Log in to your Databricks workspace.
2. In the sidebar, click Compute.
3. Select your cluster.
4. On the Configuration tab, expand Advanced options.
5. Click the JDBC/ODBC tab and copy the HTTP path.

For SQL Warehouses:

1. Log in to your Databricks workspace.
2. In the sidebar, click SQL Warehouses.
3. Select your SQL warehouse.
4. On the Connection details tab, copy the HTTP path.

4. Enable System Tables

To enable system tables, follow the Databricks system tables documentation.

After enabling system tables, grant your service principal access to the system catalog:

1. In your Databricks workspace, go to Catalog and navigate to the system catalog.
2. Click Permissions tab → Grant.
3. Select your service principal and grant: USE CATALOG, USE SCHEMA, and SELECT.

5. Retrieve Your Account ID (Identity-Federated Workspaces)

If your workspace uses identity federation (users and groups are managed at the account level in Unity Catalog), you must provide your Account ID so Coalesce can extract users at the account level. If your workspace is not identity-federated, skip this step and leave the Account ID empty.

To find your account ID:

1. Log in to the Databricks account console (for example, https://accounts.cloud.databricks.com).
2. Click your username in the top-right corner.
3. Copy the Account ID shown in the dropdown.

The service principal must also be granted the account admin role to extract account-level users.

6. Add Connection to Coalesce Catalog

Now that you have your OAuth credentials, add the Databricks integration to Coalesce:

1. Go to your integration page in Coalesce.

2. Click Add Integration and select Databricks.

   

3. Select Catalog Managed and name your integration.

4. Fill in the credential fields:

   • Host: Your Databricks workspace hostname (for example, https://dbc-abc12345-6789.cloud.databricks.com)
   • HTTP Path: Your SQL warehouse path (for example, /sql/1.0/warehouses/xxxxx)
   • Client ID: The OAuth client ID from your service principal
   • Client Secret: The OAuth client secret from your service principal
   • Account ID (only for identity-federated workspaces): Check Identity-federated workspace, then enter the Databricks account ID you retrieved in the previous section. Leave the checkbox unchecked for standard workspaces.

Standard workspace: leave the checkbox unchecked.

Identity-federated workspace: check the box and enter your Account ID.

For your first sync, it will take up to 48 hours and we will notify you when it is complete.

Databricks BI

The Databricks BI integration catalogs the AI/BI dashboards and Databricks Apps of a workspace. It is separate from the Databricks warehouse integration: add it for each workspace whose dashboards and apps you want in Catalog.

Warehouse Integration for Lineage

Lineage links dashboards to tables that Catalog already knows. Set up the Databricks warehouse integration first so that the tables your dashboards query are in Catalog.

What Catalog Extracts From Databricks BI

Extraction reads the Databricks REST API:

• AI/BI dashboards, with the workspace folder that holds each one
• Databricks Apps
• Tags assigned to dashboards and apps, which become Catalog tags formatted as key:value, or key when the tag has no value
• Workspace users
• The SQL of each dashboard dataset, which becomes a source query of its dashboard

Databricks objects appear in Catalog under Catalog's own names:

In Databricks	In Catalog
AI/BI dashboard	Dashboard, in its workspace folder
App	Dashboard, in an Apps folder

Catalog then parses the SQL of each dashboard dataset and builds lineage from the tables it reads to the dashboard. Apps have no datasets, so they have no lineage.

Service Principal Entitlements and Permissions

Databricks BI uses a service principal, like the warehouse integration. The Databricks API only returns what the service principal can access, so its entitlements and permissions decide what Catalog extracts.

Give the service principal these entitlements:

• Workspace access: without it, Databricks refuses to list apps and extraction fails.
• Databricks SQL access: Databricks requires it, in addition to Workspace access, to list dashboards.

Then grant the service principal Can View on every folder that holds dashboards you want in Catalog. Granting it on a parent folder covers the dashboards inside.

Dashboards the Service Principal Cannot View

Catalog only extracts the dashboards the service principal can view. A dashboard moved to a folder the service principal cannot view disappears from the next extraction, and Catalog removes it. Check the service principal's permissions before you reorganize folders.

Add the Databricks BI Integration

1. Create a service principal and generate its OAuth secret, as described in Generate OAuth Credentials. You can reuse the service principal of your warehouse integration.

2. Retrieve your host and an HTTP path. The HTTP path is a required field.

3. Go to your integration page in Coalesce.

4. Click Add Integration and select Databricks BI.

5. Select Catalog Managed and name your integration.

6. Enter your credentials in the following format:

   {
       "host": "https://dbc-abc12345-6789.cloud.databricks.com",
       "http_path": "/sql/1.0/warehouses/xxxxx",
       "client_id": "*****",
       "client_secret": "*****"
   }

To authenticate with a personal access token instead of OAuth, replace client_id and client_secret with a token field. Provide one authentication method only: credentials holding both a token and a client_id or client_secret are rejected.

For your first sync, it will take up to 48 hours and we will notify you when it is complete.

Exclude Folders From Ingestion

Your Catalog team can exclude folders from ingestion with folder patterns, glob-style rules matched against each dashboard's folder path, such as Users/jane.doe@example.com/Drafts/**. Contact your Catalog point of contact or Coalesce Support to set them up.

Folder patterns apply after extraction. Catalog still extracts every dashboard the service principal can view, then leaves out the dashboards in matching folders. Folder patterns don't give the service principal access to a folder, and they don't limit what it reads. To keep a folder out of extraction entirely, remove the service principal's permission on it.