# Coalesce Documentation

> Documentation for Coalesce Transform, Catalog, Marketplace, and APIs.

This file contains all documentation content in a single document following the llmstxt.org standard.

## Getting Your Token

Coalesce uses Bearer Authentication to make API requests. Bearer authentication is an HTTP authentication method that uses bearer tokens (Access Token). The token is encrypted, usually by the server. When making a request to Coalesce, it looks like this:

```text
'Authorization: Bearer <Access Token>'
```

## Getting Your Access Token

- If you need Catalog API tokens, see [Getting Your Catalog API Keys][].
- If you need Quality API access, see [Getting Started Quality API][].

Coalesce requires an access token to make API requests. Go to the **Deploy tab** and click **Generate Access Token**.

:::warning[Access Tokens and SSO]

- Access tokens function independently of SSO. Disabling a user in your SSO platform such as Okta, does not remove the access token. You'll need to also remove the user directly in Coalesce to remove any tokens attached to that user.
- Access tokens are regenerated after each login. Previously issued tokens remain valid. Tokens never expire across all environments and projects.

Access tokens are removed when:

- User is deleted in Coalesce.
- User is disabled in Coalesce.
- Password changed in Coalesce.
- Email address change in Coalesce.
- SSO users will need a new token for each new environment created.
- Turning Multi-factor Authentication on and off.

:::

## Managing Credentials

To manage your credentials, there are a few things we recommend:

- Never store API keys in Coalesce Environment.
- Pass API keys dynamically during each API or CLI call.

This allows you to manage your API keys outside of Coalesce on your own schedule.

## Base URLs

Coalesce offers several deployments, and you'll need to make your API calls using the endpoint that corresponds to your Coalesce deployment.

| Geography | Region | Base URL |
| -- | -- | -- |
| Australia | `australia-southeast1` | `https://app.australia-southeast1.gcp.coalescesoftware.io` |
| Canada | `northamerica-northeast1` | `https://app.northamerica-northeast1.gcp.coalescesoftware.io` |
| Europe | `eu-west-2` | `https://app.eu-west-2.aws.coalescesoftware.io` |
| Europe | `europe-west-3` | `https://app.eu.coalescesoftware.io` |
| Europe | `europe-west-1` | `https://app.europe-west1.gcp.coalescesoftware.io` |
| US | `us-east-1` | `https://app.us-east-1.aws.coalescesoftware.io` |
| US | `us-east-2` | `https://app.us-east-2.aws.coalescesoftware.io` |
| US | `us-west-2` | `https://app.us-west-2.aws.coalescesoftware.io/` |
| US | `centralus` | `https://app.centralus.azure.coalescesoftware.io` |
| US | `eastus2` | `https://app.eastus2.azure.coalescesoftware.io` |
| US | `westus2` | `https://app.westus2.azure.coalescesoftware.io` |
| US | `southcentral` | `https://app.southcentralus.azure.coalescesoftware.io` |
| US | `us-central1` | `https://app.coalescesoftware.io` |

:::info[SSO Authentication]

If you use SSO for Authentication, you can prefix the URL to the endpoint with your subdomain. `https://mySubdomain.<app_domain>` For example if your company name is `ABCompany` and you are using Europe primary you can use `https://abcompany.app.eu.coalescesoftware.io`.

:::

## API Request Example

<Tabs>
  <TabItem value="curl" label="curl" default>

```bash
curl --request GET \
  --url https://app.coalescesoftware.io/scheduler/runStatus \
  --header 'Authorization: Bearer <Access Token>' \
  --header 'accept: application/json'
```

  </TabItem>
  <TabItem value="sso-domain" label="SSO Domain">

```bash
curl --request GET \
  --url https://abcompany.app.eu.coalescesoftware.io/scheduler/runStatus \
  --header 'Authorization: Bearer <Access Token>' \
  --header 'accept: application/json'
```

  </TabItem>
</Tabs>

[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Getting Started Quality API]: https://docs.synq.io/api-reference/getting-started

---

## Create Environment

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Create Environment"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/api/v1/environments"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Create a new Environment in a Project. Include project, name, and oauthEnabled in the request body, plus platform connection settings for Snowflake or Databricks. This operation configures connection settings only. Add warehouse passwords or OAuth refresh tokens in the Coalesce App under Build Settings > Environments > User Credentials after creation.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./create-environment.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./create-environment.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./create-environment.StatusCodes.json")}
>
  
</StatusCodes>

---

## Create Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Create Git Account"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/api/v1/gitAccounts"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Create a new Git account.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./create-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./create-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./create-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Create Workspace Node

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Create Workspace Node"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/api/v1/workspaces/{workspaceID}/nodes"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Create a new Workspace Node with the specified configuration.

To update or change the Node data use the [Set Node](/docs/api/coalesce/set-workspace-node) endpoint.

## Get the Node Type

You can get the `nodeType` in two ways:

- Look up the `nodeType` in the Coalesce App. Go to **Build Settings > Node Types**, then click **View**. At the top you'll see the `nodeTypeID`. This is your `nodeType`. If the is this: `nodeTypeID: externalDataPackage:::372`, you'll use: `nodeType: externalDataPackage:::372`.

- You can use the List Environment Nodes or List Workspace Nodes to get a list of existing Nodes. This endpoint only returns in use Nodes and Node types. For example, if you have Node type ML Forecast and ML Anomaly Detection installed, but you are only using ML Forecast, ML Anomaly Node type won't return in the API results.

## Using `predecessorNodeIDs`

You can leave it blank, enter a single Node ID or multiple Node IDs.

**Blank**:

**Single Node**

**Multiple Nodes**
`Region` and `DT_WRK_PART` Node IDs were used.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./create-node.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./create-node.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./create-node.StatusCodes.json")}
>
  
</StatusCodes>

---

## Create Project

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Create Project"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/api/v1/projects"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Create a new Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails>
  
</ParamsDetails>

<RequestSchema
  {...require("./create-project.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./create-project.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete Environment

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete Environment"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v1/environments/{environmentID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Delete an Environment by ID. This operation is destructive. Ensure the Environment has no active runs or dependent Job Schedules before you delete it. The caller needs permission to delete the target Environment.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-environment.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-environment.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-environment.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete Git Account"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v1/gitAccounts/{gitAccountID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Delete an existing Git account.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Remove Project Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Remove Project Git Account"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v1/projects/{projectID}/gitAccounts/{userID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Remove a user's Git account from a Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-project-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-project-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-project-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete Project

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete Project"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v1/projects/{projectID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Delete a Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-project.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-project.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-project.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete Environment User Roles

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete Environment User Roles"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v2/userRoles/{userID}/environments/{environmentID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Deletes all environment roles for a user.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-user-environment-role.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-user-environment-role.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-user-environment-role.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete User's Project Roles

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete User's Project Roles"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v2/userRoles/{userID}/projects/{projectID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Deletes all project roles for a user.

:::info

Available for organization admins only.

:::

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-user-project-role.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-user-project-role.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-user-project-role.StatusCodes.json")}
>
  
</StatusCodes>

---

## Delete Workspace Node

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Delete Workspace Node"}
>
</Heading>

<MethodEndpoint
  method={"delete"}
  path={"/api/v1/workspaces/{workspaceID}/nodes/{nodeID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Delete a Workspace Node.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./delete-workspace-node.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./delete-workspace-node.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./delete-workspace-node.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Environment

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Environment"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/environments/{environmentID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve details for a specific Environment by ID.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-environment.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-environment.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-environment.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Environments

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Environments"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/environments"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve a list of Environments you can access. Use the optional project query parameter to return only Environments in one Project. When project is set, paging totals and next cursors reflect the filtered set.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-environments.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-environments.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-environments.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get a Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get a Git Account"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/gitAccounts/{gitAccountID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Return information for a single Git account.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Environment Node

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Environment Node"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/environments/{environmentID}/nodes/{nodeID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve details for a specific Node by ID.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-node.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-node.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-node.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Environment Nodes

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Environment Nodes"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/environments/{environmentID}/nodes"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve a list of all available Environment Nodes.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-nodes.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-nodes.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-nodes.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Project Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Project Git Account"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/projects/{projectID}/gitAccounts/{userID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Returns a user's Git account in a Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-project-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-project-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-project-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Project

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Project"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/projects/{projectID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Return information for a single Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-project-in-org.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-project-in-org.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-project-in-org.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Projects

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Projects"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/projects"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Returns a list of Projects in an Org.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-projects-in-org.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-projects-in-org.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-projects-in-org.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Run Results

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Run Results"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/runs/{runID}/results"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Get a collection of the results of a deploy or refresh run

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-run-results.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-run-results.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-run-results.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Run

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Run"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/runs/{runID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve details for a specific Run by ID.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-run.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-run.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-run.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Runs

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Runs"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/runs"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve a list of all available Runs.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-runs.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-runs.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-runs.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get a Single User's Role

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get a Single User's Role"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v2/userRoles/{userID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve a specific user's role.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-single-user-roles.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-single-user-roles.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-single-user-roles.StatusCodes.json")}
>
  
</StatusCodes>

---

## Get Workspace Node

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Get Workspace Node"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/workspaces/{workspaceID}/nodes/{nodeID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve details for a specific Workspace Node by ID.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-workspace-node.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-workspace-node.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-workspace-node.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Workspace Nodes

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Workspace Nodes"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/workspaces/{workspaceID}/nodes"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retrieve a list of all available Workspace Nodes.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./get-workspace-nodes.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./get-workspace-nodes.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./get-workspace-nodes.StatusCodes.json")}
>
  
</StatusCodes>

---

## List All User Roles In Organization

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List All User Roles In Organization"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v2/userRoles"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Get all user roles in the organization.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./list-all-user-role-in-org.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./list-all-user-role-in-org.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./list-all-user-role-in-org.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Org Git Accounts

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Org Git Accounts"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/gitAccounts"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Returns a list of all the Git accounts in an organization.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./list-git-accounts.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./list-git-accounts.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./list-git-accounts.StatusCodes.json")}
>
  
</StatusCodes>

---

## List All Organization Users

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List All Organization Users"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/users"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Returns a list of all users in an Org.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./list-org-users.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./list-org-users.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./list-org-users.StatusCodes.json")}
>
  
</StatusCodes>

---

## List Project Git Accounts

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"List Project Git Accounts"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/api/v1/projects/{projectID}/gitAccounts"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Return a list of all Git accounts for a Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./list-project-git-accounts.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./list-project-git-accounts.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./list-project-git-accounts.StatusCodes.json")}
>
  
</StatusCodes>

---

## Set Project Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Set Project Git Account"}
>
</Heading>

<MethodEndpoint
  method={"put"}
  path={"/api/v1/projects/{projectID}/gitAccounts/{userID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Add a users Git account to a Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./set-project-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./set-project-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./set-project-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Set User's Environment Role

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Set User's Environment Role"}
>
</Heading>

<MethodEndpoint
  method={"put"}
  path={"/api/v2/userRoles/{userID}/environments/{environmentID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Creates or updates a user's environment role.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./set-users-environment-role.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./set-users-environment-role.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./set-users-environment-role.StatusCodes.json")}
>
  
</StatusCodes>

---

## Set User's Organization Role

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Set User's Organization Role"}
>
</Heading>

<MethodEndpoint
  method={"put"}
  path={"/api/v2/userRoles/{userID}/organizationRole"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Creates or updates a user's organization role. 

:::info

Available for organization admins only.

:::

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./set-users-org-role.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./set-users-org-role.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./set-users-org-role.StatusCodes.json")}
>
  
</StatusCodes>

---

## Set User's Project Role

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Set User's Project Role"}
>
</Heading>

<MethodEndpoint
  method={"put"}
  path={"/api/v2/userRoles/{userID}/projects/{projectID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Creates or updates a user's project role.

:::info

Available for organization admins only.

:::

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./set-users-project-role.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./set-users-project-role.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./set-users-project-role.StatusCodes.json")}
>
  
</StatusCodes>

---

## Set Node

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Set Node"}
>
</Heading>

<MethodEndpoint
  method={"put"}
  path={"/api/v1/workspaces/{workspaceID}/nodes/{nodeID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Replace all fields of a Workspace Node.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./set-workspace-node.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./set-workspace-node.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./set-workspace-node.StatusCodes.json")}
>
  
</StatusCodes>

---

## Update Environment

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Update Environment"}
>
</Heading>

<MethodEndpoint
  method={"patch"}
  path={"/api/v1/environments/{environmentID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Partially update an Environment by ID. Omitted fields are unchanged; null clears nullable fields. This operation configures connection metadata only. Add warehouse passwords or OAuth refresh tokens in the Coalesce App under Build Settings > Environments > User Credentials.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./update-environment.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./update-environment.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./update-environment.StatusCodes.json")}
>
  
</StatusCodes>

---

## Update a Git Account

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Update a Git Account"}
>
</Heading>

<MethodEndpoint
  method={"patch"}
  path={"/api/v1/gitAccounts/{gitAccountID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Update an existing Git Account.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./update-git-account.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./update-git-account.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./update-git-account.StatusCodes.json")}
>
  
</StatusCodes>

---

## Update a project

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Update a project"}
>
</Heading>

<MethodEndpoint
  method={"patch"}
  path={"/api/v1/projects/{projectID}"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Update a existing Project.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./update-project.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./update-project.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./update-project.StatusCodes.json")}
>
  
</StatusCodes>

---

## hash_details

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"hash_details"}
>
</Heading>

<Schema
  {...require("./hash-details.Schema.json")}
>
  
</Schema>

---

## Hydrated_Metadata

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Hydrated_Metadata"}
>
</Heading>

<Schema
  {...require("./hydrated-metadata.Schema.json")}
>
  
</Schema>

---

## Node_Columns

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Node_Columns"}
>
</Heading>

<Schema
  {...require("./node-columns.Schema.json")}
>
  
</Schema>

---

## Node_Only

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Node_Only"}
>
</Heading>

<Schema
  {...require("./node-only.Schema.json")}
>
  
</Schema>

---

## API Overview

<!--vale Vale.Repetition = NO -->

The Coalesce APIs use REST to allow you to integrate with Coalesce and scale your application. Use the API to run Jobs and get information about your Nodes and Environments.

If you connect an MCP-capable AI assistant instead of writing REST calls directly, see [About Transform MCP][].

## Postman Collection

If you're already familiar with the Coalesce API, grab our Postman Collection.

<Postman/>

[About Transform MCP]: /docs/coalesce-ai/mcp/about-mcp

---

## Making Your First API Request

You'll make a request to the [List Environments][] endpoint.

## Tools Available

There are multiple tools to make API requests, a few are listed below.

- You can use the built in [API Explorer][].
- [Postman][]
- [Insomnia][]
- Command Line

## Get a List of Environments

You'll use the command line for this example. If you are using Linux or Mac, you can continue.

Windows users, you might need to download [cURL][].

1. Get your [Access Token][].
2. Review the request URL for List Environments. `https://app.coalescesoftware.io/api/v1/environments?detail=false`.
    1. Server URL- `https://app.coalescesoftware.io/api/v1`
    2. Base URL- `app.coalescesoftware.io`. You'll need to update this before making a request to your app address. If you don't know yours, reach out to our support team for help. It's usually the URL you use to log in.
    3. Endpoint Path- The resource on the server you want to access with your request.
    4. Query Parameters - These help determine what data is returned. The are usually optional and can help filter large amounts of data. In our example, details is false, which means it will return abridged information about any environments.

    ```text
    https://app.coalescesoftware.io/api/v1/environments?detail=false
    \_____________________________________ /\__________/ \__________/
                server URL                   endpoint    query parameters
            \______________________/           path
                base URL         
    ```

3. Build your API request. Using cURL, the request type is `GET`, since we are only reading data.
    1. The header is `application/json` which tells the server to return the data as JSON. Coalesce returns and accepts data in JSON format.
    2. Another header is set for the Authorization as bearer type.
    3. You can copy this code and run it in your command line. Remember to replace your access token.

    ```bash
    curl --request GET \
        --url 'https://app.coalescesoftware.io/api/v1/environments?detail=false' \
        --header 'accept: application/json' \
        --header 'authorization: Bearer replace-with-your-token'
    ```

4. Review your response. The information is returned in a JSON format to make it easy to use in other applications.

```json
{
  "data": [
    {
      "createdAt": "2024-03-19T16:36:11.409Z",
      "createdBy": {
        "id": "Owb7B45m0UNSL4OumkDNpcFifTA3",
        "firstName": "Ada",
        "lastName": "Lovelace"
      },
      "id": "12",
      "name": "QA",
      "status": "Waiting",
      "project": "8398bbb9-5e0d-4b17-9f79-b65e3183b187"
    },
    {
      "createdAt": "2024-04-12T19:18:47.876Z",
      "createdBy": {
        "id": "Owb7B45m0UNSL4OumkDNpcFifTA3",
        "firstName": "Ada",
        "lastName": "Lovelace"
      },
      "id": "17",
      "name": "BUILD",
      "status": "Waiting",
      "project": "007e867f-806a-48f0-b766-f13789f30b25"
    },
    {
      "createdAt": "2024-02-05T21:11:27.872Z",
      "createdBy": {
        "id": "Owb7B45m0UNSL4OumkDNpcFifTA3",
        "firstName": "Ada",
        "lastName": "Lovelace"
      },
      "id": "2",
      "name": "Production",
      "status": "Failed Deploy",
      "project": "4502de52-048f-4ab3-bf17-5c58b2b5403c"
    },
    {
      "createdAt": "2024-02-26T19:03:05.020Z",
      "createdBy": {
        "id": "Owb7B45m0UNSL4OumkDNpcFifTA3",
        "firstName": "Ada",
        "lastName": "Lovelace"
      },
      "id": "6",
      "name": "QA",
      "status": "Waiting",
      "project": "4502de52-048f-4ab3-bf17-5c58b2b5403c"
    },
    {
      "createdAt": "2024-02-27T19:06:38.317Z",
      "createdBy": {
        "id": "Owb7B45m0UNSL4OumkDNpcFifTA3",
        "firstName": "Ada",
        "lastName": "Lovelace"
      },
      "id": "8",
      "name": "New Environment",
      "status": "Waiting",
      "project": "cfedac5c-1262-4ad9-9a00-313ec1156e69"
    }
  ]
}
```

You've made your first request. Try changing `details=true` and compare the responses. Try making requests with other endpoints.

[List Environments]: /docs/api/coalesce/get-environments
[API Explorer]: /docs/api/using-the-api-explorer
[Postman]: https://www.postman.com/
[Insomnia]: https://insomnia.rest/
[cURL]: https://community.chocolatey.org/packages/curl
[Access Token]: /docs/api/authentication

---

## Query Parameters

<!-- vale Coalesce.FirstPerson = NO-->
<!-- markdownlint-disable MD036 -->

Our API provides a flexible way to retrieve data through query parameters. This guide explains the available query parameters and how to use them effectively.

## Endpoints

Query parameters are available on the following endpoints:

- GET `/api/v1/environments`
- GET `/api/v1/environments/{environmentID}/nodes`
- GET `/api/v1/workspaces/{workspaceID}/nodes`
- GET `/api/v1/runs`
- GET `/api/v1/runs/{runID}`

:::info[Varies By Endpoint]

Review each endpoint for available parameters.

:::

## What Is Cursor-Based Pagination?

Cursor-based pagination is like using a bookmark in a large dataset. Instead of asking for a specific page number, you are asking to "show me results that come after this point."

Our endpoints offer this using `startingFrom`.

:::info[Title example]

When using `startingFrom`, you must also include the `orderBy` parameter to define the sort order used for the cursor.

:::

### Example of Cursor-Based Pagination

Imagine you have a collection of 500 Nodes in the database, and you want to retrieve 10 Nodes at a time:

#### Get First 10 Nodes Request

```bash
GET /api/v1/environments/{environmentID}/nodes?limit=10&orderBy=id
````

#### Get First 10 Nodes Response

You get 10 Nodes and a cursor value in the `next` field, which points to where you left off.

```json
{
  "data": [
    {
      "database": "sales",
      "id": "27",
      "locationName": "US-East",
      "name": "Region",
      "nodeType": "Dimension",
      "schema": "public"
    }
  ],
  "total": 500,
  "limit": 10,
  "next": "eyJpZCI6IjMzIn0",
  "orderBy": "id"
}
```

#### Get Next 10 Nodes Request

Use the `next` value in the `startingFrom` parameter and include the same `orderBy`.

```bash
GET /api/v1/environments/{environmentID}/nodes?limit=10&startingFrom=eyJpZCI6IjMzIn0&orderBy=id
```

#### Get Next 10 Nodes Response

```json
{
  "data": [
    {
      "database": "sales",
      "id": "39",
      "locationName": "US-West",
      "name": "Territory",
      "nodeType": "Dimension",
      "schema": "public"
    }
  ],
  "total": 500,
  "startingFrom": "eyJpZCI6IjMzIn0",
  "limit": 10,
  "next": "eyJpZCI6IjE3OCJ9",
  "orderBy": "id"
}
```

## Using the `next` Field

When paginating, the `next` value in the response allows you to fetch the next set of results. To do this:

1. Copy the `next` value from the response.
2. Pass it as the `startingFrom` value in your next request.
3. Always include the same `orderBy` value used in the previous request.

## Query Parameter Example

Here are a few examples of using query parameters.

### Using `limit`, `startingFrom`, and `orderBy`

These examples use the Workspace endpoint to demonstrate paginated requests.

**Get the first 25 Nodes**

```bash
GET /api/v1/workspaces/{workspaceID}/nodes?limit=25&orderBy=id
```

**Response with next**

```json
{
  "data": [ /* 25 Nodes */ ],
  "total": 142,
  "limit": 25,
  "next": "eyJpZCI6IjEyOCJ9",
  "orderBy": "id"
}
```

**Get the next 25 Nodes**

```bash
GET /api/v1/workspaces/{workspaceID}/nodes?limit=25&startingFrom=eyJpZCI6IjEyOCJ9&orderBy=id
```

### Using `orderBy` with Sort Direction

This returns runs sorted by `runStartTime` from most recent to oldest.

```bash
GET /api/v1/runs?orderBy=runStartTime&orderByDirection=desc
```

### When `next` Is Null

If `"next": null`, you've reached the end of the dataset.

```json
{
  "data": [],
  "total": 0,
  "startingFrom": "eyJpZCI6IjEyOCJ9",
  "limit": 2,
  "next": null,
  "orderBy": "id"
}
```

---

## Stop Active Run

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Stop Active Run"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/scheduler/cancelRun"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Cancel a currently active run.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails>
  
</ParamsDetails>

<RequestSchema
  {...require("./cancelrun.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./cancelrun.StatusCodes.json")}
>
  
</StatusCodes>

---

##  Retry Failed Run

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={" Retry Failed Run"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/scheduler/rerun"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Retry a run the point of failure. This endpoint will start the run again and run only the nodes that failed.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails>
  
</ParamsDetails>

<RequestSchema
  {...require("./retry-failed-run.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./retry-failed-run.StatusCodes.json")}
>
  
</StatusCodes>

---

## Check Run Status

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Check Run Status"}
>
</Heading>

<MethodEndpoint
  method={"get"}
  path={"/scheduler/runStatus"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Get the status of a specific Run.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails
  {...require("./run-status.ParamsDetails.json")}
>
  
</ParamsDetails>

<RequestSchema
  {...require("./run-status.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./run-status.StatusCodes.json")}
>
  
</StatusCodes>

---

## Start a Run

<Heading
  as={"h1"}
  className={"openapi__heading"}
  children={"Start a Run"}
>
</Heading>

<MethodEndpoint
  method={"post"}
  path={"/scheduler/startRun"}
  context={"endpoint"}
>
  
</MethodEndpoint>

Create a new Run.

<Heading
  id={"request"}
  as={"h2"}
  className={"openapi-tabs__heading"}
>
  <Translate id="theme.openapi.request.title">Request</Translate>
</Heading>

<ParamsDetails>
  
</ParamsDetails>

<RequestSchema
  {...require("./startrun.RequestSchema.json")}
>
  
</RequestSchema>

<StatusCodes
  {...require("./startrun.StatusCodes.json")}
>
  
</StatusCodes>

---

## Using the API Explorer

The API explorer is divided into three parts:

- API menu
- API details
- API request

## API Menu

Use the menu to navigate between the different endpoints. Each point will list the request type next to the name.

## API Details

The API details contains the API name, endpoint, and any parameters required to make the request, and responses. An example of the response and schema can be found by toggling between Schema and the Response name.

## API Request

The API request contains code samples in multiple languages. Select your language from the options above the code sample and copy to start using it in your code. It also contains the Request and Response section.

To make a request:

1. Edit the Base URL to match yours.
2. Enter your [Access Token][] in Bearer Token.
3. Add in any optional parameters and edit the request body.
4. Click Send API Request.
5. Then review your response in the Response area.

:::warning[Jobs API]

Making a request is not available for the Jobs API.

:::

[Access Token]: /docs/api/authentication

---

## Architecture

<!-- vale off -->

## Application Architecture

The Coalesce application consists of a GUI frontend and the accompanying back-end cloud.

### Frontend

The frontend component of the Coalesce Cloud serves static assets such as JavaScript, CSS and other files needed for customers to use the GUI web client for editing Coalesce projects.

### Template Renderer

The template renderer is an API endpoint used to process Jinja templates for the frontend. Because Jinja is a python library, an API endpoint is used to render Jinja templates when requested by the JavaScript frontend.

### Backend

The backend served as a direct proxy to the customer's version control and data provider instances. Because of browser CORS policies, it is not possible to directly access data provider instances from a browser, and thus, a direct proxy is used.

Upon successful authentication of a request by the backend, the industry standard credential manager is used to retrieve git personal access tokens (PAT) or per-user data provider credentials (unless browser storage is specified by the user), and a subsequent request to the resource is made.

### Industry Standard Credential Manager

The industry standard credential manager is used to store OAuth tokens for the data platform OAuth security integration, per-user git personal access tokens (PAT) and as well as per-user data platform credentials (unless browser storage is specified by the user).

When using the data platform OAuth integration, the data platform passwords are never stored in the industry standard credential manager. Coalesce users authenticate directly with the data platform. Coalesce only stores refresh tokens that expire in 90 days by default and access tokens that rotate every 10 minutes.

Data is encrypted in transit with TLS and at rest with AES-256-bit encryption keys.

## Job Engine

When a user submits a job via the API, the job engine in the Coalesce cloud is used to run jobs on their behalf. Each job is run to completion in an isolated environment, similarly to using `coa`. Refer to CLI Architecture (below) for more details regarding the contents and architecture of each job execution.

## Metadata Repository

Coalesce stores the metadata format for each data warehouse environment in a highly scalable NoSQL database. This metadata never stores your data platform's data or passwords, it is solely used for storing the metadata configuration of your data warehouse and storing results of jobs.

### Metadata Repository Security Rules

Upon user authentication, security rules are enforced such that a user may only read, write and update the metadata for organizations of which they are a member. Requests without a valid user, or for a user not part of a organization, are rejected.

## API Architecture

The REST API exposed by Coalesce can be used by customers to access their metadata programmatically or submit jobs. API calls are authenticated via tokens that prevent unauthorized access.

If a user chooses to submit jobs to the Coalesce API, the jobs are run in an isolated environment on the Jobs Engine, similarly to the CLI architecture.

## CLI Architecture

The `coa` CLI application is used for executing Coalesce jobs and deployments via the command-line. `coa` bundles the template renderer and database execution engine into a downloadable npm package. The tool can be integrated with CI/CD tools and leveraged for high-security environments requiring database traffic to originate from in-network environments. An outgoing internet connection is required for authentication to the Coalesce platform and access to the data platform.

---

## Coalesce Subprocessors

<!-- vale off -->

Coalesce Automation Inc. (Coalesce) uses certain subprocessors to assist it in providing its services as described in the [Terms of Service (TOS)][]. Defined terms used herein shall have the same meaning as defined in the TOS.

## What Is a Subprocessor?

A subprocessor is a third-party data processor engaged by Coalesce who has or potentially will have access to or process personal data on behalf of a Customer. Coalesce engages different types of subprocessors to perform various functions as explained in the table below.

| Subprocessor                | Nature of Processing            | Territory(ies)       | Privacy Policy                                                       |
| :-------------------------- | :------------------------------ | :------------------- | :------------------------------------------------------------------- |
| Atlassian                   | Customer Support                | Global               | [https://www.atlassian.com/legal/privacy-policy][]                     |
| Amazon Web Services (AWS)   | Hosting                         | Global               | [https://aws.amazon.com/privacy/][]                                    |
| Estuary                     | Data Integration                | Global               | [https://estuary.dev/privacy-policy/][]                                |
| Fivetran                    | Data Integration                | Global               | [https://fivetran.com/legal][]                                         |
| Gong                        | Customer Support                | Global (except EMEA) | [https://www.gong.io/privacy-policy][]                                 |
| Google                      | Email, Documentation, Analytics | Global               | [https://policies.google.com/privacy][]                                |
| Google Cloud Platform (GCP) | Hosting and Data Storage        | Global               | [https://cloud.google.com/privacy][]                                   |
| Hightouch                   | Data Integration                | Global               | [https://hightouch.com/privacy-policy][]                               |
| Launch Darkly               | Product Experience Platform     | Global               | [https://launchdarkly.com/policies/privacy/][]                         |
| Microsoft Azure             | Hosting and Data Storage        | Global               | [https://www.microsoft.com/en-us/privacy/privacystatement][]           |
| Pagerduty                   | Customer Support                | Global               | [https://www.pagerduty.com/privacy-policy/][]                          |
| Pendo                       | Product Experience Platform     | Global               | [https://www.pendo.io/adopt/privacy-center/][]                         |
| Pylon                       | Customer Support                | Global               | [https://usepylon.com/privacy][]                                       |
| Rattle                      | Messaging Integration           | Global               | [https://www.gorattle.com/privacy-policy][]                            |
| Salesforce                  | CRM                             | Global               | [https://www.salesforce.com/company/privacy/][]                        |
| Sigma                       | Analytics                       | Global               | [https://www.sigmacomputing.com/privacy-policy][]                      |
| Slack                       | Messaging Integration           | Global               | [https://slack.com/trust/privacy/privacy-policy][]                     |
| Snowflake                   | Hosting and Data Storage        | Global               | [https://www.snowflake.com/privacy-policy/][]                          |
| Tableau                     | Analytics                       | Global               | [https://www.salesforce.com/company/privacy/][]                        |

[Terms of Service (TOS)]: https://coalesce.io/terms
[https://www.atlassian.com/legal/privacy-policy]: https://www.atlassian.com/legal/privacy-policy
[https://aws.amazon.com/privacy/]: https://aws.amazon.com/privacy/
[https://estuary.dev/privacy-policy/]: https://estuary.dev/privacy-policy/
[https://fivetran.com/legal]: https://fivetran.com/legal
[https://www.gong.io/privacy-policy]: https://www.gong.io/privacy-policy
[https://policies.google.com/privacy]: https://policies.google.com/privacy
[https://cloud.google.com/privacy]: https://cloud.google.com/privacy
[https://hightouch.com/privacy-policy]: https://hightouch.com/privacy-policy
[https://www.pagerduty.com/privacy-policy/]: https://www.pagerduty.com/privacy-policy/
[https://www.pendo.io/adopt/privacy-center/]: https://www.pendo.io/adopt/privacy-center/
[https://www.gorattle.com/privacy-policy]: https://www.gorattle.com/privacy-policy
[https://www.salesforce.com/company/privacy/]: https://www.salesforce.com/company/privacy/
[https://www.sigmacomputing.com/privacy-policy]: https://www.sigmacomputing.com/privacy-policy
[https://slack.com/trust/privacy/privacy-policy]: https://slack.com/trust/privacy/privacy-policy
[https://www.snowflake.com/privacy-policy/]: https://www.snowflake.com/privacy-policy/
[https://www.microsoft.com/en-us/privacy/privacystatement]:https://www.microsoft.com/en-us/privacy/privacystatement
[https://usepylon.com/privacy]: https://usepylon.com/privacy
[https://launchdarkly.com/policies/privacy/]: https://launchdarkly.com/policies/privacy/

---

## Security

<!-- vale off -->

The Coalesce platform has been engineered to satisfy the security and governance requirements of enterprise customers.

These topics are intended primarily for the Coalesce administrators, IT teams and other Coalesce users interested in the architecture of the Coalesce platform.

## Authentication Methods

Coalesce supports different methods of authentication that satisfy varying security and governance requirements. The authentication method used for an organization using Coalesce will vary depending on ease of use, separation of design and production environments as well as customer-specific data and security compliance requirements.

### Snowflake OAuth

Coalesce supports the use of a Snowflake OAuth integration. This is the preferred method of authentication because it uses tokens instead of username / password to authenticate users to Snowflake.

When using Snowflake OAuth, Coalesce uses a periodically rotating access token to access Snowflake on behalf of users when using the UI.

### Snowflake Basic (Cloud Storage)

Coalesce's basic authentication method uses a per-user username / password combination to authenticate to Snowflake. This method stores the username / password pair encrypted in an industry standard credential manager. After initial configuration, the password is never sent to the browser, and can not be retrieved by any APIs.

### Snowflake KeyPair

The `coa` CLI supports using a key-pair to authenticate to Coalesce. One can read more about this system in Snowflake's documentation [here][].

### Snowflake Basic (Local Storage)

For Coalesce users looking for an easy to use option that authenticates users to Snowflake without storing any credentials in Coalesce, the local storage option can be used to store Snowflake username / password in the browser.

## Data Flows

At design time, the Coalesce UI is an in-browser editor that allows Coalesce users to design the directed acyclic graphs for their data transformation pipelines.

In order to support CORS proxying behavior and access for users interactively to their Snowflake instances and git provider, Coalesce uses industry standard direct proxying over HTTPS/TLS (minimum TLS 1.2, up to TLS 1.3 supported) encryption for data flows involving Snowflake and git.

### Browser Git Integration

To support git interactions within a browser environment, a direct proxy is used by Coalesce that forwards git operations on behalf of customer actions submitted through the UI to the customer’s git provider. All data is encrypted via HTTPS/TLS and is never stored at rest.

When a user submits a git request from the browser, an API call is sent from the browser to the Coalesce backend (1), where it is authenticated. Upon successful authentication (2), the backend retrieves the git personal access token (PAT) for the user from the industry standard credential manager (3) in preparation for the git provider request. The backend then communicates directly over HTTPS/TLS with the git provider (4) (GitHub, Bitbucket, Azure DevOps, GitLab) proxying requests (for CORS purposes) over HTTPS/TLS back to the browser (5). The communication in part 5 uses native git http protocol over HTTPS/TLS (this is the same protocol used when performing `git clone` with a https git repository URL).

All communication uses HTTPS/TLS encryption.

### Browser Snowflake SQL Execution

In order to support Snowflake SQL execution from within a browser environment, a direct proxy is used by Coalesce that forwards SQL text on behalf of customer actions submitted through the Coalesce UI to the customer’s Snowflake instance. All data is encrypted via HTTPS/TLS and data results from SQL executions are never stored by Coalesce.

When submitting a Snowflake SQL execution, the SQL text is sent by the browser to the Coalesce backend (1), the user is successfully authenticated (2), the credentials are then retrieved from the industry standard credential manager (3) and then used to establish a connection to Snowflake (4). Results are then sent back to the browser (5) over HTTPS/TLS.

When a user requests to fetch data, the same Snowflake SQL execution path is executed, and the first 100 rows of data are returned to the browser to be displayed. The data is never stored by Coalesce.

All communication uses HTTPS/TLS encryption.

## CI/CD time (Deployments and Refresh)

### Browser Deployments

When using the in-browser deployment workflow, the application uses the [browser git integration][] and the [browser Snowflake SQL execution][] to prepare a deployment plan.

Upon creating the plan, the deployment request is sent to the backend and the deployment is executed in the Coalesce cloud. This deployment API request establishes a connection to Snowflake and a connection to Coalesce metadata.

The connection to Snowflake is used to send the SQL commands to be executed by Snowflake. The Coalesce metadata connection is used to store run results (and other execution metadata) in Coalesce's metadata store.

### Command Line Interface (CLI)

The `coa` CLI tool can be used on-premise for teams looking to run data transformations on their own infrastructure.

The `coa` CLI supports basic and Snowflake key-pair authentication. The credentials are stored locally on the machine running `coa`.

The `coa` client sends Snowflake commands directly to the customer’s Snowflake instance. Because there are no CORS policies when executing a command-line, a direct connection to Snowflake can be established.

The metadata connection to Coalesce is solely used for storing run results such as execution time, success/failure and other operational data used for reporting.

### Application Programming Interface (API)

Coalesce's REST API can be used to trigger deployment and refresh runs. The Coalesce Cloud will execute runs in an isolated environment.

The API supports Snowflake OAuth, Basic auth with password in API request, or Snowflake Basic with Cloud Storage with only username specified.

A Snowflake connection and metadata connection is established in the same manner as the CLI, originating from the Coalesce cloud.

## Multi-Tenancy

### Frontend Multi-tenancy

When editing Coalesce metadata directly from the browser, all requests are protected by [Metadata Repository Security Rules][] to prevent unauthorized access to metadata being authored in the GUI.

Each potential access (read, write, edit) to metadata requires an associated authenticated user. This user information is then used to determine if the user is a valid member of an organization. Upon validly matching a user to an organization to which they belong, access to metadata is permitted.

All accesses without a valid user, or a user not belonging to the organization are rejected.

### Backend Multi-tenancy

When submitting requests to the backend, each request comes with a token that is then used to authenticate the user. Upon successful authentication of the user, the user's organization is determined and rules enforced such that each user can only access, edit and update their own organization's information in the industry standard credential manager.

All requests without a valid user, or a user not belonging to the organization are rejected.

[here]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[browser git integration]: /docs/architecture-and-security/security#browser-git-integration
[browser Snowflake SQL execution]: /docs/architecture-and-security/security#browser-snowflake-sql-execution
[Metadata Repository Security Rules]: /docs/architecture-and-security/architecture#metadata-repository

---

## Filter Node Output with SELECT Statements

When creating data flows in Coalesce, it's common to need to filter or limit data from an upstream node before passing it downstream. For example, selecting only the latest records or a subset of columns.

It's important to understand that nodes don't actively push data to downstream nodes. Instead, nodes are queried by consumers via SQL SELECT statements that stitch together the data flow graph.

This means any row limiting logic needs to be applied in consuming nodes, not publishing nodes.

## Best Practice: Filter in Consuming Node in Join

The proper way is to place a WHERE condition in consuming node JOIN clause.

```sql
SELECT * 
FROM MyNode
WHERE RowNum = 1
```

Now only row 1 will be created in the database object defined by the node.

## Not Recommended: Filtering in Publishing Node

You CANNOT do this, for example in Post SQL.

```sql
SELECT * FROM MyNode WHERE RowNum = 1
```

This won’t actually filter what data reaches subsequent nodes.

---

## Fix Un-Linked References in Sub-Graphs in Coalesce

In Coalesce, [Sub-Graphs][] are visual tools that help the user organize complex transformation workflows into smaller units. For broader Subgraph practices, see [Using Subgraphs to Build Your Pipeline][]. In this short article we will resolve a common issue, that is unresolved visual references to nodes when referencing objects in Sub-Graphs.

In a scenario where tables feeding dimensions and fact tables are not consistently linked in the graph, users may encounter challenges despite following similar logic. Additionally, situations may arise where a Sub-Graph fails to display links between tables, even when referenced in the Pre-SQL code.

## Solution

You can solve this in two ways.

### Use Ref Functions

Use [special macros][] called `ref_link()` and `ref_no_link()` create dependencies to the Nodes without writing the fully qualified name into the script, offering a targeted solution for the visualization challenges.

```sql
FROM {{ ref('TGT_STAGE', 'STG_X_TRANSACTIONS') }}
"STG_X_TRANSACTIONS"
WHERE 1=1
AND "DOCDATE" >= '2021-01-01'
{{ ref_link('TGT_STAGE', 'STG_X_TRANSACTIONS_DELETES') }}
```

### Include the Table in the WHERE or FROM Clause

Include the table in the WHERE or FROM clause of the join but commenting it out. This ensures that Sub-Graphs recognize commented-out tables as part of the metadata, even if they are not utilized in the join.

[Sub-Graphs]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs
[special macros]: /docs/reference/ref-functions

---

## How to use GROUP BY

GROUP BY is completed in Coalesce by adding your GROUP BY clause after the FROM statement, as well as after any other precursor statements like JOINs and/or a WHERE clause, in the Join tab of your Node Editor of the Node where you will complete the GROUP BY.

You will have to ensure your columns are configured correctly in your Mapping tab in terms of aggregation Transforms on column you wish to aggregate and Sources for columns you wish to GROUP BY in order for the GROUP BY to execute successfully.

## GROUP BY Detailed Example

To provide a more detailed example, let's say you have a raw table of monthly sales amounts by store and month that you'd like to transform and load into a Fact Table of sales amounts by store in total. But rather than loading the actual store value into your Fact Table, you want to pull in the corresponding dimension key for each store from your Store Dimension, so you also need to join to your Store Dimension while completing this transform and load.

1. Add a Fact Node from your main source (`MONTHLY_SALES`).

    

2. Remove the columns you don't want to include in your new Fact (`STORE` in its raw form and `MONTH`) and configure your aggregations on your columns.

    

3. Map in the key from your Store Dimension (`DIM_STORE`).

    

4. Configure your join between your main source (MONTHLY_SALES) and your Store Dimension (`DIM_STORE`).

    

5. Add a GROUP BY clause after your join on your Store Dimension key (`DIM_STORE.DIM_STORE_KEY`).

    

6. Within the Node Editor:
    1. Validate Select.
    2. Validate Create.
    3. Create.
    4. Validate Run.
    5. Fetch data, and view grouped results.

---

## Include Multiple SQL Statements in the Override SQL Part of a View Node

When working with SQL views in your database management system, you might encounter situations where you need to include multiple SQL statements in the override SQL part of a view node. While the Coalesce does not support multiple SQL statements directly in a single override, there is an effective workaround that allows you to achieve this functionality.

You can create another stage in the overwrite SQL code with a different name than the previous stage. When the view node is deployed, both stages are executed in order.

---

## Load DISTINCT Values Into a Node

There a few ways you can load DISTINCT data into Coalesce.

## View Node Type

First load your data into a View node and select DISTINCT values. Then create your Stage or Persistent stage from the View node.

:::info[View Nodes]

The View Node Type is disabled by default, but can be enabled in your **Build Settings > Node Types**.

:::

## GROUP BY ALL Clause in Join

If you want to use an existing Node Type like a Stage or another Node Type that doesn't offer the DISTINCT toggle, you can add a "GROUP BY ALL" clause in the Join tab of the Node Editor.

> Snowflake
>
> Snowflake executes this as DISTINCT.

## Create a Custom Node Type With SELECT DISTINCT Behavior

If you wish to have a Node Type with this functionality built in so you can use it for many Nodes without having to complete one of the approaches detailed above, you can duplicate an existing Node Type and edit the Create Template (for View behavior) and the Run Template (for Table behavior) to include the DISTINCT keyword in the SELECT, so that it is automatically included for all Nodes of this new Node Type.

<figure>

<figcaption>DISTINCT in the Create Template</figcaption>
</figure>

<figure>

<figcaption>DISTINCT in the Run Template</figcaption>
</figure>

---

## Preserving Column Lineage When Using SQL Statements

When using SQL statements involving multiple columns in Coalesce, we recommend using **qualified column names**, including the node name.

When using SQL statements involving multiple columns in Coalesce, we recommend using **qualified column names**, including the node name.

## Example

You have a multi source node called `MY_NODE` and it has columns `COL_A`,`COL_B`, and `COL_C`.

You decide to build a calculated column using a statement like:

```sql title="NOT RECOMMENDED"
CASE WHEN COL_A IS NULL THEN COL_B ELSE COL_C END
```

This will not display the column lineage because its missing the node name to make it a qualified column name. Instead build your statements like the following:

```sql title="RECOMMENDED"
CASE WHEN MYNODE.COL_A IS NULL THEN MYNODE.COL_B ELSE MYNODE.COL_C END
```

`MY_NODE` is added to each column name, which keeps the lineage.

Coalesce app has an autocomplete feature to make it easier to include the node names. Following this best practice will keep your transformations future proof.

## What's Next

- [Column Lineage][]

[Column Lineage]: /docs/build-your-pipeline/column-lineage

---

## Self-Join a Coalesce Node to Itself

When building transformation pipelines with Coalesce, one may need to join a table or view with itself, known as a “self-join”. Some typical use cases for self-joins include connecting records in a hierarchical structure, comparing records to other records in the same table, debugging a table to find gaps/missing elements, etc.

DAGs are normally one directional, but using [`{{ ref_no_link () }}`][] makes this possible.

## Self-Join a Node

1. In your Workspace, open the Node you want to add the self-join to and navigate to the **Join** tab.
2. Configure the join by specifying the same Node within the `{{ ref_no_link () }}` function.
3. Be mindful of to alias both mentions of the same table under a different name to prevent ambiguous reference errors when running the query.
4. Set the appropriate join keys according to the aliasing configured in step 3.

```sql
FROM {{ ref('SRC', 'ROLE_HIERARCHY' }} "ROLE HIERARCHY1"
LEFT JOIN {{ ref_no_link('SRC', 'ROLE_HIERARCHY' }} 
ON "ROLE_HIERARCHY1"."ROLE_ID" = "ROLE_HIERARCHY"."ROLE_ID"
```

[`{{ ref_no_link () }}`]: /docs/reference/ref-functions#ref_no_link

---

## Use Snowflake User Defined Functions (UDF)

## Simple Example

Here is an example of a simple function in Snowflake that returns pi to 9-digits and doesn't accept any argument.

```sql
create function mydb.myschema.pi_udf()
returns float
as '3.141592654::FLOAT';
```

To use the above UDF as a transform in a **Node**, take the following steps:

1. Run the above statement in Snowflake and make note of the database and schema it's in.
2. Go to **Build Settings >vStorage Locations**.
3. Create a **Storage Location** - named `FUNC` in the following examples, but feel free to choose any name.
4. Map that **Storage Location** to mydb.myschema where the UDF was created in Step 1.
5. Use `ref_no_link` to specify the storage location from Step 1, with name of function as 2nd parameter. It is very important to use `ref_no_link` as we do not want to attempt to create a relationship to another Coalesce metadata object. In this example, `{{red_no_link('FUNC', 'PI_UDF')}}()`.

    **General format - `{{ref_no_link('storage_loc', 'func_name')}} (udf_argument1, udf_argument2, ...)`**

    

6. **Run** the node and you'll notice that all the entries for the `S_ACCTBAL` column will be 3.14.

## Example With UDF Arguments

For this example, we'll use a function that accepts one argument and carries out a simple calculation.

```sql
CREATE FUNCTION mydb.myschema.area_of_circle(radius FLOAT)
RETURNS FLOAT
AS
$$
  pi() * radius * radius
$$;
```

To use the above UDF as a transform in a **Node**, take the following steps:

1. Run the above statement in Snowflake and make note of the database and schema it's in.
2. If you went through the previous example, skip to the next step. Otherwise, follow Steps 2 to 4.
3. Use `ref_no_link` to specify the storage location from Step 1, with the name of the function as the 2nd parameter, and the UDF argument will be in parenthesis `()` afterward.

    **General format - `{{ref_no_link('storage_loc', 'func_name')}} (udf_argument1, udf_argument2, ...)`**

    

4. **Run** the node and you'll notice that all the entries for the `S_ACCTBAL` column will be much larger numbers, as they were multiplied times themselves and pi.

---

## Use the Coalesce Test Stage Macro

Coalesce provides [built-in data validation tests][] that can be configured via the Testing UI and metadata. However, sometimes more custom or complex tests are needed.

By leveraging the `test_stage()` built-in macro and SQL templates, you can create user-defined test nodes to validate data flows.

## Test Stage Limitations

The out-of-the-box testing functionality has some limitations:

- Manual configuration of test logic required
- Hard to reuse tests across nodes
- Limited flexibility in SQL and controls

## Creating Your Test

You'll need to:

- Author a macro with your test logic and name it `test_<stg_name>`. In the examples we use `My Custom Test`.
- Invoke the Macro within a custom node’s `Run Template`.
- Control failure handling behavior with a true/false clause in the stage initiation.

This tests for NULLs and does not continue the test on a failure and will fail the node execution. Using false is what will cause the failure.

```sql
{{ test_stage('My Custom Test', false) }} 

SELECT 1
FROM {{this}}
WHERE Col IS NULL
```

If you set the conditional to `true`, the test will continue the node execution after the test failure.

```sql
{{ test_stage('My Custom Test', true) }} 

SELECT 1
FROM {{this}}
WHERE Col IS NULL
```

## What's Next

- [Data Quality Testing][]
- [Macros][]

[built-in data validation tests]: /docs/build-your-pipeline/data-quality-testing/
[Data Quality Testing]: /docs/build-your-pipeline/data-quality-testing/
[Macros]: /docs/reference/macros/

---

## How To Build Your Data Pipelines

Use these how to guides to help you build your Coalesce integration.

<DocCardList />

---

## Migrate a CTE to Coalesce

Coalesce believes pipelines, rather than Common Table Expressions (or CTEs), are the ideal approach for data transformations. We've included a high level overview of each approach below, and our article [CTEs vs Pipelines for Complex Data Processing][] details our philosophy on this approach as well as provides an introductory walkthrough of the approach. In this article, we dig deeper into the approach with a step by step example detailing how to migrate a CTE to a pipeline in Coalesce.

## Common Table Expressions (CTEs)

Common table expressions, or CTEs, are used to embed complexity in several SQL blocks pulled together into a single SELECT or DML operation. Within the context of data transformation, by processing data this way, developers are able to consolidate logic within the context of a single SQL file. This means there are less files to manage overall, but managing the logic contained within those files becomes increasingly more complex as data transformation projects grow.

## Pipelines

Pipelines allow for each logical process in a typical CTE to be broken into their own SQL blocks, or what Coalesce calls Nodes. In addition to reducing complexity, the pipeline approach provides lineage at the object and column level that is not possible with CTEs.

## Build with a Pipeline Approach in Coalesce Video

Coalesce believes in a pipeline approach for data modeling and transformation. However, if you are migrating from a code first solution that uses CTEs to Coalesce, it is not always obvious where to start. In the following video, we show how you can take a CTE and what migrating that into Coalesce looks like.

<YouTube id="v69fqn80s-A" />

## Migrating a CTE to Coalesce

We'll now walk you through the process of migrating a CTE to Coalesce. Here is the CTE code that we will be migrating to Coalesce:

```sql
with stg_location as (
    select location_id,
           placekey,
           location,
           upper(city) as city,
           region,
           iso_country_code,
           country
    from FROSTBYTE_TASTY_BYTES.RAW_POS.LOCATION
    where location_id is not null
      and location_id != 0
),

stg_customer_loyalty as (
    select customer_id,
           first_name,
           last_name,
           upper(city) as city,
           country,
           left(postal_code, 5) as postal_code,
           e_mail,
           phone_number,
           coalesce(e_mail, phone_number) as contact_info
    from FROSTBYTE_TASTY_BYTES.RAW_CUSTOMER.CUSTOMER_LOYALTY
    where customer_id is not null
      and (e_mail is not null or phone_number is not null)
),

stg_truck as (
    select truck_id,
           menu_type_id,
           primary_city,
           year,
           upper(make) as make,
           upper(model) as model,
           ev_flag,
           franchise_id,
           truck_opening_date
    from FROSTBYTE_TASTY_BYTES.RAW_POS.TRUCK
    where truck_id is not null and TRUCK_OPENING_DATE::date < '2024-01-01'
),

stg_order_header as (
    select order_id,
           truck_id,
           order_id || truck_id as primary_key,
           location_id,
           customer_id,
           discount_id,
           shift_id,
           shift_start_time,
           shift_end_time,
           order_channel,
           order_ts,
           order_ts::date as order_date,
           served_ts,
           order_currency,
           order_amount,
           order_tax_amount,
           order_discount_amount,
           order_total
    from FROSTBYTE_TASTY_BYTES.RAW_POS.ORDER_HEADER
    where order_id is not null
      and truck_id is not null
      and order_total >= 0
),

stg_order_totals as (
    select order_id,
           order_currency,
           sum(ORDER_AMOUNT) as total_order_amount,
           sum(ORDER_TAX_AMOUNT) as total_tax_amount,
           sum(ORDER_DISCOUNT_AMOUNT) as total_discount_amount,
           sum(order_total) as total_order_value
    from stg_order_header
    group by order_id,
             order_currency
),

orders_cleaned as (
            select soh.order_id,
                       soh.truck_id,
                       primary_key,
                       soh.location_id,
                       soh.customer_id,
                       discount_id,
                       order_ts,
                       order_date,
                       served_ts,
                       sot.order_currency,
                       total_order_amount,
                       total_tax_amount,
                       total_discount_amount,
                       total_order_value,
                       sl.city,
                       location,
                       make as truck_make,
                       model as truck_model,
                       first_name as cust_first_name,
                       last_name as cust_last_name,
                       contact_info,
                       row_number() over (partition by primary_key order by order_ts) as dupe_count
                from stg_order_header as soh
                inner join stg_location as sl
                on soh.LOCATION_ID = sl.LOCATION_ID
                inner join stg_truck as st
                on soh.TRUCK_ID = st.TRUCK_ID
                inner join stg_customer_loyalty as scl
                on soh.CUSTOMER_ID = scl.CUSTOMER_ID
                inner join stg_order_totals as sot
                on soh.order_id = sot.order_id
            where order_date is not null
              and contact_info is not null
            qualify dupe_count = 1
)

select *
from orders_cleaned
```

### Identify Your Data Sources

When migrating a CTE to Coalesce, the first thing you should do is identify all of the data sources that will need to be brought into your pipeline.

:::info[CTE Dependency]

It is possible that a CTE will have a dependency that will need to be created first, before you can migrate the CTE. You should create any dependencies first within Coalesce.

:::

Once you have identified your data sources, you can create them within your Coalesce workspace. In the example CTE used here, each of the sources is a raw data table, so these tables can be added in as sources into the Coalesce workspace.

In the case of a CTE having a dependency that is not a raw data source, you would create or use existing nodes within your pipeline, to support the logic of the CTE.

### Migrate Your Logic in Sequential Order

Once you have data sources added to your pipeline, you can begin migrating the logic from your CTE in the order in which the CTE is written (top to bottom). This will ensure that any dependencies created **within** the CTE are available to be processed as you complete your migration.

For example, in the image below, you can see the _`stg_order_totals` CTE is dependent upon the `stg_order_header`_ CTE. If these were to be created out of sequential order, the `stg_order_header` CTE would not be available for the `stg_order_totals` CTE to reference.

### Migrate Each CTE

Once you have identified the order in which you are migrating your CTEs, you can begin migrating each CTE into Coalesce. Let’s take a look at migrating the first CTE from our code.

It can be helpful to think of each SELECT statement in a CTE as its own node in Coalesce. Although this will not always be the case, in most scenarios, a single CTE can be represented by a single node. Since this CTE is processing some raw data and applying some transformations, we can use a stage node to reflect these transformations in Coalesce.

Coalesce will automatically bring in all of the columns from the source table. From there, we can apply the UPPER function to the city column, within the **Transform** field of Coalesce.

There is a WHERE statement that has two filter conditions applied. We can apply the same logic within the **JOIN** tab of Coalesce.

With that, we’ve just migrated the `stg_location` CTE into Coalesce.

## Migrating a CTE Complex Example

Let’s walk through one more example that’s a little more complex.

The `order_cleaned` CTE is joining multiple other CTEs together, adding a column, and applying some filters.

First, we need to ensure that the upstream dependencies have been created.

- `stg_order_header`
- `stg_location`
- `stg_truck`
- `stg_customer_loyalty`
- `stg_order_totals`

For this example, let’s assume you’ve already migrated the rest of these nodes:

With these nodes available, we can create a new node that joins all of these nodes together: just like in the CTE. You can hold down shift and select all of the nodes together, right click on any node, and select **Join Nodes > Stage**.

Coalesce will automatically generate the join syntax in the Join tab of the new node.

From here, all you need to do is replace the `/*COLUMN*/` with the actual column reference for the join.

With the join complete, we can go back to the mapping grid and delete the columns we don’t need, so they can match the same columns from the CTE. Additionally, we can add the window function called `DUPE_COUNT` by double clicking on the Column Name area of the mapping grid.

With the window function added, we can apply the filters to the node.

With the filters added, we have successfully migrated a complex CTE into Coalesce, while taking advantage of several of the features of Coalesce that allow you to streamline development.

Once each CTE has been migrated to Coalesce, your pipeline using the code above, would look something like this:

## Migrate Intentionally

Migrating CTEs to Coalesce is a streamlined process that allows you to gain additional transparency, adaptability, and scalability. When migrating CTEs, it is important to follow a migration process to ensure a CTE can be fully created in Coalesce:

1. Identify data sources
2. Identify any dependencies
3. Migrate CTEs in sequential order

By following this process, migrations can be quickly and easily completed, allowing you to gain all the advantages of building your data transformations in a pipeline pattern. For questions about migrating CTEs, you can open a [support request][].

[CTEs vs Pipelines for Complex Data Processing]: https://coalesce.io/product-technology/ctes-vs-pipelines-for-complex-data-processing-why-coalesce-offers-the-better-approach/
[support request]: mailto:support@coalesce.io

---

## SCD Type 2 Change Detection and String Case

This page explains how Coalesce decides a tracked attribute changed for SCD Type 2 Dimension Nodes, how letter case affects that decision, and what you can do in SQL when you want values such as `Monday` and `monday` treated as the same.

## How Type 2 Detects a Change

Coalesce uses your Dimension Node **Advanced Deploy** settings, including change tracking columns and business keys, to build MERGE or equivalent refresh logic for your warehouse. When Coalesce compares the incoming row with the current row, it relies on SQL equality for the columns that participate in that logic. If the database treats the old and new values as not equal, the pipeline can treat the row as changed, which may create a new historical version for Type 2.

### Change Tracking Columns

The attributes you selected so that updates create a new historical version instead of only overwriting the current row. Changes to those columns drive Type 2 behavior, together with your business keys and any Last Modified Comparison options your package documents in **Advanced Deploy**. Coalesce does not apply a separate string comparison mode: detection follows the comparison rules of the warehouse for the data types and expressions in the generated SQL.

### String Comparison by Warehouse

Equality for string-like types follows your platform defaults unless you change expressions or collation. The examples below use simple string literals. Always confirm behavior for your data types, collation, and session settings.

<Tabs>
  <TabItem value="bigquery" label="BigQuery" default>

    In BigQuery:

    - **STRING** comparisons are **case-sensitive** by default.
    - Example: `'Monday' != 'monday'`.
    - BigQuery SQL has no **`ILIKE`**. For case-insensitive pattern matching, use **`REGEXP_CONTAINS`** with a case-insensitive regular expression, for example the `(?i)` flag in the pattern, or normalize with **`UPPER()`** or **`LOWER()`** and compare those results.
    - **`UPPER()`** and **`LOWER()`** are available for case normalization in expressions.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    In Databricks with Spark SQL:

    - **String comparisons** are **case-sensitive** by default.
    - Example: `'Monday' != 'monday'`.
    - **`ILIKE`** is available for case-insensitive pattern matching.
    - You can use **`UPPER()`** or **`LOWER()`** for case-insensitive equality.

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

    In Snowflake:

    - **`VARCHAR` equality** is **case-sensitive** by default.
    - Example: `'Monday' != 'monday'`.
    - You can make comparisons case-insensitive in supported contexts using a **`COLLATE`** clause when your account and data type allow case-insensitive collation, or by comparing **`UPPER()`** or **`LOWER()`** results.
    - **Default collation** is case-sensitive.

  </TabItem>
</Tabs>

## When Letter Case Alone Appears as a Change

If a source sends `Monday` today and `monday` tomorrow on the same business key, and the changed attribute is change-tracked, the pipeline may open a new Type 2 version because the values are not equal under SQL rules. That is expected when you have not normalized strings upstream.

If your requirement is that only semantic value matters and case-only differences should not create history, you must make those values compare equal before they reach the Dimension Node. Typical approaches include standardizing case in a Stage, View, or column mapping.

## What You Cannot Configure Away in the UI

Dimension **Advanced Deploy** does not expose a setting to turn on case-insensitive Type 2 behavior or to normalize strings inside Coalesce merge comparison for all tracked columns.

Your package may document **Exclude Columns from Merge** on Snowflake Base Dimension behavior. That option applies to SCD Type 1 merge options: it excludes columns from comparison and update phases for that merge style. It is not a general control to ignore letter case for Type 2.

:::info[DDL Case Insensitive applies to external load naming]

The **DDL Case Insensitive** option on External Data Nodes in the [External Data Package][] concerns how column identifiers and load DDL are handled for certain file-based loads. It does not change how a Dimension Node compares cell values during SCD Type 2 refresh. Don't use that toggle expecting different merge value semantics for dimensions.

:::

## Normalize Strings Upstream

The supported approach is to apply a consistent expression to any business key or change-tracked attribute where casing would otherwise cause false change signals. Typical patterns use warehouse functions such as `UPPER`, `LOWER`, or `INITCAP` in the layer before the Dimension Node so both the existing target row and the incoming batch share the same canonical form.

Example patterns in a Stage or View select list:

```sql
UPPER(customer_name) AS customer_name,
LOWER(email) AS email,
INITCAP(product_title) AS product_title
```

Pick one standard per column and keep it stable so historic rows and new loads stay aligned.

### Optional: Tracked Column Versus Display Column

If you must keep exact source casing for reporting but still want stable Type 2 detection, you can sometimes maintain the following:

- **Tracked column:** Normalized expression used in mappings and change detection.
- **Display column:** Raw or lightly transformed value for presentation.

This adds modeling and mapping overhead. Verify that business keys, grain, and **Advanced Deploy** selections still match your Kimball or internal standards and that the generated SQL does what you expect.

## Trade-Offs

Use the following table to compare normalizing case upstream with leaving source strings as loaded.

| Approach | What you gain | What to watch for |
| --- | --- | --- |
| Normalize upstream | Fewer spurious Type 2 rows from benign case churn | The stored value in the tracked column may not preserve original letter case unless you keep a separate display field |
| Leave values as loaded | Preserves literal strings from the source | Case-only flips can inflate history when the warehouse treats them as different values |

## Troubleshooting

If case behavior causes unexpected Type 2 rows or you relied on the wrong UI control, start with the following table.

| Issue | Resolution |
| --- | --- |
| New Type 2 rows appear when only letter case changed on a change-tracked string | Normalize that column upstream with `UPPER`, `LOWER`, or `INITCAP`, or use warehouse-specific case-insensitive comparison in the layer that feeds the Dimension Node so equality matches your intent. |
| You expected **DDL Case Insensitive** to fix dimension merges | That setting applies to External Data Node load DDL and naming, not to Dimension Node value comparison. See [External Data Package][]. |

## What's Next?

- [Nodes and Node Types][] for how Dimension Nodes use change tracking for Type 2.
- [Snowflake Base Node Types Advanced Deploy][] for Dimension MERGE, change tracking columns, and **Exclude Columns from Merge** scope.
- [BigQuery Base Node Types Advanced Deploy][] for BigQuery Dimension SCD Type 2 options.
- [Handling Duplicate Records in SCD Type 2 Dimension Nodes][] if merge errors point to non-unique business keys rather than value comparison.
- [Coalesce Foundational Hands-On Guide][] for hands-on Type 2 practice.
- [External Data Package][] for **DDL Case Insensitive** on external loads.

---

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Snowflake Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy
[BigQuery Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types-advanced-deploy
[Handling Duplicate Records in SCD Type 2 Dimension Nodes]: /docs/faq/build-faq-troubleshooting/handling-duplicate-records-scd-type-two-nodes
[Coalesce Foundational Hands-On Guide]: /docs/guides/coalesce-foundational-hands-on-guide
[External Data Package]: /docs/marketplace/package/coalesce_snowflake_external-data-package

---

## Bulk Editing

Coalesce supports bulk editing of columns in Column grid view and Node Editor.

## Access Bulk Editing

1. You can access bulk editing from the browser by changing your view to **Column Grid**, or directly in a node from the **Mapping View**.
2. Select the columns you want to by holding Command (Mac) or CTRL (Windows).
3. Right click and select **Bulk Edit Columns**.
4. This will open up a window where you can select the **Attributes** you would like to edit.
5. Click on Preview to view the changes and click on Update to apply those changes to the columns.

## Available Attributes

### Transform

SQL from any of our data platforms is supported. Jinja, as well as macros, may be used, however, they will be stored in their raw Jinja form. However, the Jinja will be rendered when the node is executed.

**Helper Tokens Allowed:** Yes

### Data Type

Any of our data platforms data types.

**Helper Tokens Allowed:** No

### Source

- Operation:
  - Node: Replace column reference with a matching column from the specified node in the **Source** selector.
  - Node and Column: Replace column reference with the specified node and column in the **Source** selector.
- Source: Used to specify `{ Node | Node and Column }` to be used in the bulk edit.

**Helper Tokens Allowed:** Not applicable

### Nullable

**Helper Tokens Allowed:** Not applicable

### Default Value

**Helper Tokens Allowed:** Not applicable

### Description

**Helper Tokens Allowed:** Not applicable

### Helper Tokens

These tokens are replaced during a bulk operation to provide column or node context to the operation.

|Token|Renders|Snowflake|Databricks|
|--- |--- |--- | ---|
|`{{SRC}}`|fully qualified source node and column|`"CUSTOMER"."C_CUSTKEY"`|`` `CUSTOMER`.`C_CUSTKEY` ``|
|`{{SRC_COL}}`|source column name|`"C_CUSTKEY"`| `` `C_CUSTKEY` ``|
|`{{SRC_NODE}}`|source node name|`"CUSTOMER"`| `` `CUSTOMER` ``|
|`{{TGT}}`|fully qualified current node and column|`"STG_CUSTOMER"."C_CUSTKEY"`|`` `STG_CUSTOMER`.`C_CUSTKEY` ``|
|`{{TGT_COL}}`|current column name|`"C_CUSTKEY"`| `` `C_CUSTKEY` `` |
|`{{TGT_NODE}}`|current node name|`"STG_CUSTOMER"`|`` `STG_CUSTOMER` ``|

:::info[Helper Tokens Rules]

- The tokens are case-sensitive and space-sensitive.
- They must be uppercase with no spaces.
- Helper Tokens are not Jinja variables despite their appearance.

:::

### Using Helper Token

Let's say you want to add an `IFNULL` transformation to several columns.

In this case, you can use `{{SRC}}` to make this operation quick and easy. First, select the columns from the grid that you wish to modify the transform on. Then, select `Transform` from the attributes list.

Add the transform as `NVL( {{SRC}} , 'some other value' )`.

## Editing Column Name

:::note[Common Transformations]

More transformation examples can be found in [Common Transformations][].

:::

Column bulk editor also supports Jinja for the Column name attribute.

```json
{
  column: {
    name
    id
    dataType
    description
    nullable
    defaultValue
  }, // the column being edited
  columns: [], // all columns, including the current one
  storageLocations: [] // currently available storage locations
  config: {} // any user defined config attributes defined in node spec
  sources: [] // all sources for current node
  node: {} // all current node metadata
  this: {} // {{ ref_no_link for current node }} 
  parameters: "" // runtime parameters
}
```

### Add in a Prefix

```sql
myPrefix_{{ column.name }}
```

### Removing Foo From a Column Name

```sql
{{ column.name | replace("foo", "") }}
```

### Change All Column Names to Upper

```sql
{{ column.name | upper }}
```

[Common Transformations]: /docs/category/common-transformations

---

## Column Lineage and Propagation

Coalesce includes a way to quickly view the lineage of each column as it is transformed throughout your Nodes.

## View Column Lineage

To view **Column Lineage**:

1. Either open a Node or change to Column Grid view.
2. Right-click on any populated column and select **View Column Lineage**.

    <figure>
    
    <figcaption>Mapping Grid</figcaption>
    </figure>

    <figure>
    
    <figcaption>Column Grid</figcaption>
    </figure>

3. A new tab with the column lineage will open.

## Explore Column Lineage

* Column lineage shows automatically opens with all Nodes collapsed to save on space. You can expand the Nodes one at a time.
* The `t` next to the column name indicates that a transformation has been applied.
* Click the `:` to edit and View Lineage.

:::info

When viewing column propagation, the view changes to All, so all Nodes with that column can be shown.

:::

## Column Propagation

You can propagate column changes across nodes across Nodes with a few clicks. Each individual column can have its addition and deletion to other nodes propagated as well by clicking on the three dots next to a the column name and selecting the relevant option.

1. Open the Node with the changes you want to propagate.
2. Right-click on any populated column and select **View Column Lineage**.
3. You'll now be in a new tab with the current Node highlighted in blue.
4. To propagate column changes click `⋮` and then select **Propagate Addition** or **Propagate Deletion**.

    

5. You'll then be able to select the Node you want to propagate the changes to, preview, and apply the changes.

---

## Create and Manage Nodes Using the API

You can create and manage Nodes in your Workspace using the Coalesce API. This guide covers how to create new Nodes and update existing ones using API endpoints.

## Before You Begin

Before you begin, you'll need:

- Your [API token][] for authentication.
- Your [Workspace ID][].

## Create a Workspace Node

Use this endpoint to create a new Node with Workspace default values.

**Endpoint**: `POST /api/v1/workspaces/{workspaceID}/nodes`

### Getting the Node Type

You can find the `nodeType` in two ways:

1. **Through the Coalesce App**: Go to **Build Settings > Node Types**, then click **View**. The `nodeTypeID` shown is your `nodeType`.
2. **Through the API**: Use the [List Workspace Nodes][] endpoint to see existing Nodes and their types.

### Request Parameters

- `nodeType` (required): Can be a standard type or a custom `nodeTypeID`.
- `predecessorNodeIDs` (required): Array of Node IDs that should precede this Node. Use an empty array `[]` if there are no predecessors.

### Example: Create a Stage Node

```bash
curl -X POST "https://app.coalescesoftware.io/api/v1/workspaces/<YOUR_WORKSPACE_ID>/nodes" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "nodeType": "Stage",
    "predecessorNodeIDs": []
  }'
```

### Example: Create a Node with Predecessors

```bash
curl -X POST "https://app.coalescesoftware.io/api/v1/workspaces/<YOUR_WORKSPACE_ID>/nodes" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "nodeType": "Fact",
    "predecessorNodeIDs": ["node123", "node456"]
  }'
```

## Set Node

Use this endpoint to replace all fields of an existing Node in a Workspace.

**Endpoint**: `PUT /api/v1/workspaces/{workspaceID}/nodes/{nodeID}`

:::warning

This endpoint replaces the entire Node object. You must provide all required fields, not just the ones you want to change.

:::

### Required Fields

The complete Node object must include:

- `description`: Node description
- `id`: Node ID
- `locationName`: Storage Mapping location
- `name`: Node name
- `nodeType`: Node type
- `metadata`: Object containing columns, joins, and source mappings

### Example: Update a Node

```bash
curl -X PUT "https://app.coalescesoftware.io/api/v1/workspaces/YOUR_WORKSPACE_ID/nodes/YOUR_NODE_ID" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "YOUR_NODE_ID",
    "name": "STG_CUSTOMERS",
    "description": "Stage table for customer data",
    "locationName": "STAGE",
    "nodeType": "Stage",
    "metadata": {
      "columns": [
        {
          "columnID": "col1",
          "name": "CUSTOMER_ID",
          "dataType": "NUMBER",
          "description": "Unique customer identifier",
          "nullable": false
        }
      ]
    }
  }'
```

## Response Handling

Both endpoints return the complete Node object on success (HTTP 200).

---

## What's Next?

- [Set Node][] - Update an existing Node in a Workspace.
- [Create Workspace Node][] - Create a new Workspace Node.

---

[List Workspace Nodes]: /docs/api/coalesce/get-workspace-nodes
[Set Node]: /docs/api/coalesce/set-workspace-node
[Create Workspace Node]: /docs/api/coalesce/create-node
[API Token]: /docs/api/authentication
[Workspace ID]: /docs/reference/whats-my-id/#workspace-id

---

## Data Quality Testing

Coalesce provides built-in testing capabilities to assess the quality of your data.

Data quality tests are crucial for identifying issues with data, such as missing values, incorrect data types, outliers, and violations of business rules. Testing data quality within Coalesce offers the advantage of validating transformed data against required standards before loading it into the target system. Integration of data testing into the existing data pipeline workflow enables early issue detection and ensures consistency across different data sources and transformations.

## Coalesce's Approach to End-to-End Testing

Coalesce addresses this challenge by providing built-in Node testing capabilities . These tests can be at the [Column][] level or at the [Node][]level.

At the column level, you can test individual columns for uniqueness and null values. By clicking on the Testing icon after selecting a node. There, you can indicate the tests to run and which columns to run the tests on.

You can write custom tests at the node level with SQL queries. If the query returns a result, the test is considered failed. You can also leverage the Options to indicate whether or not to halt the pipeline execution upon test failure and whether to run the tests before or after the node refresh.

## Leveraging Environments for Comprehensive Testing

[Environments][]in Coalesce offer another powerful framework for end-to-end testing. Users can deploy data pipelines to different sets of databases and schemas, allowing them to simulate scenarios using 'dummy' data. By configuring different target Storage Location mappings for the desired output database and schema, users can thoroughly test their pipelines.

For example, you can designate an environment known as “DEV” that contains database objects that match edge cases for certain data (data with duplicates, null valued records, etc.) and once pipelines are deployed, can trigger certain tests/checks written into the nodes upon Refresh. Only once all “tests” pass in the QA environment, are the pipelines eligible to be deployed to production.

This approach enables you to point the pipeline to simulated data, ensuring thorough testing, and publish the outputs to a designated test dataset. It provides a flexible and robust solution for users aiming to verify their pipelines comprehensively.

## What To Test For

- **Missing or null values**: Test for columns that should not have null or missing values based on your business requirements.
- **Data types**: Ensure that columns have the correct data types (strings, integers, dates) to avoid downstream issues.
- **Uniqueness**: Test for columns or combinations of columns that should have unique values.
- **Value ranges**: Validate that numerical or date columns fall within expected ranges based on your business rules.
- **Referential integrity**: Test for foreign key relationships between tables to ensure data consistency.
- **Custom business rules**: Implement tests that enforce specific business rules or data quality requirements specific to your organization.

[Column]: /docs/build-your-pipeline/data-quality-testing/testing-columns
[Node]: /docs/build-your-pipeline/data-quality-testing/testing-nodes
[Environments]: /docs/get-started/coalesce-fundamentals/environments

---

## Testing Columns

Column tests are meant to check for null and unique values, and are always run after transformations. This cannot be changed unless using a user-defined node. Column-level tests must run after Node execution since views are dynamically materialized and don't appear in refresh statements. Node-level tests can run before execution using custom SQL, but column-level tests require the view to be materialized first.

## Example Column Test

1. Click on **Testing** → **Column**.
2. Select both **Unique** and **Null**.

    

3. Now click on **Null** next to `C_NATIONKEY`. This will check to ensure all entries for this column are populated (not null).
4. **Run** the node and it should succeed, as there's no null entries.

    

5. Now try also selecting **Unique** for the `C_NATIONKEY` column. This will check to ensure all entries for this column are unique.
6. **Run** the node and it will fail, as `C_NATIONKEY` has repeated entries. Notice that the data was still processed, as column tests will always continue on failure.

## Viewing Column Level Tests

After Create and Run you Nodes, you can see if the test passed or failed in the Results view.

The following image shows an example of a passing and failing test.

* The Node level test `Length is less than 0` is failing.
* The column level test `N_NATIONKEY: UNIQUE` is passing.
* The column level test `N_REGIONKEY: UNIQUE` is failing.

:::info

You can also see column and Node level test failures as part of [Refresh Results][].

:::

[Refresh Results]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes

---

## Testing Nodes

You’ll need to create a SQL SELECT statement to test nodes. If the results match any of the data in the node’s columns, the test will fail. Let's go through a simple example to demonstrate.

Node tests can be run before or after the Node's transformations. You can set the run option in the Node test.

## Example Node Test

The column `C_NATIONKEY` is an integer that corresponds to a specific country, and we know that this number must be positive. No countries in the data set should have a negative number, so we will test for this.

1. Click on **Testing**.
2. Click on **New Test** in the **Node** section to create a new test.
3. In the text box, type in `SELECT * FROM {{ ref_no_link('TARGET','STG_CUSTOMER')}} WHERE C_NATIONKEY < 0` but make sure to replace TARGET with the name of your storage mapping.
4. Uncheck the **Continue on Failure** toggle.

    

5. **Run** the node, it should succeed. You can confirm by opening the **Results** pane and seeing the different stages.

    

6. Try reversing the `<` to `>`, which will match all of our records (nation keys are all positive integers). This will cause the test to fail and the node's data will not be processed.

## Test Utility Package

Coalesce offers a Node testing package that allows you to deploy test Great Expectations like testing directly in Coalesce. Take a look at the [Test Utility Package][] for more.

## Viewing Column Level Tests

After Create and Run you Nodes, you can see if the test passed or failed in the Results view.

The following image shows an example of a passing and failing test.

* The Node level test `Length is less than 0` is failing.
* The column level test `N_NATIONKEY: UNIQUE` is passing.
* The column level test `N_REGIONKEY: UNIQUE` is failing.

:::info

You can also see column and Node level test failures as part of [Refresh Results][].

:::

[Refresh Results]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes
[Test Utility Package]: /docs/marketplace/package/coalesce_snowflake_test-utility

---

## Deleting Records from Fact and Dimension Nodes

For incremental staging and multi-source targets (including high-water mark loads and delete propagation patterns), see [Incremental loading strategies][].

## Default Behavior

By default, Coalesce Dimension Nodes don’t perform hard deletes. Instead, they use `MERGE` statements to manage updates and inserts.

## Recommended Approaches

### 1. Soft Deletes (Recommended)

The most common pattern is to use an **IS_ACTIVE** or **IS_DELETED** flag to mark records as inactive rather than physically removing them. This approach:

* Preserves historical data for auditing.
* Allows you to recreate data states at specific points in time.

### 2. Custom Deletion Logic

You can add deletion handling through custom logic, including:

* **Pre-SQL or Post-SQL** statements in your Node.
* **Custom `MERGE` statements** that include `DELETE` clauses when conditions are met.
* **Source-to-target comparison logic** to detect and remove obsolete records.

### 3. Using Snowflake Streams for Delete Detection

[Snowflake Streams](https://docs.snowflake.com/en/user-guide/streams-intro) can capture `DELETE` operations in source tables. You can use these changes to propagate deletes through your pipeline.

### 4. Full Refresh Strategy

Some organizations handle deletes with periodic full refreshes. While this removes deleted records, it can be inefficient for larger datasets.

## Implementation Considerations

When planning delete strategies, consider the following:

* **Fact tables**: Often use persistent staging with `MERGE` statements that include delete logic.
* **Dimension tables**: Can be configured for deletes through custom Node types or post-processing.
* **Source systems**: Ideally, configure source systems to use soft deletes instead of hard deletes when possible.

---

## What's Next?

* [Pre-SQL and Post-SQL](/docs/build-your-pipeline/pre-post-sql)
* [Incremental loading strategies][]
* [Snowflake Streams](https://docs.snowflake.com/en/user-guide/streams-intro)

---

[Incremental loading strategies]: /docs/build-your-pipeline/incremental-loading-strategies

---

## Generate Hash Columns

Generate Hash Column creates a new column containing a hash value calculated from selected columns in your Node.

## Creating a Hash Column

To generate a hash column:

1. Select one or more columns in your Node.
2. Right-click and choose **Generate Hash Column**.
3. Choose your preferred algorithm.
4. Customize the column name if needed.
5. Create and Run the Node.
6. The hash column appears at the bottom of your column list by default.
7. The selected columns **Contained in Hash** with the hash column name.

:::info

A column can belong to more than one hash.

:::

<Wistia id="2klvo5cr30" />

## Delete a Hash Column

If you delete the hash column, Coalesce automatically removes it from **Contained in Hash**.

## How It Works

When you select columns and right-click **Generate Hash Column**, the following happens:

- Creates a new column at the bottom of your column list.
- Calculates a hash value using your selected columns as input.
- Handles proper concatenation, delimiters, and null values automatically.
- Allows you to choose from multiple hashing algorithms.

The new column uses `GH` as the default name. You can change this name.

## Available Hash Algorithms

You can choose from these hashing algorithms when generating your hash column:

- **MD5**: Fast processing, smaller hash size.
- **SHA1**: Balanced security and performance.
- **SHA256**: Higher security, larger hash size.

## Use Cases

Hash columns solve common data problems and make your work faster. Here are a few ways to use them.

### Efficient Data Comparison

Hash columns simplify multi-column comparisons. Instead of writing complex WHERE clauses like `code = code AND type = type AND data = data`, you can compare two hash values to detect changes across multiple columns.

### Data Vault Architecture

Hash columns are essential for Data Vault modeling. They enable you to create:

- **Hub hash keys**: Business key hashes for hub entities.
- **Hash diffs for satellites**: Track changes in attributes over time.
- **Link hash keys**: Connect related entities across the data model.

### Change Detection

Hash columns excel in incremental loading and change data capture scenarios. They help you identify modified records since the last data load, making your ETL processes more efficient.

### Performance Optimization

For wide tables with many columns, pre-calculated hash values improve query performance. You compare a single hash column instead of multiple individual columns, reducing computational overhead.

## Best Practices

Follow these tips to get the most out of hash columns in your projects

### Column Selection

Choose columns that represent the business key or the attributes you want to monitor for changes. Avoid including frequently changing timestamps unless they're relevant to your use case.

### Algorithm Selection

- Use **MD5** for internal processing where speed is prioritized over cryptographic security.
- Use **SHA1** for balanced performance and security requirements.
- Use **SHA256** for scenarios requiring higher security standards.

### Naming Conventions

Use descriptive names for your hash columns. Instead of the default "GH", consider names like:

- `customer_hash` for customer-related hash keys.
- `product_hash` for product dimension hashes.
- `change_hash` for change detection scenarios.

## When to Use This Feature

Hash columns work well in data warehouse projects. They're especially helpful when you're building Data Vault models or need to spot changes across many columns in big data sets.

---

## Incremental Loading

With incremental loading, you process only new or changed rows since the last good run instead of reading the full data set every time. This page explains how incremental loading works in general and how you can apply it to your data platform.

## Incremental Loading Compared to a Full Refresh

Use this table to compare full refreshes and incremental loads at a high level.

| Aspect | Full refresh | Incremental load |
| ------ | ------------ | ---------------- |
| **Data processed** | The full source every run | Only new or changed rows |
| **Cost** | Rises with total size | Rises with how much changes |
| **Latency** | Often high for large sources | Often lower when changes are small |
| **Complexity** | Simpler patterns | You need a watermark, CDC, or careful state tracking |
| **Delete handling** | If you compare two full snapshots, missing rows can mean deletes | You usually model deletes with flags, CDC, or reconciliation |
| **Typical fit** | Small sources, full exports, planned rebuilds | Large facts, APIs with `UPDATED_AT`, stream-style CDC |

## Core Concepts

These ideas show up on every warehouse: how you bound each run, where you store loaded rows, what counts as changed data, and when change data capture is the right tool.

### Watermark

A **watermark** is the latest value of a column you've already loaded, often a timestamp. The next run only pulls rows above that line, like `WHERE incremental_column > watermark`.

### Persistent Target

With a **persistent table**, you keep the business rows you've already loaded from the incremental stage, not just bookkeeping metadata. The next run often takes the watermark from `MAX(incremental_column)` on that table. Stay consistent about where that watermark value comes from in your graph.

### Delta

The **delta** is what you want this run: inserts, updates, and deletes if you track them. Timestamp-style loads see inserts and updates that appear in the extract. Hard deletes in the source don't show up unless you add CDC, soft-delete columns, snapshot compares, or a delete feed.

### Change Data Capture

**CDC** means the source, or a layer in front of it, sends change events or change rows for inserts, updates, and deletes. Snowflake Streams, Kafka topics, database log CDC, and SaaS event APIs are examples.

## Choose an Incremental Strategy

This section will help you figure out which strategy works best for your data.

### Batch Jobs and Timestamp Column

This pattern is the most common. You run jobs on a schedule such as hourly or daily and have an `UPDATED_AT` column.

**Use:** Watermark with a timestamp column

<Tabs groupId="batch-timestamp-watermark">
  <TabItem value="snowflake" label="Snowflake" default>

On Snowflake, install the [Incremental Loading package][] and use the **Incremental Load** Node type.

1. In **Build Settings > Packages**, install the [Incremental Loading package][], then add an **Incremental Load** Node after your source. Add a **Persistent Stage**, **Dimension**, or **Fact**, or another persistent target, to hold rows you have already loaded.
2. For the first run, leave **Filter data based on Persistent table** off, **Create** the Node, and **Run** the persistent target so it is populated. For later runs, turn **Filter data based on Persistent table** on so the template compares staging to that table.
3. Set **Persistent table location** and **Persistent table name** to match your persistent Node. Set **Incremental load column** to the source column you use as the watermark, for example `UPDATED_AT`.
4. On the **Join** tab, use **Generate Join** and **Copy to Editor** so the incremental predicate matches your column names. Use **Pre-SQL** or **Post-SQL** when you need extra filters or housekeeping.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

On Databricks, install [Incremental Nodes][] and use the **Incremental Load** Node.

1. In **Build Settings > Packages**, install [Incremental Nodes][].
2. Add an **Incremental Load** Node downstream of your source. Set **Create As** to **table** or **view**, and set **Persistent table location** and **Persistent table name** to the table where loaded history lives.
3. For the initial load, set **Filter data based on Persistent table** to **False**, deploy, and load. For incremental runs, set it to **True** so the Node limits rows using the persistent table.
4. Set **Incremental load column (date)** to your watermark column. Review the generated SQL and adjust if your keys or filters need custom handling.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

BigQuery uses [BigQuery Base Node Types][] and [BigQuery Base Node Types Advanced Deploy][], not a single incremental Package.

1. Model your source in a **Work**, **Persistent Stage**, **Dimension**, or **Fact** Node, and point a downstream persistent Node at the table that should keep history.
2. Write the incremental slice in Node SQL with a predicate on your watermark column. Read the current high-water mark from the persistent table inside a subquery. Use [ref_no_link()][] when you need the fully qualified name without the DAG edge you would get from [ref()][] alone, and use [ref()][] when downstream order must depend on that target.
3. Open **Advanced Deploy** on Dimensions or Facts when you want **Last Modified Comparison** or documented merge options that fit incremental batches.
4. First run: either disable the incremental filter until the target has data, or seed a safe minimum watermark so you don't drop history by mistake.

  </TabItem>
</Tabs>

---

### Batch Jobs and Very Large Data

You run scheduled jobs, but data is too large for one run. Use when a single pass over the source is too large and you need buckets, history, or parameterized windows:

**Use:** Looped or batched incremental

<Tabs groupId="batch-large-volume">
  <TabItem value="snowflake" label="Snowflake" default>

On Snowflake, use **Looped Load** from the [Incremental Loading package][]. It builds load buckets from your grouping keys, loops through them, and can maintain a load history table for orchestration.

1. Install the package in **Build Settings > Packages**, then add a **Looped Load** Node downstream of the source or staging you need to chunk.
2. Configure **Load Type** and Job **Parameters** so each run uses the right mode. The Node supports values such as full load, full reload, incremental load, and reprocess load. Use reprocess-style runs when you need to retry failed buckets without reloading successes. See **Looped Load Parameters** on the [Incremental Loading package][] listing.
3. Under **Looped Load Incremental Load Options**, turn on **Set Incremental Load Options** when you still want a watermark inside each bucket, pick **Incremental Load Column (date)**, and set **Group Incremental** when you need more than one bucket per run.
4. Under **Looped Load Group Table Options**, choose **Dedicated Load History Table** and **Load History Table Location** when you want this Node's history isolated from other looped loads.
5. Use **Pre-SQL** and **Post-SQL** for setup and cleanup around each loop when the platform allows.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

Build batches with base Nodes and SQL you control.

1. Keep the **Incremental Load** Node from [Incremental Nodes][] when each batch is small enough. When it isn't, add **Work** or **Persistent Stage** Nodes whose SQL limits each run to a slice of the source, for example a date range, numeric key band, or file group.
2. Drive the slice with [Parameters][] you set at deploy or refresh so you can change the window without editing the Node again. One Job can process one `batch_id` value today and another tomorrow.
3. Chain Nodes or stagger Jobs so heavy batches don't overlap on one cluster. Use [Multi Source Nodes][] when several slices should land in one target with a defined load order.
4. Reuse the same watermark ideas as a standard incremental load, but apply them inside the batch predicate so you don't widen the scan past the current chunk.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

On BigQuery, batching is SQL and scheduling.

1. Filter on partition columns, ingest timestamps, or business dates so each query prunes storage instead of scanning the full data set.
2. Expose batch start, end, or shard id as [Parameters][] and reference them in **Work** or **Persistent Stage** SQL. Run the Job once per window, or run parallel Jobs with different parameter sets when your process allows it.
3. Land each batch in a staging Node, then merge or append into the persistent Fact or Dimension with [BigQuery Base Node Types Advanced Deploy][] so keys stay consistent across batches.
4. Watch bytes processed and slot time when sizing windows. Smaller batches cost more round trips; larger batches risk timeouts.

  </TabItem>
</Tabs>

---

### Batch Jobs and Full Dumps Without a Timestamp

You only receive full snapshots of data.

**Use:** Snapshot Comparison

When each Job receives a full snapshot and you must diff it against what is already in the warehouse:

<Tabs groupId="batch-snapshot-no-timestamp">
  <TabItem value="snowflake" label="Snowflake" default>

Compare snapshots with base Nodes, optionally combined with the [Incremental Loading package][] when your diff is already a view the package can load.

1. Land the latest extract in a **Work** or **Persistent Stage** Node from [Snowflake Base Node Types][], usually with **Truncate Before** so the object holds only the current file or batch.
2. Add **Work** Nodes that implement the diff, or use **Create with Override SQL** on staging paths for the same logic: for example keys present in the snapshot but absent from the target, row-level hashes or column compares for updates, and optional `NOT EXISTS` patterns for rows that disappeared if your design treats that as a delete signal.
3. Point **Dimension Advanced Deploy** or **Fact Advanced Deploy** from [Snowflake Base Node Types Advanced Deploy][] at the reduced row set. Use documented **Insert Strategy**, **Update Strategy**, **Unmatched Record Strategy**, and **MERGE** methods so inserts and updates land in one Job.
4. When the compare SQL lives in a single view you want to treat like an incremental source, you can still add **Incremental Load** from the [Incremental Loading package][] for load hygiene. You own the predicate or full replace semantics in the view logic.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

On Databricks, build the compare entirely in SQL on [Databricks Base Node Types][] **Work**, **Persistent Stage**, **Dimension**, and **Fact** Nodes.

1. Stage the full extract in **Work** with **Truncate Before** or **INSERT OVERWRITE** behavior so each run replaces the staging table with the new snapshot.
2. Join or subtract against the current persistent table in another **Work** Node. Use patterns your SQL engine supports, for example `EXCEPT`, `ANTI JOIN`, or hash compares, to emit insert and update rows.
3. For deletes inferred from missing keys, branch that logic in SQL or handle it in **Post-SQL** with a `MERGE` when you need database-native merge semantics that handle inserts and updates together.
4. Load the target **Fact** or **Dimension** with append or overwrite settings that match whether you are loading a full recomputed table or a delta table. Use [Multi Source Nodes][] when the compare splits into multiple ordered inserts.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

BigQuery follows the same staging-and-compare pattern using [BigQuery Base Node Types][].

1. Materialize the incoming snapshot in **Work** or **Persistent Stage** SQL, keeping grain and keys identical to the persistent model.
2. Compare to the target using `EXCEPT DISTINCT`, `NOT EXISTS`, or a hash column you calculate in SQL. Keep compare sets selective so you don't scan more bytes than needed.
3. Apply [BigQuery Base Node Types Advanced Deploy][] on the downstream **Dimension** or **Fact** so `MERGE` or other documented load methods apply your inserts and updates. Turn **Truncate Before** off when you merge into an existing table instead of reloading it wholesale.
4. Deleting keys missing from the snapshot requires explicit merge delete rules or a separate delete step. See [Deleting Records from Fact and Dimension Nodes][] when conformed models must stay aligned.

  </TabItem>
</Tabs>

---

### Batch Jobs and Full Accuracy With Deletes

You must detect inserts, updates, and deletes without CDC.

**Use:** Full Reload and Compare

When the staging table represents the full current source and you need **MERGE** behavior that can insert, update, and remove keys that disappeared from the extract, without CDC:

<Tabs groupId="batch-full-reload-deletes">
  <TabItem value="snowflake" label="Snowflake" default>

On Snowflake, [Snowflake Base Node Types][] hold the full snapshot and [Snowflake Base Node Types Advanced Deploy][] supplies **MERGE** with **Unmatched Record Strategy**.

1. Load every run's extract into **Work** or **Persistent Stage** with [Snowflake Base Node Types][], typically **Truncate Before** on so staging matches the file or table your Job just received.
2. Point **Dimension Advanced Deploy** or **Fact Advanced Deploy** at that staging Node. Set **Business Key** when the listing requires it, choose **MERGE** as the load method, and set **Update Strategy** to **Merge** so unmatched handling is available.
3. Set **Unmatched Record Strategy** to **NO DELETE**, **SOFT DELETE**, or **HARD DELETE** depending on governance. Dimension Advanced Deploy documents **SOFT DELETE** and **HARD DELETE** for rows no longer in the source. Fact Advanced Deploy documents **HARD DELETE** and **NO DELETE** where the listing applies.
4. Keep dependent Facts and Dimensions in **DAG** order when deletes propagate. If a Node must run after a delete staging step but doesn't reference it in SQL, enforce order with [ref_link()][]. Read [Deleting Records from Fact and Dimension Nodes][] for patterns that keep facts and dimensions consistent.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

Databricks relies on [Databricks Base Node Types][] plus SQL **MERGE** you own.

1. Stage the full extract in **Work** or **Persistent Stage** with **Truncate Before** so the table mirrors the latest snapshot.
2. Write a `MERGE` into the target **Fact** or **Dimension** table using the staging table as the source. Include `WHEN MATCHED` for updates, `WHEN NOT MATCHED` for inserts, and a `WHEN NOT MATCHED BY SOURCE` branch that deletes or flags rows when policy allows keys missing from the snapshot to leave the target.
3. Run that statement from **Post-SQL** on the target Node, or split into a **Work** Node whose sole job is the **MERGE**, depending on how you split deploy and refresh steps.
4. Document orphan-key rules for Facts before you **HARD DELETE** Dimension rows. Use [Multi Source Nodes][] only when multiple pieces of data must land in order before the **MERGE** runs.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

BigQuery uses [BigQuery Base Node Types][] for the snapshot and [BigQuery Base Node Types Advanced Deploy][] for **MERGE** with unmatched handling.

1. Materialize the full source in **Work** or **Persistent Stage** SQL each run so downstream **MERGE** always sees the complete current set.
2. On **Dimension Advanced Deploy** or **Fact Advanced Deploy**, choose **MERGE** and set **Unmatched Record Strategy** to **NO DELETE**, **SOFT DELETE** on Dimensions where documented, or **HARD DELETE** so rows absent from the source follow your retention policy.
3. Keep **Business Key** and **Truncate Before** aligned with the listing: you usually turn **Truncate Before** off on the **MERGE** target when you are merging into an existing table rather than replacing it.
4. For conformed stars, follow [Deleting Records from Fact and Dimension Nodes][] so Fact and Dimension deletes stay coordinated.

  </TabItem>
</Tabs>

---

### Streaming or Frequent Jobs and Change Events

Data arrives continuously or includes change metadata.

**Use:** Change data capture, **CDC**

When change rows or stream metadata are already available and you need near-continuous loads instead of one large batch window:

<Tabs groupId="streaming-cdc">
  <TabItem value="snowflake" label="Snowflake" default>

Snowflake CDC in Coalesce centers on the [Snowflake Streams and Tasks][] package: **Stream** Nodes hold the offset, and **Task** Nodes run the work on a schedule or when the stream shows data.

1. Install the package under **Build Settings > Packages**.
2. Add a **Stream** Node on the Snowflake table or view you want to track so Snowflake records inserts, updates, and deletes according to your stream options, for example **Append Only** or **Propagate Deletes** where the matrix applies.
3. Add **Stream and Insert or Merge** or **Delta Stream Merge** downstream. **Stream and Insert or Merge** applies the stream delta with **INSERT** or **MERGE**. **Delta Stream Merge** uses stream metadata such as `METADATA$ACTION` when you need tighter alignment on updates and deletes.
4. Attach **Work with Task**, **Dimension with Task**, **Fact with Task**, or **Insert or Merge with Task** when you need **Schedule**, **Stream Has Data**, warehouse sizing, or **Multi-Stream** orchestration the matrices describe.
5. Use **Task DAG Create Root** and **Task DAG Resume Root** so suspended Snowflake tasks actually resume in the right dependency order.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

Databricks streaming incremental loads often start with [Databricks Lakeflow Declarative Pipelines][]. That package, abbreviated LDP, creates streaming tables Lakeflow can refresh.

1. Install [Databricks Lakeflow Declarative Pipelines][] under **Build Settings > Packages** and add an LDP Node in the **Storage Location** where the streaming table should live.
2. Set **Read Files** to **True** when ingestion starts in cloud storage, or **False** when the pipeline reads from an existing table source. Match **File Location**, patterns, and **File Type** to your landing zone when files are the source.
3. Turn on **Schedule refresh** when you want the listing's schedule options to drive refreshes, and use **Refresh Stream** when structure changes require a full streaming table rebuild.
4. After column inference changes, use **Include Columns Inferred** and **Re-Sync** behavior the package describes so the streaming table schema stays aligned with files.
5. Downstream Coalesce Nodes on [Databricks Base Node Types][] can consume the streaming table on a Coalesce **Job** cadence you choose when you need joins or curated layers outside LDP.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

You run **CDC** by landing change rows or log tables in BigQuery with your replication tooling, then calling Coalesce often enough that those tables stay small.

1. Keep CDC tables or change streams in BigQuery current with the product you already use for replication so each Coalesce run sees only recent events.
2. Schedule Coalesce **Jobs** at the frequency your SLA allows, whether that is minutes or hours. Shorter intervals mean smaller reads and fresher marts.
3. In [BigQuery Base Node Types][], filter on the change timestamp, sequence, or `_PARTITIONTIME` column your CDC pattern provides. Use [ref_no_link()][] or [ref()][] the same way you would for timestamp-based incremental loads so watermarks stay consistent.
4. When CDC delivers explicit insert, update, or delete rows, apply [BigQuery Base Node Types Advanced Deploy][] **MERGE** options or staging **Work** Nodes that normalize operations before the **MERGE**.
5. Use [Parameters][] for look-back windows or source system IDs when one Workspace processes multiple CDC feeds.

  </TabItem>
</Tabs>

---

### Multiple Sources Feeding One Target

You have multiple inputs updating one table.

**Use:** Multi-source incremental, often paired with another scenario in this section

When more than one source must land in the same conformed table and you still want incremental or parameterized runs, align business keys and grain before you combine branches. If identifiers differ across sources, fix them upstream in **Work** Nodes.

<Tabs groupId="multi-source-incremental">
  <TabItem value="snowflake" label="Snowflake" default>

Both patterns live in the [Incremental Loading package][]. Pick **Run View** for parameterized **UNION**-style runs across separately named sources. Pick **Grouped Incremental Load** when the same template should run across **Storage Mapping** locations and the target stores which location each row came from.

#### Run View

Use **Run View** when you need parameterized **UNION**-style runs across separately named sources.

1. Add a **Run View** Node and turn **Multi Source** on. Choose the **Multi Source Strategy** the listing documents so branches combine the way you expect.
2. Name or suffix source Nodes so the **`sourcesuffix`** value in your deploy or refresh configuration can select one source, a list, or `ALL`, as described in **Run View Parameters** on the [Incremental Loading package][] listing.
3. Optional: enable **Filter data based on source** and **Incremental load based on Persistent table** when a single shared watermark on the persistent target should bound every branch you include in that run.

#### Grouped Incremental Load

Use **Grouped Incremental Load** when the same template should run across **Storage Mapping** locations and you track which location each row came from.

1. Add **Grouped Incremental Load** and set **Storage Mapping ID** to the comma-separated list of Storage Locations this load should visit.
2. Enter **Persistent table location**, **Persistent table name**, **Persistent table Column Name** for the column that stores the location ID, and **Grouped Incremental load column** for the date or timestamp that drives increments.
3. Toggle **Batch Load** when you need batched passes per location instead of a straight pass. Use **Joining Table Column Name** when the package's join template needs an extra key between locations.
4. **Generate join** and **Copy to Editor** when the template prompts you so the join matches your schema.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

[Databricks Base Node Types][] implements **Multi Source** directly on **Work**, **Persistent Stage**, **Dimension**, and **Fact** Nodes. Read [Multi Source Nodes][] for how **Insert**, **UNION**, and **UNION ALL** differ and how load order works.

1. Decide whether you need separate **INSERT** statements with the **Insert** strategy, duplicate removal with **UNION**, or retained duplicates with **UNION ALL**. Order sources top to bottom on the Multi Source screen so failures and retries behave the way you expect.
2. Turn **Multi Source** on in each Node that should own the combine, then wire one upstream branch per source table or view.
3. Normalize columns in upstream **Work** Nodes so every branch shares the same grain and column names before the union.
4. Add filters or branch-specific logic with [Parameters][] when only some sources run in a given Job, or when you must stamp a source system column you will use later in **MERGE** logic.
5. If joins fit the problem better than a union, keep **Multi Source** off and express the join in Node SQL instead.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

[BigQuery Base Node Types][] exposes the same **Multi Source** pattern: **UNION DISTINCT** or **UNION ALL** across branches.

1. Enable **Multi Source** on the Node that should materialize the combined data set, then attach each source branch in the order you want statements to run.
2. Align schemas across systems in **Work** Nodes first. Add a literal or computed source or `source_system` column when downstream **MERGE** or delete rules need to know which feed produced a row.
3. Use [Parameters][] in template SQL when each Job should include different sources or date windows without editing the graph.
4. Pair the combined stage with an incremental or **Advanced Deploy** **MERGE** pattern from an earlier scenario in this section so inserts, updates, and deletes stay consistent after the **UNION**.

  </TabItem>
</Tabs>

---

## Watermark Columns and Pitfalls

Pick and test the column that bounds each run. For Incremental Load, Run View, Looped Load, and Grouped Incremental field-by-field setup, use the scenario subsection that matches your load shape earlier on this page.

### Choosing a Watermark Column

Pick a column type that matches how the source updates and how wide your batch windows are.

| Type | Notes |
| ---- | ----- |
| **DATE** | Broad strokes. Good for simple daily batches. |
| **TIMESTAMP or DATETIME** | Usual choice when the source keeps it current. |
| **Monotonic version or sequence** | Use when timestamps lie or drift. |
| **Avoid** | Using only non-monotonic or very dense surrogate keys unless you accept the scan cost. |

Use **`COALESCE(updated_at, created_at)`** when new rows only fill `created_at` until the first update.

### Watermark Pitfalls

Watch for sparse values, time zones, and first-run behavior before you rely on a filter in production.

- **Sparse timestamps:** If many rows have null in the watermark column, your filter skips them. Fix upstream or normalize with `COALESCE`.
- **Time zones:** Put warehouse and source on one rule before you compare values.

<Tabs groupId="watermark-timezone">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
-- Compare in one zone
WHERE CONVERT_TIMEZONE('UTC', 'America/New_York', src_ts) > :watermark_nyc;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
-- Use Databricks time zone functions for your runtime
WHERE from_utc_timestamp(src_ts, 'America/New_York') > :watermark_nyc;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
-- Civil datetime in the zone that defines your watermark
WHERE DATETIME(src_ts, 'America/New_York') > @watermark_nyc;
```

  </TabItem>
</Tabs>

- **First empty persistent table:** Use the first-run pattern in Batch Jobs and Timestamp Column, or seed a safe minimum watermark so you don't drop history.

Hard cases, such as custom consensus or special CDC, usually mix custom SQL with [Pre-SQL and Post-SQL][] and [ref()][], [ref_no_link()][], and [ref_link()][].

---

## Handle Deletes With Incremental Loads

A high-water mark filter tells you what is new or changed. It doesn't remove rows when the source hard deletes a key unless you add explicit delete logic.

If a Node must run after a deletes staging Node but doesn't reference it in `FROM`, add a line with [ref_link()][] so the DAG enforces the correct order.

### Soft Deletes

Mark rows as inactive instead of removing them. Incremental loads treat these as updates and pass them through merge logic.

Common patterns:

- Maintain `IS_DELETED` or `DELETED_AT` in the source
- Filter active records downstream:

  - `WHERE IS_DELETED = 0`
  - `WHERE DELETED_AT IS NULL`

Apply this filtering as early as possible so downstream Nodes don't depend on implicit rules.

### Hard Deletes

Physically remove rows when required for compliance or cost.

This requires explicit handling because incremental filters alone won't catch removed records. Typical approaches include:

- Merge logic with delete conditions
- Separate delete feeds or CDC streams
- Pre-SQL or Post-SQL cleanup

Coordinate carefully with facts and aggregates to avoid orphaned keys.

### Delete Feeds and CDC

Capture deletes explicitly from the source system rather than inferring them.

Options include:

- Staging delete records separately
- Reading CDC metadata, for example operation types
- Applying deletes through merge logic or dedicated Packages

This is the most reliable pattern when available.

### Snapshot or Full Comparison

Compare a previous extract to the current one and treat missing rows as deletes.

Use when:

- No delete flag or CDC exists
- Full extracts are available

This approach is more compute-intensive but ensures completeness.

### Periodic Rebuilds

Run incremental loads between scheduled full or scoped rebuilds.

Use rebuilds to:

- Reconcile missed deletes
- Clean up drift over time

This is a practical fallback when real-time delete handling is incomplete.

### Advanced Deploy Merge Options

Fact and Dimension Advanced Deploy can apply unmatched-record and delete behavior where supported.

Use these features when:

- You want built-in merge handling
- Your Node type supports delete strategies

Refer to product documentation for supported configurations.

### Cascading Soft Deletes Through the DAG

Soft-deleted dimensions can still appear in reports if facts continue to join to them.

To prevent this:

- Filter active rows in dimension outputs, or
- Join facts with `IS_DELETED = 0`, or
- Propagate delete flags into facts and aggregates

Apply the rule early to avoid inconsistent downstream behavior.

### Multi-Source Delete Consensus

When multiple sources define delete status differently, define a clear policy before implementation:

- **Any-source delete**
  Delete if any source marks the record deleted, for example `MAX(is_deleted)`

- **All-sources delete**
  Delete only if all sources agree

- **Priority source**
  One system determines delete status

These are custom patterns implemented in staging or merge logic, not default behavior.

### Multi-Source Targets and Deletes

- **Per-source ownership**
  Apply deletes only to rows owned by that source

- **Conformed keys and tombstones**
  Prefer a single shared key and unified delete logic

- **Partial runs with `sourcesuffix`**
  Skipped sources retain their rows until processed again

- **Orphan cleanup**
  Retired sources may require one-time cleanup

## Custom Multi-Source State

Optionally, when sources refresh on different schedules, you can store per-source rows in a small state table for last watermark, last load time, and status. After a failure, move only successful sources forward so the next run retries stale sources without losing good progress.

### Resilience

- **All-or-nothing** - One transaction across all sources. Simplest consistency, but one failure can block everything.
- **Per source** - Separate commits. Sources reach eventual consistency. Often easier to run in production.

```sql
-- Watermark for one source name
SELECT COALESCE(MAX(last_watermark), TIMESTAMP '1900-01-01')
FROM incremental_source_state
WHERE source_name = :source_name;
```

## Edge Cases and Gotchas

These issues often appear after the design is working for the happy path.

- **Truncate Before upstream** - If you truncate staging in a way that wipes the persistent incremental target, you can reset the watermark and trigger reloads or duplicates. Keep persistent targets away from volatile truncates unless you want a rebuild.
- **Identity or sequence gaps** - If surrogate keys reset after you drop and recreate tables, facts can point at wrong dimensions. Prefer stable keys or the recovery steps your warehouse documents.
- **NULL keys or NULL hashes** - Bad keys make merges unpredictable. Fix keys upstream or drop bad rows on purpose.
- **Multi-source partial failure** - If one branch of a UNION fails mid-run, you can end up with a mixed state. Prefer the Insert strategy with multiple INSERT statements on Multi Source, split Nodes by source, or use transactions your platform supports.
- **Soft deletes hidden in facts** - Join to active dimension versions or carry delete flags on purpose. See [Cascading Soft Deletes Through the DAG](#cascading-soft-deletes-through-the-dag).

## Troubleshooting

Use this section when merges fail, watermarks look wrong, or Pre-SQL and Post-SQL behave unexpectedly.

### Non-Deterministic Merge

Duplicate merge keys, unstable row order, or ties on the watermark can cause errors or double updates.

- **Checks:** Run `GROUP BY business_key HAVING COUNT(*) > 1` on staging. Remove duplicates upstream. Strengthen the business key.

### Watermark Appears Stuck

Check that the persistent table still shows the `MAX(incremental_column)` you expect, the filter uses the same column and time zone, and no stray truncate cleared history.

### Statement Count or Stage Separation

Give each Pre-SQL or Post-SQL stage one clear job when the platform allows one statement per phase, or split work across stages in the graph when supported.

## Performance and Cost

Tune warehouse compute and batch sizing so incremental Jobs finish inside your SLA without over-scanning.

- On Snowflake, use clustering keys or query pruning on partition-like columns. Snowflake isn't like OLTP B-tree indexes. Keep filters selective.
- Batch sizing: Bigger windows mean fewer Jobs. Smaller windows mean fresher data. Balance SLA and spend.
- **Looped Load** - Use it when one full scan won't finish in your window.

## Monitoring Ideas

For each source or job, track:

- Delay between max timestamp in source and max timestamp loaded
- Job failures and last success time
- Row counts and error tables if you quarantine bad rows

## What's Next?

- [Incremental loading tutorial][], step-by-step
- [Incremental Loading package][], Snowflake Marketplace reference
- [Incremental Nodes][], Databricks package
- [Snowflake Base Node Types][], Marketplace reference
- [Snowflake Base Node Types Advanced Deploy][], merge and load strategies
- [Databricks Base Node Types][], Work through Fact Node types
- [Databricks Lakeflow Declarative Pipelines][]
- [Databricks External Data Package][]
- [Databricks Materialized View][]
- [BigQuery Base Node Types][]
- [BigQuery Base Node Types Advanced Deploy][]
- [Multi Source Nodes][]
- [Snowflake Streams and Tasks][]
- [Deleting Records from Fact and Dimension Nodes][]
- [Pre-SQL and Post-SQL][]
- [ref() and ref_no_link()][]
- [Surrogate Keys][]
- [SCD Type 2 change detection][]
- [Incremental processing strategies][]
- [Parameters][], deploy and refresh overrides

---

[Incremental loading tutorial]: /docs/guides/using-incremental-nodes
[Incremental Loading package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading
[Snowflake Base Node Types]: /docs/marketplace/package/coalesce_snowflake_base-node-types
[Snowflake Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy
[Databricks Base Node Types]: /docs/marketplace/package/coalesce_databricks_base-node-types
[Incremental Nodes]: /docs/marketplace/package/coalesce_databricks_incremental-nodes
[Databricks Lakeflow Declarative Pipelines]: /docs/marketplace/package/coalesce_databricks_lakeflow-declarative-pipelines
[Databricks External Data Package]: /docs/marketplace/package/coalesce_databricks_external-data-package
[Databricks Materialized View]: /docs/marketplace/package/coalesce_databricks_materialized-view
[BigQuery Base Node Types]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types
[BigQuery Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types-advanced-deploy
[Multi Source Nodes]: /docs/build-your-pipeline/multisource-nodes
[Snowflake Streams and Tasks]: /docs/marketplace/package/coalesce_snowflake_streams-and-tasks
[Deleting Records from Fact and Dimension Nodes]: /docs/build-your-pipeline/deleting-records-fact-dim-nodes
[Pre-SQL and Post-SQL]: /docs/build-your-pipeline/pre-post-sql
[Surrogate Keys]: /docs/get-started/coalesce-fundamentals/surrogate-keys
[SCD Type 2 change detection]: /docs/build-your-pipeline/build-how-to/scd-type2-case-and-change-detection
[ref() and ref_no_link()]: /docs/reference/ref-functions
[ref_no_link()]: /docs/reference/ref-functions
[ref_link()]: /docs/reference/ref-functions
[ref()]: /docs/reference/ref-functions
[Incremental processing strategies]: https://coalesce.io/product-technology/incremental-processing-strategies/
[Parameters]: /docs/deploy-and-refresh/parameters

---

## Multi Source Nodes

Multi Source nodes represent Coalesce's implementation of SQL UNIONs.

:::info[Node Sources]

Nodes can have multiple sources, using JOINs or UNIONs (or both) to ultimately create a new table/view.

:::

## Multi Source Strategy

There are three options for how a multi source node will load source tables into the target table.

### Insert (Default Option)

This will break down the operation into multiple INSERTs. This allows individual parts to fail before moving on to others, which can improve performance. This operation will retain duplicates.

### UNION

This will make a UNION of sources and get rid of duplicates.

### UNION ALL

This will make a UNION ALL of sources and keep duplicates.

## Multi Source Node Load Order

Multi source nodes will load in the order they appear on the Multi Source node screen, top to bottom.

## Example UNION with Three Tables

Let's walk through a simple example using a multi source node to form a UNION of 3 tables. We're looking to consolidate all the crew members of the Nostromo onto one table, but they're currently scattered across three similar, but not identical tables.

<details>
    <summary>Sample Nostromo Data</summary>

    ```sql title="Sample Nostromo Data"
    CREATE OR REPLACE TABLE NOSTROMO_SRC1 (
        N_CREWKEY NUMBER(38,0),
        N_FIRSTNAME VARCHAR(50),
        N_LASTNAME VARCHAR(50),
        N_COMMENT VARCHAR(152)
    );

    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC1
    (N_CREWKEY, N_FIRSTNAME, N_LASTNAME, N_COMMENT)
    VALUES(1, 'Ellen', 'Ripley', 'Warrant Officer');
    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC1
    (N_CREWKEY, N_FIRSTNAME, N_LASTNAME, N_COMMENT)
    VALUES(2, 'Arthur', 'Dallas', 'Captain');
    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC1
    (N_CREWKEY, N_FIRSTNAME, N_LASTNAME, N_COMMENT)
    VALUES(3, 'Thomas', 'Kane', 'Executive Officer');

    CREATE OR REPLACE TABLE NOSTROMO_SRC2 (
        N_CREWKEY NUMBER(38,0),
        N_FIRSTNAME VARCHAR(50),
    N_MIDDLEINITIAL VARCHAR(1),
        N_LASTNAME VARCHAR(50),
        N_COMMENT VARCHAR(152)
    );

    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC2
    (N_CREWKEY, N_FIRSTNAME, N_MIDDLEINITIAL, N_LASTNAME, N_COMMENT)
    VALUES(4, 'Ash', NULL, NULL, 'Science Officer');
    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC2
    (N_CREWKEY, N_FIRSTNAME, N_MIDDLEINITIAL, N_LASTNAME, N_COMMENT)
    VALUES(5, 'Joan', 'M', 'Lambert', 'Navigation Officer');
    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC2
    (N_CREWKEY, N_FIRSTNAME, N_MIDDLEINITIAL, N_LASTNAME, N_COMMENT)
    VALUES(6, 'Dennis', 'M', 'Parker', 'Chief Engineer');

    CREATE OR REPLACE TABLE NOSTROMO_SRC3 (
        N_CREWKEY NUMBER(38,0),
        N_FIRSTNAME VARCHAR(50),
        N_LASTNAME VARCHAR(50),
        N_COMMENT VARCHAR(152)
    );

    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC3
    (N_CREWKEY, N_FIRSTNAME, N_LASTNAME, N_COMMENT)
    VALUES(7, 'Samuel', 'Brett', 'Engineers Mate');
    INSERT INTO <YOUR_DATABASE>.<YOUR_SCHEMA>.NOSTROMO_SRC3
    (N_CREWKEY, N_FIRSTNAME, N_LASTNAME, N_COMMENT)
    VALUES(8, 'Jones', NULL, 'Ship''s Cat');
    ```

</details>

1. Add the 3 tables using the above commands.
2. Add the 3 tables as [sources][] in Coalesce.
3. Add a **Stage Node** from `NOSTROMO_SRC1` and rename it `STG_NOSTROMO_COMBINED`
4. From within `STG_NOSTROMO_COMBINED` select **Options** and toggle **Multi Source** to on.

    

5. With multi source enabled, now inside of `STG_NOSTROMO_COMBINED` you'll see a column on the left listing all the sources for the multi source node. Inside of `STG_NOSTROMO_COMBINED`, you can edit their names by clicking on them and editing the relevant text field. Rename the one source node to just `SRC1`.

    There are two different zones for drag and drop, and they work differently. The top drop zone (blue in the image below) will attempt to map columns from the new source into those of the existing source, like in a UNION. The bottom drop zone (orange in the image below) is available in all nodes, and will add the columns as separate columns, like in a JOIN.

    

6. Add a new source to `STG_NOSTROMO_COMBINED` by clicking on the + symbol beneath `SRC1`.
7. Select and rename it `SRC2`.
8. Now we will add `NOSTROMO_SRC2` by dragging it from the **Nodes** list on the left into the blue drop zone.
9. Notice it didn't copy one of the columns, `N_MIDDLEINITIAL`, as this doesn't correspond to any from `SRC1`. Select that column specifically and drag it into the orange drop zone.
10. Move the new `N_MIDDLEINITIAL` column to order it after `N_FIRSTNAME`.
11. Open the **Join** tab for SRC2 >**Generate Join**\>**Copy to Editor**.
12. Return to the Mapping area for `STG_NOSTROMO_COMBINED`.
13. Add another source, renaming it `SRC3`.
14. Drag `NOSTROMO_SRC3` into the blue area.
15. Open the **Join** tab for `SRC3` > **Generate Join**\> **Copy to Editor**.
16. Return to the mapping area for `STG_NOSTROMO_COMBINED`.

    

17. Create and Run the `STG_NOSTROMO_COMBINED` node and you'll see all the data combined.

[sources]: /docs/setup-your-project/add-a-data-source

---

## Node Aliases

By default, Coalesce uses the source table and node name as the SQL alias when referencing data in a Node. For example:

```sql
FROM {{ ref('SAMPLE', 'ORDERS') }} "ORDERS"
```

This can make SQL expressions longer or harder to follow, especially when joining multiple tables. To simplify your SQL, you can assign a shorter alias, like `custorders` in the JOIN tab:

```sql
FROM {{ ref('SAMPLE', 'ORDERS') }} "custorders"
```

## Adding the Alias

To use the alias properly, update the Transform field for each column.

1. Open the **Mapping** tab of your Node in the Node Editor.
2. Select all the Node columns.
3. Go to **Bulk Edit** and select the Transform attribute.
4. Enter in the alias and column. For example `"custorders".{{SRC}}`.
5. Preview to make sure the change is correct.

## Apply SQL Functions

You can still use SQL functions such as `UPPER`, `TRIM`, or `CAST` with your alias. Just reference the column using the alias in the Transform field. For example:

If your alias is `custorders` and you want to uppercase `O_ORDERSTATUS`, enter this in the Transform field:

```sql
UPPER("custorders"."O_ORDERSTATUS")
```

---

## Understanding Node Run Order in Coalesce

In Coalesce, the order in which Nodes execute is determined by their dependencies. A Node will only run after all the Nodes it depends on have completed. This approach ensures that data transformations occur in the correct sequence, maintaining data integrity throughout the pipeline.

To influence or modify the execution order, you can adjust the dependencies between Nodes. This can be done by specifying upstream Nodes that a particular Node relies upon. By configuring these relationships, you control the sequence in which Nodes execute.

## Run Flow Example

Here is an example of a run and how each Node waits for the upstream Node to finish before running.

1. **Source Nodes**: These Nodes are raw input tables with no dependencies, and are the **first** to run.
   1. `CUSTOMER`
   2. `ORDERS`
   3. `PARTSUPP`
2. **Stage Node**: These Nodes will run after the Source Nodes are finished.
   1. `STG_CUSTOMER` (depends on `CUSTOMER`)
   2. `STG_PARTSUPP` (depends on `PARTSUPP`)
3. **Dimension Node**: This Node will run after `STG_CUSTOMER` is finished.
   1. `DIM_CUSTOMER` (depends on `STG_CUSTOMER`)
4. **Orders Staging**: This Node can't run until both `ORDERS`(Source Node) and `DIM_CUSTOMER`(Dimension Node) are complete.
   1. `STG_ORDERS` (depends on `ORDERS` and `DIM_CUSTOMER`)
5. **Final Fact Table**: The final Node and only run after `STG_ORDERS` is done.
   1. `FCT_ORDERS` (depends on `STG_ORDERS`)

## Ways To Control Run Order

Since runs are controlled by dependencies, you can influence the processing order or trigger specific operations by adjusting those dependencies or using features designed to manage when and how data is transformed.

### Adjusting Node Dependencies

The most direct way to alter run order is by modifying the dependencies between Nodes. By ensuring that a Node depends on another, you can control the sequence in which they execute. This is typically done by specifying upstream Nodes that a particular Node relies upon. You can set this by using [Ref Functions][].

### Marketplace Packages

We have a variety of [packages][] that can add Node Types or functionalities that affect run order.

- **Dynamic Tables Package**: This package provides tools for creating, deploying, and managing dynamic tables, which can help in automating table updates and ensuring data consistency.
- **Deferred Merge Package**: Allows for deferring certain operations, which can be strategically used to control when specific transformations occur within the pipeline.

### Creating Custom Node Types

For more granular control, you can create a [custom Node Type][].

- **Node Definition**: Specify UI elements and configurations.
- **Run Template**: Define how the node will process data, allowing for the inclusion of specific logic that can affect the data processing for that Node.

### Join Templates

[Join Templates][] can be used to define how joins are performed between Nodes. By specifying particular join conditions or sequences, you can control the order in which data from different nodes is combined.

```sql
capitalized: 'My Node Name'
short: 'MNN'
plural: 'My Node Names'
tagColor: '#FF5A5F'

joinTemplate: |
  {%- for dep in sources[0].dependencies -%}
    {%- if loop.first %} FROM {% endif -%}
    {%- if not loop.first %}LEFT JOIN {% endif -%}
    {{- ref_raw(dep.node.location.name, dep.node.name) }} {{ dep.node.name }}
    {% if not loop.first %} ON {{ sources[0].dependencies[loop.index0].node.name }}./*COLUMN*/ = {{ sources[0].dependencies[loop.index0 - 1].node.name }}./*COLUMN*/ {%- endif %}
  {% endfor -%}
  
  WHERE 1=1
```

### Using Pre-SQL and Post-SQL Scripts

You can run a custom SQL script before and after a Node runs.

- **Pre-SQL**: Executed before the main transformation, can be used to set up necessary conditions or dependencies.
- **Post-SQL**: Executed after the main transformation, useful for cleanup or triggering subsequent processes.

### Using Macros and Transforms

You can extend and customize how Nodes behave using macros and transforms:

- **Transforms**: Prebuilt or custom logic blocks applied to nodes that define how data should be shaped or filtered. You can chain or modify transforms to control how data flows through the pipeline.
- **Macros**: Reusable snippets of SQL or logic that can be injected into Nodes, transforms, or scripts. Macros are helpful for inserting consistent logic or dynamically altering behavior at runtime.

---

## What's Next?

- [Custom Node Types][]
- [Join Templates][]
- [Pre-SQL and Post-SQL][]
- [Packages][]
- [Transforms][]
- [Macros][]
- [Ref Functions][]
- [Using Subgraphs to Build Your Pipeline][]

---

[packages]: /docs/marketplace
[custom Node Type]: /docs/build-your-pipeline/user-defined-nodes/
[Custom Node Types]: /docs/build-your-pipeline/user-defined-nodes/
[Join Templates]: /docs/build-your-pipeline/user-defined-nodes/join-template
[Pre-SQL and Post-SQL]: /docs/build-your-pipeline/pre-post-sql
[Transforms]: /docs/build-your-pipeline/transforms
[Macros]: /docs/reference/macros
[Ref Functions]: /docs/reference/ref-functions
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Organizing Your DAG

A well-organized DAG is essential for building maintainable and scalable data pipelines in Coalesce. This guide will help you effectively organize your DAG using graphs, subgraphs, and filters to improve visibility and manageability.

For Subgraph naming, editing, when to add another Subgraph, and how Subgraphs work with Jobs and version control, see [Using Subgraphs to Build Your Pipeline][].

## Main Graph View

The main graph shows your entire DAG with all Nodes and connections.

## Subgraphs

Subgraphs allow you to view specific parts of your Node graph in isolation, breaking down your pipeline into smaller segments.

To create a subgraph:

1. In the Build Interface, click on Subgraph, then click the plus sign (+) to add a new subgraph.

    

2. Click on the new subgraph which will take you to an empty DAG.
3. Update the subgraph name to something descriptive.
4. You can add Nodes by click on Nodes and then dragging them into the subgraph, or right-click a Node and select Add to a Subgraph.

When you add a Node to a subgraph, you're creating a reference to the original Node. This means:

- A Node can exist in multiple subgraphs.
- Changes to a Node in one subgraph are reflected across all instances.
- Dependencies between Nodes are maintained across subgraphs.

## Adding Nodes to the Subgraph

Nodes can be added to existing subgraphs in two different ways:

- Using Add to Subgraph from the context menu.
- Dragging and dropping nodes into an existing subgraph. This is supported in Graph, Node Grid, and Column Grid views.

<Wistia id="3shka7uilu" subtitle="Creating a subgraph using a filter and drag and drop Nodes into the subgraph" />

:::info[Using Filters]

Use [filters][] in the Node view to limit the number of Nodes and make it faster to select them for the subgraph.

:::

## Change the View

You can also change the view in subgraphs by selecting an option from **View As**.

- **Graph**: Shows the visual representation of Nodes and connections.
- **Node Grid**: Displays nodes in a table format for bulk editing.
- **Column Grid**: Shows all columns across nodes for inspecting dependencies.

[filters]: /docs/reference/selector
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Parsers

## JSON Parsing

Open up any non-source node, right-click on the column, and select **Derive JSON mappings**.

**Derive JSON mappings will recursively**:

- Create a column in the mapping grid for every primitive type (string, number, boolean, and null) within the object with the appropriate transform to parse that value.
- Flatten every JSON array using a table function in the [Join Tab][].

### Setting JSON Sample Size

By default, the JSON parser captures only one record of the variant column. To scan for variation in the JSON shape across entries, the parser sample size (applies to both JSON and XML) can be increased via **User Menu → Org Settings > Preferences**. This preference is stored organization wide.

This number affects the number of rows we will attempt to scan for variances in JSON shape. Coalesce uses Snowflake's [table sample][] to construct a sample of data. This is a sample of what is queried:

```sql
SELECT /* variant_column_name */ FROM /* fully qualified table */ SAMPLE ( /* sample size */ ROWS)
```

:::warning[Sample Size Limit]

If a user enters a sample size larger than the number of rows in the data, all the rows in the data will be scanned for structure.

:::

### Data Type Mapping JSON

When the JSON parser scans for variances in JSON shape, a mapping occurs to convert JSON data types to the appropriate corresponding Snowflake data type.

| JSON Data Type  | JSON Data Type Description                    | Snowflake Data Type                            |
|-----------------|-----------------------------------------------|------------------------------------------------|
| Number          | Double-precision floating type                | Double                                         |
| String          | Double-quoted Unicode with backslash escaping | String                                         |
| Boolean         | True or False                                 | Boolean                                        |
| Array           | An ordered sequence of values                 | Flattens compound values into multiple rows    |
| Object          | An unordered collection of key:value pairs    | Explodes compound values into multiple columns |

:::info[Coalesce Versions < 5.4]

Prior to Coalesce version 5.4, JSON Numbers were mapped to Snowflake NUMBER(38,0), resulting in truncation of decimal places and converting them to integers.

:::

## XML Parsing (Beta Feature)

Open up any non-source node, right-click on the variant column, and select **Derive Mappings > From XML**.

> Snowflake
>
> Only supported for Snowflake.

**Derive XML mappings will recursively**:

- Create a column in the mapping grid for every element and every attribute within the XML with the appropriate transform to parse that value.
- Flatten any XML sibling elements using a function in the [Join Tab][]

### Setting XML Sample Size

By default, the XML parser captures only one record of the variant column. To scan for variation in the XML shape across entries, the parser sample size can be increased via **User Menu → Org Settings → Preferences**. The parser sample size selected here applies to both JSON and XML and is used org-wide.

This number affects the number of rows we will attempt to scan for variances in XML shape. Coalesce uses Snowflake's [table sample][] to construct a sample of data. This is a sample of what is queried:

```sql
SELECT /* variant_column_name */ FROM /* fully qualified table */ SAMPLE ( /* sample size */ ROWS)
```

:::warning[Sample Size Limit]

If a user enters a sample size larger than the number of rows in the data, all the rows in the data will be scanned for structure.

:::

:::info[Snowflake Data Types]

When the XML parser scans for variances in XML shape, all values in the XML are converted to Snowflake strings.

:::

## Derive Mappings Bulk Operation

You can use bulk operations to derive mappings for multiple `VARIANT` columns simultaneously from the Column Grid View.

### Requirements for Bulk Derive Mappings

All of the following conditions must be met to use the bulk derive mappings feature:

- All selected columns must have a `VARIANT` data type.
- None of the selected columns can be from Source Nodes.
- The selected columns must be from different Nodes to prevent cartesian join conditions.

### Using Bulk Derive Mappings

1. Navigate to the Column Grid view.
2. Select multiple columns that meet the requirements above using Command (Mac) or CTRL (Windows)
3. Right click and select **Derive Mappings**, then choose either:
   - From JSON
   - From XML

### Validation

The bulk derive mappings operation will validate your selection against the requirements before proceeding. If any of the conditions are not met (such as mixing VARIANT with other data types or selecting multiple columns from the same node), the operation will not be available.

[Join Tab]: /docs/build-your-pipeline/the-build-interface/join-tab
[table sample]: https://docs.snowflake.com/en/sql-reference/constructs/sample.html

---

## Pre-SQL and Post-SQL

Some Nodes come with Pre-SQL and Post-SQL boxes which allow you to run SQL either before or after the main operations.

* Pre‑SQL box lets you specify SQL that runs before the Node’s primary DML/DDL operations.
* Post‑SQL box executes after the primary operation has completed.

This guide has examples showing how pre-sql and post-sql can be used.

## Creating a Process Log and Masking Sensitive Data

Before processing supplier data from a CSV file, you want to create a log table to capture processing details. After loading data into `STG_SUPPLIER`, you update sensitive fields for suppliers with high account balances.

**How this works:**

* The Pre‑SQL step sets up a logging table to track processing details.
* The Post‑SQL step masks part of the address for high‑value suppliers.
* There must a stage in between SQL statements to run multiple.
* `ref_no_link('WORK', 'STG_SUPPLIER')` ensures the correct fully qualified table name is used without linking Nodes in the DAG.

<Tabs groupId="pre-post-sql-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
--Create a process log table.
CREATE TABLE DOCUMENTATION.LOGS.PROCESS_LOG_NINE ( 
    -- DATABASE.SCHEMA.TABLE
    LOG_ID NUMBER IDENTITY(1,1),
    PROCESS_NAME VARCHAR(100),
    START_TIME TIMESTAMP,
    END_TIME TIMESTAMP,
    STATUS VARCHAR(20),
    RECORD_COUNT NUMBER
);
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
--Create a process log table.
CREATE TABLE DOCUMENTATION.LOGS.PROCESS_LOG_NINE ( 
    LOG_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),
    PROCESS_NAME STRING,
    START_TIME TIMESTAMP,
    END_TIME TIMESTAMP,
    STATUS STRING,
    RECORD_COUNT BIGINT
);
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
--Create a process log table.
CREATE TABLE DOCUMENTATION.LOGS.PROCESS_LOG_NINE ( 
    LOG_ID INT64 GENERATED BY DEFAULT AS IDENTITY,
    PROCESS_NAME STRING,
    START_TIME TIMESTAMP,
    END_TIME TIMESTAMP,
    STATUS STRING,
    RECORD_COUNT INT64
);
```

  </TabItem>
</Tabs>

See [Identity and Surrogate Keys in Coalesce][] for how Coalesce generates identity columns on each platform.

<Tabs groupId="pre-post-sql-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
UPDATE {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
SET "S_ADDRESS" = SUBSTRING("S_ADDRESS", 1, 10) || '***'
WHERE "S_ACCTBAL" > 5000;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
SET `S_ADDRESS` = SUBSTR(`S_ADDRESS`, 1, 10) || '***'
WHERE `S_ACCTBAL` > 5000;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
SET `S_ADDRESS` = CONCAT(SUBSTR(`S_ADDRESS`, 1, 10), '***')
WHERE `S_ACCTBAL` > 5000;
```

Some `UPDATE` Post-SQL patterns fail against views or certain partitioned tables. Run the Post-SQL step in your Project before you schedule it in a Job.

  </TabItem>
</Tabs>

<Tabs groupId="pre-post-sql-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
{{stage('INSERT VALUES INTO THE LOG PROCESS TABLE')}}

INSERT INTO DOCUMENTATION.LOGS.PROCESS_LOG_NINE (PROCESS_NAME, START_TIME, END_TIME, STATUS, RECORD_COUNT)
VALUES (
    'Supplier Load Process',
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP,
    'SUCCESS',
    (SELECT COUNT(*) FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }})
);
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
{{stage('INSERT VALUES INTO THE LOG PROCESS TABLE')}}

INSERT INTO DOCUMENTATION.LOGS.PROCESS_LOG_NINE (PROCESS_NAME, START_TIME, END_TIME, STATUS, RECORD_COUNT)
VALUES (
    'Supplier Load Process',
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP,
    'SUCCESS',
    (SELECT COUNT(*) FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }})
);
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
{{stage('INSERT VALUES INTO THE LOG PROCESS TABLE')}}

INSERT INTO DOCUMENTATION.LOGS.PROCESS_LOG_NINE (PROCESS_NAME, START_TIME, END_TIME, STATUS, RECORD_COUNT)
VALUES (
    'Supplier Load Process',
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP,
    'SUCCESS',
    (SELECT COUNT(*) FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }})
);
```

  </TabItem>
</Tabs>

<figure>
  
  <figcaption>Results of running the Pre-SQL and Post-SQL</figcaption>
</figure>

## Back Up and Audit Supplier Data

You need to back up the existing data for auditing purposes. After the load, you want to identify any duplicate supplier IDs and log these duplicates for further review.

**How it works**:

* The Pre‑SQL creates or replaces a backup table by copying the current contents of `STG_SUPPLIER` from the `WORK` mapping. This backup ensures you have a snapshot of the data before any transformations occur.
* The Post‑SQL checks for an audit log table, then audits the stage data for duplicate supplier IDs. Any duplicates found are inserted into the audit log for further investigation.
* `{{ ref_no_link('WORK', 'STG_SUPPLIER') }}` makes sure the correct reference to the `STG_SUPPLIER` table without creating dependencies.

```sql title="Pre-SQL Create Audit Table"
CREATE OR REPLACE TABLE DOCUMENTATION.DEV.SUPPLIER_BACKUP AS
SELECT * FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }};
```

<Tabs groupId="pre-post-sql-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
-- Create the audit log table if it doesn't exist
CREATE TABLE IF NOT EXISTS DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (
    S_SUPPKEY NUMBER,
    S_NAME VARCHAR(100),
    DUPLICATE_COUNT NUMBER,
    AUDIT_TIME TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

{{stage('Log Duplicates')}}

-- Log duplicates found in the stage table
INSERT INTO DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (S_SUPPKEY, S_NAME, DUPLICATE_COUNT)
SELECT "S_SUPPKEY", "S_NAME", COUNT(*) AS DUPLICATE_COUNT
FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
GROUP BY "S_SUPPKEY", "S_NAME"
HAVING COUNT(*) > 1;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
-- Create the audit log table if it doesn't exist
CREATE TABLE IF NOT EXISTS DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (
    S_SUPPKEY BIGINT,
    S_NAME STRING,
    DUPLICATE_COUNT BIGINT,
    AUDIT_TIME TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

{{stage('Log Duplicates')}}

-- Log duplicates found in the stage table
INSERT INTO DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (S_SUPPKEY, S_NAME, DUPLICATE_COUNT)
SELECT `S_SUPPKEY`, `S_NAME`, COUNT(*) AS DUPLICATE_COUNT
FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
GROUP BY `S_SUPPKEY`, `S_NAME`
HAVING COUNT(*) > 1;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
-- Create the audit log table if it doesn't exist
CREATE TABLE IF NOT EXISTS DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (
    S_SUPPKEY INT64,
    S_NAME STRING,
    DUPLICATE_COUNT INT64,
    AUDIT_TIME TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

{{stage('Log Duplicates')}}

-- Log duplicates found in the stage table
INSERT INTO DOCUMENTATION.DEV.SUPPLIER_AUDIT_ONE (S_SUPPKEY, S_NAME, DUPLICATE_COUNT)
SELECT `S_SUPPKEY`, `S_NAME`, COUNT(*) AS DUPLICATE_COUNT
FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }}
GROUP BY `S_SUPPKEY`, `S_NAME`
HAVING COUNT(*) > 1;
```

  </TabItem>
</Tabs>

## Run Multiple SQL Statements

You can either run them as separate stages or as one stage. If you try to run multiple SQL statements without this, you'll get an error: `Actual statement count 2 did not match the desired statement count 1.`

```sql title="Two stages (all platforms)"
SELECT * FROM 

{{stage('SOME STAGE INFO')}}

INSERT INTO 
```

<Tabs groupId="pre-post-sql-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
BEGIN
SELECT * FROM ;
INSERT INTO ;
END;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
BEGIN
  SELECT * FROM ;
  INSERT INTO ;
END;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
BEGIN
  SELECT * FROM ;
  INSERT INTO ;
END;
```

  </TabItem>
</Tabs>

## What's Next?

* [Ref Functions][]

[Ref Functions]: /docs/reference/ref-functions
[Identity and Surrogate Keys in Coalesce]: /docs/get-started/coalesce-fundamentals/surrogate-keys

---

## How To Refresh Node Data

## Refresh SQL Pipeline Sources

To refresh the data sources you can add to your DAG, click [**Add Source**][]. Then click **Refresh**.

## Refresh Node Data

- If you've added new columns to a source node, open the node and click **Re-Sync Columns**.
- If you want to retrieve records on any node type, click **Fetch Data**.

[**Add Source**]: /docs/setup-your-project/add-a-data-source

---

## Reordering Columns in Coalesce

You can reorder columns in Coalesce Nodes using the Mapping Grid. However, there are important differences in how column ordering behaves depending on whether you're working with new Nodes or existing Nodes, and between development Workspaces and deployed Environments.

- **New Nodes**: Column order is preserved during initial deployment.
- **Existing Nodes**: Adding columns in the middle may result in different column order due to your data platforms limitations.

## Why Coalesce Doesn't Require Specific Column Order

Coalesce pipelines are designed so that column order doesn't impact functionality. We fully qualify all statements, ensuring they run correctly regardless of the underlying column sequence.

## New and Existing Node Behavior

### New Nodes (Initial Deployment)

When you deploy a new Node for the first time column order is preserved as defined in your Coalesce Node.

- **Development Workspaces**: Executes `CREATE OR REPLACE`. Column order matches Coalesce.
- **Environments**: Executes `CREATE`. Column order matches Coalesce.

### Existing Nodes (Subsequent Deployments)

When you modify an **existing Node** and redeploy:

#### Development Workspaces

- Always executes `CREATE OR REPLACE`.
- Column order always matches what you see in Coalesce.

#### Environments

- Uses imperative deployment by default.
- Adding new columns results in `CLONE, ALTER, SWAP, DROP` operations.
- New columns to the **end** of existing tables during `ALTER` operations, regardless of where you positioned them in Coalesce.

## Why Column Order Changes During Deployment

When you modify an existing Node by:

1. Adding a new column in a specific position within the Mapping Grid.
2. Deploying to an Environment.

The deployment process will add the new column to the **end** of the existing table, regardless of where you positioned it in the Coalesce Mapping Grid. This only affects existing Nodes. New Nodes will have their column order preserved during initial deployment.

This can cause issues if you have:

- Pre-SQL scripts that insert data based on column position.
- Dependencies on specific column ordinal positions.

Pre-SQL executes **before** the Node's primary DML/DDL operations. If your Pre-SQL contains INSERT statements that rely on column position rather than column names, these statements will fail or insert data into wrong columns when the actual table structure differs from what you defined in Coalesce.

## Approaches for Changing Column Order

When modifying existing Nodes and encountering column order issues there are a couple of options to take.

### Reorder in the Mapping Grid

To reorder columns in a Node's Mapping Grid:

1. **Drag and drop** the columns into your desired position within the Mapping Grid
2. Click **Create** to generate the corresponding CREATE TABLE statement

This is recommended for:

- **New Nodes**: Column order will be preserved during initial deployment to Environments.
- **Development work**: Column order will always match what you see in Coalesce.
- **When Environment column order isn't critical**: If you only need correct order in development.

### Add New Columns to the End (Recommended for Existing Nodes)

When modifying existing Nodes and encountering column order issues:

1. **Add new columns to the end** of your existing Coalesce Node
2. **Update any Pre-SQL scripts** to account for the new column position
3. This ensures the Coalesce Node matches the data platform table structure in both Workspaces and Environments

**Benefits**:

- Simplest approach.
- Maintains consistency between Coalesce and your data platform.
- No data reload required.

**Drawback**:

- Column might not be in your desired ordinal position.

### Use Transient Deployment Strategy

For ongoing automated column ordering, but it requires creating custom Nodes.

Create a custom Node with the [transient strategy][], which will `CREATE OR REPLACE` the table when deploying to an Environment.

**Benefits**:

- Maintains your desired column order.

**Drawbacks**:

- Any data loaded into the Node prior to the change will need to be reloaded.
- Requires creation of a custom Node.

### Manual Column Reordering Workaround (One-Time Fix)

To fix the column order in a table that’s already deployed to an Environment as a one-time operation, you must work outside of Coalesce and execute SQL commands directly in your data platform.

1. Rename the existing table, for example `TABLE_NAME_BACKUP`.
2. Redeploy the Node from Coalesce, which will create the table with your desired column order.
3. Run an INSERT statement to copy data from the backup table into the new table.

:::warning[Requires Dropping and Recreating]

This process requires dropping and recreating the table, which means all data must be preserved through the backup and restore process.

:::

[transient strategy]:/docs/build-your-pipeline/user-defined-nodes/deployment-strategies#transient-strategy

---

## Changing the Snowflake Warehouse

When creating a connection, the user can specify a warehouse that should be used to refresh a job. However, in some use cases, you must override the warehouse for a subset of Nodes within a DAG.

## Setting Warehouse in a Workspace

When creating a Workspace for Snowflake, you enter the Role and Warehouse during the credential setup step. This warehouse becomes the default for all operations in that workspace.

## Overriding Warehouse for Specific Nodes

You can override the warehouse for specific Nodes within a DAG using a stage to set the current warehouse. This is useful in several scenarios:

- Using larger compute resources to get a performance boost on a complex transformation.
- Keep track of costs by running certain transformations on department-specific warehouses

To override the warehouse, define a stage anywhere within a UDN template:

```sql
{{ stage('Set Warehouse') }}
USE WAREHOUSE <warehouse_name>
```

Coalesce maintains the state of which warehouse is used before and after a Node is executed. When you execute the `USE WAREHOUSE` query, this persists for the entirety of the Node run but reverts to the previous warehouse when continuing to the next Node in the DAG.

:::warning[Not supported in Development Workspaces at this time]

Override warehouse is only supported in Environments.

Although `USE WAREHOUSE` can be executed in a development workspace, Coalesce does not revert the warehouse back to the default before continuing. In addition, within a development workspace, there is a possibility that the set warehouse affects other Nodes running in parallel.

Override warehouse will be coming to Development Workspaces in a future release.

:::

## Setting Warehouse for Tasks

For task-based Nodes, you can specify the warehouse in the Scheduling Options:

- **Select Warehouse**: Enter the name of the warehouse you want the task to run on without quotes (visible when Scheduling Mode is set to Warehouse Task).

You can also use the `targetTaskWarehouse` environment parameter to specify different warehouses across environments.

---

## Sync or Import Table and Column Descriptions for Existing Nodes

When working with existing Nodes in Coalesce, you may need to sync or import table and column descriptions from your data platform. There are several ways to accomplish this depending on your workflow and requirements.

## Enable Column Resync on Node Types

You can add [column resync functionality][] to existing Node types by modifying the Node definition to include `isColumnResyncEnabled: true`. This adds a **Re-Sync Columns** button that allows you to pull in descriptions and other metadata from your data platform.

Once enabled, select the **Re-Sync Columns** button in the Node Editor to obtain updated columns and metadata from your data source. Select **OK** to apply the column changes.

## API-Based Import

You can use the set [Workspace Node endpoint][] to update the descriptions.

## AI-Generated Descriptions

Use the [Intelligent Documentation Assistant][]  to create meaningful descriptions for Node and columns based on metadata and naming patterns. You can enable this in organizational settings.

[column resync functionality]: /docs/build-your-pipeline/user-defined-nodes/node-config-options#re-sync-non-source-columns
[Workspace Node endpoint]: /docs/api/coalesce/set-workspace-node
[Intelligent Documentation Assistant]: /docs/coalesce-ai/intelligent-documentation

---

## Auto-Preview

Auto-Preview controls whether Coalesce automatically fetches and displays data in the Data Viewer after a successful Node run in a Workspace. When enabled, Coalesce runs a query such as `SELECT * LIMIT 100` against the target table and shows the results. When disabled, that query is skipped, which reduces compute costs and lets you decide when previews run. The default is set in Project Settings and can be overridden per Node during your session.

## Configuring Auto-Preview

You can set Auto-Preview at the project level for all Nodes, or override it temporarily for individual Nodes while you work.

### Project Settings (Default Behavior)

The project-level setting defines the default Auto-Preview behavior for every Node in your Project. It persists across sessions.

1. Open **Project Settings**.
2. Go to the **Preview Control** tab.
3. Toggle **Enable Auto-Preview** on or off.

When enabled, Coalesce automatically fetches a sample of the data after an individual Node run. When disabled, automatic data fetching is turned off so you can manually control when preview queries run.

### Node-Level Override (Temporary)

You can temporarily override the project default for a specific Node during your current session. This is useful when you want to preview results for one Node without changing the project-wide setting.

1. Open the **Data Viewer** panel at the bottom of the Workspace.
2. Find the **Auto-Preview** status indicator in the status bar.
3. Click the **menu button** (three dots) to open the settings popover.
4. Toggle Auto-Preview on or off for that Node.

:::warning[Temporary Override]

This override applies only to your current session. When you close the Workspace or start a new session, the Node reverts to the project default.

:::

## Loading Preview Data Manually

You can click **Load Preview** at any time to manually fetch a sample of the current table data without re-running the Node. Load Preview is always available regardless of your Auto-Preview setting.

## Behavior Summary

- **Single-node runs** respect your Auto-Preview setting. If it is on, data loads automatically after a successful run. If it is off, you choose when to load.
- **Multi-node runs** never auto-fetch data regardless of settings, since previewing results across many Nodes at once is typically unnecessary.
- **Node-level overrides are session-only.** They let you make quick adjustments without affecting the rest of your Project or other team members.

---

## What's Next?

- [The Build Interface][] for an overview of the Results and Data Pane
- [The Node Editor][] for Node execution modes and data preview

---

[The Build Interface]: https://docs.coalesce.io/docs/build-your-pipeline/the-build-interface
[The Node Editor]: https://docs.coalesce.io/docs/build-your-pipeline/the-build-interface/node-editor

---

## Build Settings

To access the **Build Settings**, click on the cogwheel icon towards the bottom left.

The **Build Settings** page allows you to manage:

- [Storage Locations and Storage Mappings][]
- [Development Workspaces][]
- [Environments][]
- [Macros][]
- [Node Types][]
- [Packages][]

[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[Development Workspaces]: /docs/get-started/coalesce-fundamentals/workspaces/
[Environments]: /docs/get-started/coalesce-fundamentals/environments
[Macros]: /docs/reference/macros/
[Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Packages]: /docs/marketplace/manage-packages

---

## The Build Interface

The Build Interface is where you'll spend most of your time creating nodes, building graphs, and transforming your data.

For how Mapping, the Join tab, and graph actions work together when dependencies or lineage look wrong, see [Managing Dependencies in the Node Editor][].

## DAG or Graph View

A graph represents a SQL pipeline. Each node is a logical representation and can be materialized as a table or a view in the database. Currently, you can add the following out-of-the-box types of nodes: Source, Stage, Persistent Stage, Dimension, and Fact. You can also create their own Node Types.

<figure>

<figcaption>Show the DAG or Graph View.</figcaption>
</figure>

1. **Include and exclude** - Click and drag Nodes into the filters to either include or exclude Nodes from the Graph view.
2. **Filter and dropdowns** - Add [filters][] to only show the Nodes you want to see. Change the dropdown to:
   1. **Graph** - Shows the DAG.
   2. **Node Grid** - A table style list of Nodes. Use bulk Node editor from this view to change Node Type, Deploy Enabled, or Storage Location.
   3. **Column Grid** - A table style list of all the columns in all the Nodes. You can [bulk edit][] or view [Column Lineage][]. Right-click on a column to view all available options.
3. **Node View Options**
   1. **All** - View all Nodes
   2. **Upstream** - Shows all Nodes that feed into the selected Node.
   3. **Downstream** - Shows all Nodes that depend on the selected Node.
   4. **Related** - Shows both upstream and downstream Nodes for the selected Node.
   5. **Spotlight** - Shows both upstream and downstream Nodes up to four levels deep.
4. **Mini Map** - The mini map shows a view of the entire graph and highlights your current view port so you can jump quickly to any area.
   1. The mini map reflects filters so what you see always matches your current Graph view.
   2. Colors in the mini map match Node type colors in the Graph View for quick orientation.
   3. Drag the DAG to see the mini map view port change.
5. **Results and Data Pane** - This section provides the user with feedback as to exactly what SQL was queried and the ability to preview the data results within the application. See the status of your run. Validated, Created or Data Loaded.
6. **Footer** - The footer contains the branch the Workspace is attached to, the last commit made, the data platform name, user name, user role in the data platform, user email address, and organization name.

## Build Interface Sidebar

1. **Workspace name** - The name of the workspace you're working in.
    1. Search bar - Search the node name.
    2. Click the plus sign to Add Sources(data) or Create a New Node(add an existing node type to the DAG).
2. **Categories** - Change the category to:
    1. Nodes, the default
    2. Subgraphs (see [Using Subgraphs to Build Your Pipeline][] for workflows)
    3. Jobs
3. **Other Options**
    1. The Problem Scanner - List problems with the current workspace.
    2. Git - Shows working branch, commits, and code differences.
    3. Build Settings - Opens a tab with the Build Settings.

## Browser

1. **Tabs** - Any nodes you have open including column lineage.
2. **Include and Exclude** - The filter that enables Selector syntax to be used. Drag and drop Nodes into the view or type in selector syntax.
3. **View as** - Change the way you view the Node Graph.
    1. Graph
    2. Node Grid
    3. Column Grid
4. **Close** - Close all tabs.
5. **Selector** - A tooltip showing available search syntax.
6. **Run All nodes**. You can also:
    1. Validate all Nodes.
    2. Create all Nodes.
    3. Validate create all Nodes.
7. **Node Graph Controls**:
    1. Change the Node view options.
    2. Toggle the mini map
    3. Reload or refresh the Graph positions.
    4. Zoom out
    5. Zoom in
    6. Reset the zoom level to the default.

[filters]: /docs/reference/selector/
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs
[bulk edit]: /docs/build-your-pipeline/column-bulk-editor/
[Column Lineage]: /docs/build-your-pipeline/column-lineage
[Managing Dependencies in the Node Editor]: /docs/build-your-pipeline/the-build-interface/node-editor#managing-dependencies

---

## Join Tab

The Join tab allows users to write SQL that will refine the results or include additional sources to add using Joins. Most SQL operators for your data platform are supported. The Join tab is part of the [Node Editor][].

:::info[SELECT]

Select is done by the Mapping Grid so it typically doesn't need to be included in the Join tab.

:::

Typically, a SQL SELECT statement’s clauses are evaluated in the order shown below, which can be helpful when structuring the contents of the Join tab.

1. FROM
2. WHERE
3. GROUP BY
4. HAVING
5. WINDOW
6. QUALIFY
7. ORDER BY
8. LIMIT

The Join tab builds the `FROM`, filter, and grouping portions of your Node SQL. The Mapping Grid supplies the column `SELECT` list.

<Tabs groupId="join-tab-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
FROM {{ ref('WORK', 'ORDERS') }} "orders"
QUALIFY ROW_NUMBER() OVER (PARTITION BY "orders"."O_ORDERKEY" ORDER BY "orders"."O_ORDERDATE") = 1
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY `O_ORDERKEY` ORDER BY `O_ORDERDATE`) AS rn
  FROM {{ ref('WORK', 'ORDERS') }}
) `orders`
WHERE `orders`.`rn` = 1
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY `O_ORDERKEY` ORDER BY `O_ORDERDATE`) AS rn
  FROM {{ ref('WORK', 'ORDERS') }}
) `orders`
WHERE `orders`.`rn` = 1
```

  </TabItem>
</Tabs>

## Generate Join

Generate Join opens a small modal with an automatically generated FROM statement that includes the **Source Node** used in the current node. If more than one **Source Node** was used, it will attempt to generate an INNER JOIN statement where you can configure which columns the JOIN will use. If you want to combine multiple sources without a JOIN, refer to our [Multi Source Nodes][] feature instead.

:::info[Using Ref Functions]

You must use [Ref Functions][] to refer to Nodes here, otherwise you may find unexpected behavior.

:::

### Copy to Clipboard

Copies the generated SQL statement to your operating system's clipboard.

### Copy to Editor

Copies the generated SQL statement to the text box in the Join tab. Note that this will **not** erase the current contents of the text box, and will append to the existing text instead.

## Formatting Joins

Coalesce generates SQL using the identifier quoting style for your Project platform. Match that style when you write identifiers by hand in the Join tab. [Ref Functions][] and [helper tokens][] use the same rules when they expand.

<Tabs groupId="join-tab-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
FROM {{ ref('WORK', 'ORDERS') }} "orders"
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} "customer"
  ON "orders"."O_CUSTKEY" = "customer"."C_CUSTKEY"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
FROM {{ ref('WORK', 'ORDERS') }} `orders`
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} `customer`
  ON `orders`.`O_CUSTKEY` = `customer`.`C_CUSTKEY`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
FROM {{ ref('WORK', 'ORDERS') }} `orders`
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} `customer`
  ON `orders`.`O_CUSTKEY` = `customer`.`C_CUSTKEY`
```

  </TabItem>
</Tabs>

If a Join fails with an invalid identifier error, see [Quoted Identifiers in Coalesce][] and [Error: Invalid Identifier][].

[Multi Source Nodes]: /docs/build-your-pipeline/multisource-nodes
[Ref Functions]: /docs/reference/ref-functions
[Node Editor]: /docs/build-your-pipeline/the-build-interface/node-editor
[helper tokens]: /docs/reference/helper-tokens
[Quoted Identifiers in Coalesce]: /docs/faq/build-faq-troubleshooting/double-quoted-identifiers
[Error: Invalid Identifier]: /docs/reference/error-library/invalid-identifier

---

## The Node Editor

The Node Editor can be opened by double-clicking on any node from the sidebar, from within the graph or subgraphs. For Subgraph layout, naming, and how Subgraphs relate to the main graph, see [Using Subgraphs to Build Your Pipeline][].

1. The **Mapping Grid and Join Tab** is where you'll be able to transform, edit, and parse your data. These two areas combined define not only the current node structure but what source will be used to populate the current node and the mapping of that data to each column. Some of the things you can do include:
    1. General transformations
    2. Bulk Data Editing
    3. Testing
    4. JSON and XML transformations
    5. Edit Node and column data. Use [Coalesce's AI][]  to generate your Node and column descriptions.

Organization administrators can go to **Org Settings > Preferences** and turn the features on or off for the whole organizations. These features are not enabled by default, reach out to your Coalesce account manager to sign up.

1. **Config**
    1. Node Properties - Review and edit your nodes Storage Location, Type, and other information.
    2. Options - The options available will depend on the node type. To see potential options, review [Node Config Options][] .
2. **Column Editor** \- Select a column to add transforms. You can select multiple columns to bulk edit the data.
3. **Testing**\- Test your node to assess the data quality.
4. **Results and Data Pane** - This section provides the user with feedback as to exactly what SQL was queried and the ability to preview the data results within the application.

## Node Actions

A node has several different execution modes that can be executed on a single node. They are in the **Results and Data Pane**.

### Validate SQL

Validate SQL constructs a simple `SELECT` statement combining transformations and source mapping metadata with the `JOIN` string defined in the join tab. The `SELECT` statement is wrapped with `EXPLAIN USING TEXT`. `EXPLAIN` compiles the SQL statement, but does not execute it. This removes any complexity of the underlying template and only validates the user inputted transformations and joins.

### Create

Create renders and executes all SQL stages of the [create template][] defined in the [Node Type][] definition.

### Validate Create

Validate Create renders and validates all SQL stages of the [create template][] defined in the [Node Type][] definition. However, unlike [Create][] each SQL stage is wrapped with `EXPLAIN USING TEXT`. `EXPLAIN` compiles the SQL statement, but does not execute it.

### Run

Run renders and executes all SQL stages of the [run template][] defined in the [Node Type][] definition.

### Validate Run

Validate Run renders and validates all SQL stages of the [run template][] defined in the [Node Type][] definition. However, unlike [Run][] each SQL stage is wrapped with `EXPLAIN USING TEXT`. `EXPLAIN` compiles the SQL statement, but does not execute it.

## Add Columns

### Inherit From Nodes

Right-click in the graph or sidebar with one Node selected and choose **Add Nodes**, or multi-select Nodes and choose **Join Nodes**, then pick a Node type. For the Workspace sidebar, **Add Sources**, **Create a New Node**, and graph controls, see [The Build Interface][]. To remove a Node and keep the warehouse aligned with Coalesce, use **Delete Node** as described in [Tables Dropped Outside of Coalesce][].

### Manually Add Columns

1. Drag in one or many nodes from sidebar Node Mapping View.
2. Drag in individual columns from sidebar column preview pane.

## Naming and Renaming a Node

Each data platform has it's own rules about naming Nodes.

| Data Platform | Spaces | Lowercase | Uppercase | Dashes | Underscore | Quotes |
| ------------- | ------ | --------- | --------- | ------ | ---------- | ------ |
| Snowflake     |        | ✔         | ✔         | ✔      | ✔          |        |
| Databricks    |        | ✔         |           | ✔      | ✔          |        |

When you rename a Node in Coalesce, it results in the creation of a new table in your data platform. You'll need to delete any tables directly. Renaming the Node directly in the data platform will result in a `TABLE_OR_VIEW_NOT_FOUND` in Coalesce.

After a rename, review Join tab SQL, transforms, and hard-coded names and update them to match the new Node and column names.

## Change a Nodes Source

You can change a Node’s source if the upstream logic of your pipeline changes or if you need to point to a different table or Node for input. This is useful when refactoring your DAG or correcting data dependencies.

1. Open the **Node Editor**.
2. Select all the columns in the Node and then click **Bulk Editor**.
3. Change the attribute to **Source**.
4. Then choose either **Node and Column** or **Node** depending on your situation.
5. Click **Preview** to make sure the changes are correct.

:::warning[Use Caution When Changing a Nodes Source]

- If the new source has different columns or data types, your transformations may fail or produce incorrect results. Always check the Mapping View after changing the source.
- Changing a Node’s source can affect all downstream Nodes. You may need to update joins, filters, or transformations to align with the new data structure.

:::

## Managing Dependencies

Dependencies in the DAG come from creating downstream Nodes in the graph, from column **Source** settings in the Mapping grid, and from how other Nodes are referenced on the Join tab with ref macros. For the graph and sidebar workflows, see [The Build Interface][]. For Join SQL and `ref()` usage, see the [Join Tab][] and [Ref Functions][].

:::info[Mapping and Join Tab]

Mapping stores column-level links to specific upstream Node identities. Join SQL that uses `ref()` may still look correct after you replace or refactor a Source even when Mapping or lineage needs an update. After large upstream changes, review the Mapping grid and **Bulk Editor** **Source** settings, not only the **Join Tab**.

:::

When Mapping, lineage, or Node names do not match what you expect after you change Sources or join logic, use this table.

| Issue | What to try |
| ----- | ----------- |
| **Join Tab** looks fine after replacing a Source, but Mapping, lineage, or deploy warnings look wrong | Update column **Source** values in the Mapping grid or **Bulk Editor** so they reference the current upstream Node and columns. |
| **Source** bulk edit with **Node** seems to do nothing or preview looks broken | **Node** requires an exact column name match, including case, on the upstream Node. Use **Node and Column** when names differ. See [Bulk Editing][]. |
| Graph or Subgraph omits an edge even though SQL references the table | Use `ref_link()` or `ref_no_link()` as in [Ref Functions][], or the patterns in [Fix Un-Linked References in Sub-Graphs][]. |
| After a rename, Join tab SQL or transforms still reference old names | Manually update Join tab SQL and transforms to the new Node and column names. |

[create template]: /docs/build-your-pipeline/user-defined-nodes/
[Node Type]: /docs/get-started/coalesce-fundamentals/nodes
[Create]: /docs/build-your-pipeline/the-build-interface/node-editor
[run template]: /docs/build-your-pipeline/user-defined-nodes/
[Run]: /docs/build-your-pipeline/the-build-interface/node-editor/
[Node Config Options]:/docs/build-your-pipeline/user-defined-nodes/node-config-options/
[Coalesce's AI]: /docs/coalesce-ai/
[The Build Interface]: /docs/build-your-pipeline/the-build-interface
[Tables Dropped Outside of Coalesce]: /docs/faq/build-faq-troubleshooting/tables-dropped-outside-of-coalesce
[Join Tab]: /docs/build-your-pipeline/the-build-interface/join-tab
[Ref Functions]: /docs/reference/ref-functions
[Bulk Editing]: /docs/build-your-pipeline/column-bulk-editor
[Fix Un-Linked References in Sub-Graphs]: /docs/build-your-pipeline/build-how-to/how-to-fix-un-linked-references-in-sub-graphs-in-coalesce
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Problem Scanner

Towards the bottom left of the Build interface you'll notice a small triangle icon. This is the **Problem Scanner**, which will alert you to potential problems in your project, such as lack of database connections or Git configuration. The problem scanner will only show problems with the workspace currently in use, not for the other workspaces.

---

## Column Transforms

Transforms are optional pieces of logic that can be applied to a column to either alter its data or generate new separate data.

## Supported Transformations

Coalesce supports all transformation functions supported by your data platform. Transformation logic can be authored in several ways:

- Inline by double-clicking in the **Transform** field.
- Within the **Column Editor**.
- Using [Macros][] to encapsulate logic and reduce code duplication for frequently used transformations.

By default, if no transformation is specified, a column will reference the column specified in the **Source** column of the Mapping Grid.

:::info[Common Transformations]

More transformation examples can be found in [Common Transformations][].

:::

## Formatting Data Transformations

Coalesce generates SQL using the identifier quoting style for your Project platform. When you reference columns in a Transform, match that style in any manual identifiers you add. [Helper tokens][] expand with the same quoting rules.

<Tabs groupId="transforms-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REPLACE({{SRC}}, 'Supplier#', '')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
ROUND(`forecast_hourly_metric`.`temperature`, 1)
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REPLACE(`supplier`.`s_name`, 'Supplier#', '')
```

  </TabItem>
</Tabs>

If a Transform fails with an invalid identifier error, check hand-written identifiers against [Quoted Identifiers in Coalesce][] and [Error: Invalid Identifier][].

## Source Column Behavior

The source of a column is typically displayed in the Source column of the Mapping Grid. However, sources can also be defined within transformations.

- **Source added in the Transformation field**:
  - The Source cell will be disabled.
  - On hover, more information about the source and an explanation for the cell's disabled state are shown.

    

- **Source added in transformation using an alias**:
  - The Source column will show the resolved source and the alias.
  - On hover, the alias will be displayed.

    

- **Source added in transformation and column referenced using alias and non-alias**:
  - The same source will be shown twice, with one source indicated as aliased.

    

## Best Practices

When working with column transformations, keep the following best practices in mind:

- **Data Type Matching**: Always set the data type of your column to match the expected output of your transformation.
- **Use Macros for Repeated Logic**: If you find yourself using the same transformation logic frequently, consider creating a Macro to encapsulate that logic and reduce code duplication.
- **Be Aware of Source Definitions**: Pay attention to where sources are defined (in the Source column or within transformations) to avoid conflicts and ensure your transformations behave as expected.
- **Leverage Aliases**: When working with complex transformations, using aliases can make your code more readable and maintainable.

[Macros]: /docs/reference/macros
[Common Transformations]: /docs/category/common-transformations
[Helper tokens]: /docs/reference/helper-tokens
[Quoted Identifiers in Coalesce]: /docs/faq/build-faq-troubleshooting/double-quoted-identifiers
[Error: Invalid Identifier]: /docs/reference/error-library/invalid-identifier

---

## Accessing Hydrated Metadata

In this guide you'll learn about using dot notation in Coalesce and how it works with Jinja to help you build Nodes.

## What Is Dot Notation?

Dot notation provides a way to access nested data in structured documents like YAML files. Dot notation is used for nested properties and keys. For example, if you wanted to access `zipcode`, it would b e `person.address.zipcode`.

```yaml
person:
  name: Alice
  age: 25
  address:
    city: New York
    zipcode: 10001
```

## Coalesce Hydrated Metadata Structure

Coalesce Hydrated Metadata is structured using objects and arrays.

:::info[]

These examples focus on returning the value. Depending on your use, you may need to adjust the notation.

:::

### Identifying Objects

- Use key-value pairs (key: value)
- No dashes before items
- Nested properties use indentation

```yaml title="Object"
config:
  testsEnabled: true    # No dash, just key: value
  preSQL: ""           # No dash, just key: value
  postSQL: ""          # No dash, just key: value
  myTextbox: hello     # No dash, just key: value
```

### Identifying Arrays

- Start with a dash (-)
- Items are in a list format
- Each item begins at the same indentation level

```yaml title="Array"
storageLocations:
  - database: SNOWFLAKE_SAMPLE_DATA # Starts with -
    schema: TPCH_SF100
    name: SAMPLE
  - database: TATIANA_DOCS_DYNAMIC_TABLES # Starts with -
    schema: HYDRATED
    name: WORK
  - database: TATIANA_DOCS_DYNAMIC_TABLES # Starts with -
    schema: QA
    name: DEV
```

### Identifying Mixed Objects and Arrays

The following examples show how to identify mixed data.

```yaml title="Object Containing an Array"
config:    # This is an object
  myTabularConfig:    # This is a nested object
    items:    # This contains an array
      - tabconfigdropdown: option1    # Array item 1
      - tabconfigdropdown: option2    # Array item 2
```

```yaml title="Array Containing Objects"
columns:    # This contains an array
  - id: "3aeb7039-cc9c-405c"    # First object in array
    name: "S_SUPPKEY"
    dataType: "NUMBER(38,0)"
    
  - id: "e559a0ba-fa32-4ded"    # Second object in array
    name: "S_NAME"
    dataType: "VARCHAR(25)"
```

## Using Dot Notation in Coalesce

The following section contains examples on accessing values in the Hydrated Metadata and what the values would be. This can vary based on the Node data.

### Accessing Object Properties

Object properties are accessed directly using a dot.

- Use the exact attribute name
- Add a dot between each level
- No spaces around the dots

```yaml title="Config Object"
config.testsEnabled → true
config.myTextbox → "hello"
config.myButton → true

#config object
config:
  dropdownSelectorExample: option1
  preSQL: ""
  postSQL: ""
  myDropdown: option1
  myColumnDropdown:
    id: 3aeb7039-cc9c-405c-a179-08453f651b2f
    name: S_SUPPKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  insertStrategy: INSERT
  truncateBefore: true
  testsEnabled: true
  myTextbox: hello
  myButton: true

```

### Accessing Array Items

Arrays are counted starting at 0. This means the first element in an array is 0, and the second item is, 1, and so on.

- Use [0] for the first item
- Use [1] for the second item
- Always include the brackets and number

```yaml title="Storage Location Array"
# Array access with index
storageLocations[0].database → "SNOWFLAKE_SAMPLE_DATA"
storageLocations[0].schema → "TPCH_SF100"
storageLocations[0].name → "SAMPLE"

storageLocations[1].database → "TATIANA_DOCS_DYNAMIC_TABLES"
storageLocations[1].schema → "HYDRATED"
storageLocations[1].name → "WORK"

## storage location array
storageLocations:
  - database: SNOWFLAKE_SAMPLE_DATA #0
    schema: TPCH_SF100
    name: SAMPLE
  - database: TATIANA_DOCS_DYNAMIC_TABLES #1
    schema: HYDRATED
    name: WORK
  - database: TATIANA_DOCS_DYNAMIC_TABLES #2
    schema: QA
    name: DEV
```

### Accessing Tabular Config Items

These items use a slightly different syntax to access the items because of how it's structured. Tabular config uses `get()` to access item. Get is used access a value for a given key in a dictionary.

Review [Tabular Configuration Component][].

```yaml title="Tabular Config"
config.myTabularConfig.get('items')[0].myColumnDropdown2.tests[0].description # The first myColumnDropdown2
config.myTabularConfig.get('items')[0].myColumnDropdown2.isSurrogateKey
config.myTabularConfig.get('items')[1].myColumnDropdown2.name # The second myColumnDropdown2

config:
  postSQL: ""
  myTabularConfig:
    items:
      - myColumnDropdown2:
          id: ea286ca2-d281-4bec-b0ff-669cb90d266d
          name: DIM_REGION_KEY
          dataType: NUMBER
          description: ""
          nullable: true
          defaultValue: ""
          tests:
            - name: "Null"
              description: checks for any null values
              continueOnFailure: true
              runOrder: After
              templateString: |2-

                            SELECT * FROM {{ ref(node.location.name, node.name)}} 
                            WHERE "DIM_REGION_KEY" IS NULL
          isSurrogateKey: true
        myToggleButton2: true
        tabconfigdropdown: option2
      - myColumnDropdown2:
          id: 4b07200a-2939-4b38-9cb2-f91732518a61
          name: SYSTEM_VERSION
          dataType: NUMBER
          description: ""
          nullable: true
          defaultValue: ""
          tests: []
          isSystemVersion: true
        tabconfigdropdown: option1
        myToggleButton2: true
```

### Dot Notation in Jinja Templates

Below are a few examples of using Hydrated Metadata and dot notation in Jinja templates. The format can be used in the Create or Run templates.

```sql title="Access Object Properties"
-- Check direct properties
{% if config.testsEnabled %}
    -- Test logic here
{% endif %}

-- Check nested properties
{% if node.override.create.enabled %}
    {{ node.override.create.script }}
{% endif %}
```

```sql title="Loop Through Arrays"
-- Basic array loop
{% for location in storageLocations %}
    {{ location.database }}.{{ location.schema }}
{% endfor %}

-- Nested array access
{% for source in sources %}
    {% for col in source.columns %}
        {{ col.name }}
    {% endfor %}
{% endfor %}
```

```sql title="Basic Create Table"
CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
(
    {% for col in columns %}
        -- Access column properties in a loop
        "{{ col.name }}" {{ col.dataType }}
        
        -- Check nullable property
        {%- if not col.nullable %} NOT NULL
            -- Check default value length
            {%- if col.defaultValue | length > 0 %}
                DEFAULT {{ col.defaultValue }}
            {% endif %}
        {% endif %}
        
        -- Add column description
        {%- if col.description | length > 0 %}
            COMMENT '{{ col.description | escape }}'
        {% endif %}
        
        -- Handle commas between columns
        {%- if not loop.last -%}, {% endif %}
    {% endfor %}
)
```

```sql title="Basic Create Views"
CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
(
    {% for col in columns %}
        -- Column name from array
        "{{ col.name }}"
        
        -- Optional column comment
        {%- if col.description | length > 0 %}
            COMMENT '{{ col.description | escape }}'
        {% endif %}
        
        -- Comma handling
        {%- if not loop.last -%}, {% endif %}
    {% endfor %}
)
```

## What's Next?

- Review the [Hydrated Metadata Dot Notation Reference][]

[Tabular Configuration Component]: /docs/build-your-pipeline/user-defined-nodes/node-config-options#tabular-configuration-component
[Hydrated Metadata Dot Notation Reference]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata-reference#hydrated-metadata-dot-notation

---

## Create and Run Templates

:::note[Data Platform Identifiers]

Each data platform uses different identifiers in the Create and Run Template. Make sure you are using the right one

- Snowflake uses quotes.
- Databricks uses backticks.

Examples will show each platform.

:::

## Create Template

A **Create Template** is an **optional** template that defines the creation of the table's structure, and is equivalent to DDL.

<details>
<summary>**Snowflake Dimension Node Create Template Example**</summary>

  ```sql
  {% if node.materializationType == 'table' %}
    {{ stage('Create Dimension Table') }}

    CREATE OR REPLACE TABLE {{this}}
    (
      {% for col in columns %}
        "{{ col.name }}" {{ col.dataType }}
        {% if col.isSurrogateKey %}
              identity
        {% endif %}
        {%- if not col.nullable %} NOT NULL
          {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
        {% endif %}
        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
        {%- if not loop.last -%}, {% endif %}
      {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

  {% elif node.materializationType == 'view' %}
    {{ stage('Create Dimension View') }}

    CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
    (
      {% for col in columns %}
        "{{ col.name }}"
        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
        {%- if not loop.last -%},{% endif %}
      {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
    AS
    {% for source in sources %}

      {% if loop.first %}SELECT {% endif %}

      {% for col in source.columns %}
        {% if col.isSurrogateKey or col.isSystemUpdateDate or col.isSystemCreateDate %}
                  NULL
        {% else %}
                  {{ get_source_transform(col) }}
        {% endif %}
        AS "{{ col.name }}"
        {%- if not loop.last -%}, {% endif %}
      {% endfor %}
      {{ source.join }}

      {% if not loop.last %} UNION ALL {% endif %}
    {% endfor %}

  {% endif %}
  ```

</details>

<details>
  <summary>**Databricks Dimension Node Create Template Example**</summary>

```sql
  {#
    Copyright (c) 2023 Coalesce. All rights reserved.
This script and its associated documentation are confidential and proprietary to Coalesce.
Unauthorized reproduction, distribution, or disclosure of this material is strictly prohibited.
Coalesce permits you to copy and modify this script for the purposes of using with Coalesce but
does not permit copying or modification for any other purpose.
#}
{# == Node Type Version        : 1  == #}
{# == Node Type Name           : Dimension  == #}
{# == Node Type Description    : This node creates dimension table,view and also override create sql for view  == #}

{# Override CreateSQL for view #}

{% if node.override.create.enabled %}

    {{ node.override.create.script }}

{% elif node.materializationType == 'table' %}
    {{ stage('Create Dimension Table') }}

{# CreateSQL for Table #}

    CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
    (
        {% for col in columns %}
            `{{ col.name }}` {{ col.dataType }}
            {% if col.isSurrogateKey %}
                GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1)
            {% endif %}
            {%- if not col.nullable %} NOT NULL
            {% endif%}
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%}, {% endif %}
        {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT  '{{ node.description | escape }}'{% endif %}
  TBLPROPERTIES('delta.columnMapping.mode' = 'name')

{% elif node.materializationType == 'view' %}
    {{ stage('Create Dimension View') }}

{# CreateSQL for View #}

    CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
    (
        {% for col in columns %}
            `{{ col.name }}`
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%},{% endif %}
        {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
    AS
    {% for source in sources %}

        {% if loop.first %}SELECT {% endif %}
        {% if config.selectDistinct %}DISTINCT{% endif %}

        {% for col in source.columns %}
            {% if col.isSurrogateKey or col.isSystemUpdateDate or col.isSystemCreateDate %}
                NULL
            {% else %}
                {{ get_source_transform(col) }}
            {% endif %}
            AS `{{ col.name }}`
            {%- if not loop.last -%}, {% endif %}
        {% endfor %}
        {{ source.join }}
        {% if config.groupByAll %} GROUP BY ALL {% endif %}
        {% if not loop.last %} UNION ALL {% endif %}
    {% endfor %}

{% endif %}
```

</details>

Below is an example of a Create Template that will create an empty table with two columns.

<Tabs>
  <TabItem value="snowflake-empty-table" label="Snowflake" default>

```sql
{{ stage('Create Stage Table') }}
CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
  (
    "MY_COLUMN" NUMBER(38,0),
    "MY_COLUMN2" NUMBER(38, 0)
  )
```

  </TabItem>
  <TabItem value="databricks-empty-table" label="Databricks">

```sql
{{ stage('Create Stage Table') }}

CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
(
    `MY_COLUMN` NUMBER(38,0),
    `MY_COLUMN2` NUMBER(38,0)
)

```

  </TabItem>
</Tabs>

Below is a more complex (dynamic) example, where the columns of the source node are created using the underlying metadata for the current node. Null values, default values, and the description column are being processed accordingly to create a DDL SQL statement and resulting table.

<Tabs>
  <TabItem value="snowflake-complex-create" label="Snowflake" default>

```sql
{{ stage('Create Stage Table') }}
CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
  (
    {% for col in columns %}
      "{{ col.name }}" {{ col.dataType }}
      {%- if not col.nullable %} NOT NULL
        {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
      {% endif %}
      {%- if col.description | length > 0 %} COMMENT '{{ col.description }}'{% endif %}
      {%- if not loop.last -%}, {% endif %}
    {% endfor %}
  )
```

  </TabItem>
  <TabItem value="databricks-complex-create" label="Databricks">

```sql
{{ stage('Create Stage Table') }}
CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
(
  {% for col in columns %}
    `{{ col.name }}` {{ col.dataType }}
    {%- if not col.nullable %} NOT NULL
      {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
    {% endif %}
    {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
    {%- if not loop.last -%}, {% endif %}
  {% endfor %}
)
```

  </TabItem>

</Tabs>

## Run Template

A **Run Template** is a **optional** template that defines how the table will be populated with data and is equivalent to DML.

<details>
<summary>**Snowflake Dimension Node Run Template Example**</summary>

  ```sql

  {% set is_type_2 = columns | selectattr("isChangeTracking") | list | length > 0 %}

      {% for test in node.tests if config.testsEnabled %}
          {% if test.runOrder == 'Before' %}
              {{ test_stage(test.name, test.continueOnFailure) }}
              {{ test.templateString }}
          {% endif %}
      {% endfor %}

  {% if node.materializationType == 'table' %}

    {% if config.preSQL %}
      {{ stage('Pre-SQL') }}
      {{ config.preSQL }}
    {% endif %}
    
      {% if is_type_2 %}

          {% for source in sources %}
              {{ stage('MERGE ' + source.name | string) }}
              MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
              USING (
              /* New Rows That Don't Exist */
              SELECT
              {% for col in source.columns if not col.isSurrogateKey %}
                  {% if col.isSystemVersion %}
                      1
                  {% elif col.isSystemCurrentFlag %}
                      'Y'
                  {% else %}
                    {{ get_source_transform(col) }}
                  {% endif %}
                  AS "{{ col.name }}",
              {% endfor %}
                  'INSERT_INITAL_VERSION_ROWS' AS "DML_OPERATION"
              {{ source.join }}
              LEFT JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                      {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
              {% endfor %}
              WHERE
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  "DIM"."{{ col.name }}" IS NULL
              {% endfor %}
              UNION ALL
              /* New Row Needing To Be Inserted Due To Type-2 Column Changes */
              SELECT
              {% for col in source.columns if not col.isSurrogateKey %}
                  {% if col.isSystemVersion %}
                      "DIM"."{{ col.name }}" + 1
                  {% elif col.isSystemCurrentFlag %}
                      'Y'
                  {% else %}
                    {{ get_source_transform(col) }}
                  {% endif %}
                  AS "{{ col.name }}",
              {% endfor %}
                  'INSERT_NEW_VERSION_ROWS' AS "DML_OPERATION"
              {{ source.join }}
              INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
              {% endfor %}
              WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
              {% for col in source.columns if (col.isChangeTracking == true) %}
                  {% if loop.first %}
                      AND (
                  {% else %}
                      OR
                  {% endif %}
                  ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                  {% if loop.last %}
                      )
                  {% endif %}
              {% endfor %}
              UNION ALL
              /* Rows Needing To Be Expired Due To Type-2 Column Changes
              In this case, only two columns are merged (version and end date) */
              SELECT
              {%- for col in source.columns if not col.isSurrogateKey %}
                  {% if col.isSystemEndDate %}
                      DATEADD(MILLISECONDS, -1, CAST(CURRENT_TIMESTAMP AS TIMESTAMP))
                  {% elif col.isSystemCurrentFlag %}
                      'N'
                  {% else %}
                      "DIM"."{{ col.name }}"
                  {% endif %}
                  AS "{{ col.name }}",
              {% endfor -%}
                  'update_expired_version_rows' AS "DML_OPERATION"
              {{ source.join }}
              INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
              {% endfor %}
              WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
              {% for col in source.columns if (col.isChangeTracking == true) %}
                  {% if loop.first %}
                      AND (
                  {% else %}
                      OR
                  {% endif %}
                  ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                  {% if loop.last %}
                      )
                  {% endif %}
              {% endfor %}
              {# The if-block below avoids unnecessary updates when no type 2 column changes are present #}
              {% if source.columns 
                  | rejectattr('isSurrogateKey')
                  | rejectattr('isBusinessKey')
                  | rejectattr('isChangeTracking')
                  | rejectattr('isSystemVersion')
                  | rejectattr('isSystemCurrentFlag')
                  | rejectattr('isSystemStartDate')
                  | rejectattr('isSystemEndDate')
                  | rejectattr('isSystemCreateDate')
                  | rejectattr('isSystemUpdateDate') 
                  | list | length == 0 
              %}
                  {# Skip Section #}
              {% else %}
                UNION ALL
                /* Rows Needing To Be Updated Due To Changes To Non-Type-2 columns
                This case merges only when there are changes in non-type-2 column updates, but no changes in type-2 columns*/
                SELECT
                {%- for col in source.columns if not col.isSurrogateKey %}
                    {% if col.isSystemVersion or col.isSystemCreateDate or col.isSystemStartDate or col.isSystemEndDate %}
                        "DIM"."{{ col.name }}"
                    {% elif col.isSystemCurrentFlag %}
                        'Y'
                    {% else %}
                        {{ get_source_transform(col) }}
                    {% endif %}
                    AS "{{ col.name }}",
                {% endfor -%}
                    'UPDATE_NON_TYPE2_ROWS' AS "DML_OPERATION"
                {{ source.join }}
                INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                {% for col in source.columns if col.isBusinessKey -%}
                    {% if not loop.first %}
                        AND
                    {% endif %}
                    {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                {% endfor %}
                WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                AND (
                {% for col in source.columns if (col.isChangeTracking) -%}
                    {% if not loop.first %}
                        AND
                    {% endif %}
                    {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                {% endfor %} )
                {% for col in source.columns if not (   col.isBusinessKey or
                                                        col.isChangeTracking or
                                                        col.isSurrogateKey or
                                                        col.isSystemVersion or
                                                        col.isSystemCurrentFlag or
                                                        col.isSystemStartDate or
                                                        col.isSystemEndDate or
                                                        col.isSystemUpdateDate or
                                                        col.isSystemCreateDate) -%}
                    {% if loop.first %}
                        AND (
                    {% endif %}
                    {% if not loop.first %}
                        OR
                    {% endif %}
                    NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**')
                    {% if loop.last %}
                        )
                    {% endif %}
                {% endfor %}
              {% endif %}
          ) AS "SRC"
          ON
          {% for col in source.columns if col.isBusinessKey -%}
              {% if not loop.first %}
                  AND
              {% endif %}
              "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
          {% endfor %}
          AND "TGT"."{{ get_value_by_column_attribute("isSystemVersion") }}" = "SRC"."{{ get_value_by_column_attribute("isSystemVersion") }}"
          WHEN MATCHED THEN UPDATE SET
          {%- for col in source.columns if not (col.isBusinessKey or col.isSurrogateKey or col.isSystemCreateDate) %}
              "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
              {% if not loop.last %}, {% endif %}
          {% endfor -%}
          WHEN NOT MATCHED THEN INSERT (
          {%- for col in source.columns if not col.isSurrogateKey %}
              "{{ col.name }}"
              {% if not loop.last %}, {% endif %}
          {% endfor -%}
          )
          VALUES (
          {%- for col in source.columns if not col.isSurrogateKey %}
              "SRC"."{{ col.name }}"
              {% if not loop.last %}, {% endif %}
          {% endfor -%}
          )

      {% endfor %}

      {% else %}
          {% for source in sources %}
              {{ stage('MERGE ' + source.name | string ) }}
              MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
              USING (
                  SELECT
                  {% for col in source.columns if not col.isSurrogateKey %}
                      {% if col.isSystemVersion %}
                        1
                      {% elif col.isSystemCurrentFlag %}
                        'Y'
                      {% else %}
                          {{ get_source_transform(col) }}
                      {% endif %}
                      AS "{{ col.name }}"
                      {%- if not loop.last %}, {% endif %}
                  {% endfor %}
                  {{ source.join }})
                  AS "SRC"
              ON
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  "SRC"."{{ col.name }}" = "TGT"."{{ col.name }}"
              {% endfor %}
              WHEN MATCHED
              {% for col in source.columns if not (   col.isBusinessKey or
                                                      col.isSurrogateKey or
                                                      col.isSystemVersion or
                                                      col.isSystemCurrentFlag or
                                                      col.isSystemStartDate or
                                                      col.isSystemEndDate or
                                                      col.isSystemUpdateDate or
                                                      col.isSystemCreateDate) %}
                  {% if loop.first %}
                      AND (
                  {% else %}
                      OR
                  {% endif %}
                  NVL( CAST("SRC"."{{ col.name }}" as STRING), '**NULL**') <> NVL( CAST("TGT"."{{ col.name }}" as STRING), '**NULL**')
                  {% if loop.last %}
                      )
                  {% endif %}
              {% endfor %}
              THEN UPDATE SET
              {%- for col in source.columns if not (  col.isBusinessKey or
                                                      col.isSurrogateKey or
                                                      col.isSystemVersion or
                                                      col.isSystemCurrentFlag or
                                                      col.isSystemStartDate or
                                                      col.isSystemEndDate or
                                                      col.isSystemCreateDate) %}
                      "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                  {% if not loop.last %}, {% endif %}
              {% endfor %}
              WHEN NOT MATCHED THEN
              INSERT (
              {%- for col in source.columns if not col.isSurrogateKey %}
                  "{{ col.name }}"
                  {% if not loop.last %}, {% endif %}
              {% endfor -%}
              )
              VALUES (
              {%- for col in source.columns if not col.isSurrogateKey %}
                  "SRC"."{{ col.name }}"
                  {% if not loop.last %}, {% endif %}
              {% endfor -%}
              )
          {% endfor %}
      {% endif %}
    
    {% if config.postSQL %}
      {{ stage('Post-SQL') }}
      {{ config.postSQL }}
    {% endif %}
  {% endif %}

  {% if config.testsEnabled %}
    {% for test in node.tests %}
      {% if test.runOrder == 'After' %}
        {{ test_stage(test.name, test.continueOnFailure) }}
        {{ test.templateString }}
          {% endif %}
    {% endfor %}

    {% for column in columns %}
      {% for test in column.tests %}
        {{ test_stage(column.name + ": " + test.name) }}
        {{ test.templateString }}
      {% endfor %}
    {% endfor %}
  {% endif %}
  ```

</details>

<details>
  <summary>**Databricks Dimension Node Run Template Example**</summary>

```sql
{#
    Copyright (c) 2023 Coalesce. All rights reserved.
This script and its associated documentation are confidential and proprietary to Coalesce.
Unauthorized reproduction, distribution, or disclosure of this material is strictly prohibited.
Coalesce permits you to copy and modify this script for the purposes of using with Coalesce but
does not permit copying or modification for any other purpose.  
#}
{# == Node Type Version        : 1  == #}
{# == Node Type Name           : Dimension  == #}
{# == Node Type Description    :This node loads data into work table using config options distinct,groupby all,order by ,multi-source  == #}

{# == Variable check to identify type of dimension == #}

{% set is_type_2 = columns | selectattr("isChangeTracking") | list | length > 0 %}

{# == To run data quality tests before data insertion == #}

    {% for test in node.tests if config.testsEnabled %}
        {% if test.runOrder == 'Before' %}
            {{ test_stage(test.name, test.continueOnFailure) }}
            {{ test.templateString }}
        {% endif %}
    {% endfor %}

{% if node.materializationType == 'table' %}

{# == Queries to be executed before data insertion  == #}

    {% if config.preSQL %}            
        {{ stage('Pre-SQL') }}
        {{ config.preSQL }}
    {% endif %}

 {# == Truncate data before data insertion  == #}

    {% if config.truncateBefore %}
        {{ stage('Truncate Dimension Table') }}
        TRUNCATE TABLE {{ ref_no_link(node.location.name, node.name) }}
    {% endif %}
    
    {% if is_type_2 %}
         
         {# SCD-Type 2 Dimension == #}
        
            {{ stage('MERGE ' + ' Sources' | string) }}
            MERGE INTO {{ ref_no_link(node.location.name, node.name) }} `TGT`
            USING (
      

        {% for source in sources %}

           {% set joinclause = source.join %}
           
            /* New Rows That Don't Exist */
            (SELECT
            {% if config.selectDistinct %}
            DISTINCT
            {% endif %}
            {% for col in source.columns if not col.isSurrogateKey %}
                {% if col.isSystemVersion %}
                    1
                {% elif col.isSystemCurrentFlag %}
                    'Y'
                {% else %}
                   {{ get_source_transform(col) }}
                {% endif %}
                AS `{{ col.name }}`,
            {% endfor %}
                'INSERT_INITIAL_VERSION_ROWS' AS `DML_OPERATION`
            {{  get_clause(joinclause,'from')  }}
            LEFT JOIN {{ ref_no_link(node.location.name, node.name) }} `DIM` ON
            {% for col in source.columns if col.isBusinessKey -%}
                {% if not loop.first %}
                    AND
                {% endif %}
                    {{ get_source_transform(col) }} = `DIM`.`{{ col.name }}`
            {% endfor %}
            WHERE
            {% for col in source.columns if col.isBusinessKey -%}
                {% if not loop.first %}
                    AND
                {% endif %}
                `DIM`.`{{ col.name }}` IS NULL
            {% endfor %}
            {{ get_clause(joinclause) }}
            {% if config.groupByAll %}
            GROUP BY ALL
            {% endif %}
            {{ sortorder_by_colv() }}
            )

            UNION ALL
            /* New Row Needing To Be Inserted Due To Type-2 Column Changes */

            (SELECT
            {% if config.selectDistinct %}
            DISTINCT
            {% endif %}
            {% for col in source.columns if not col.isSurrogateKey %}
                {% if col.isSystemVersion %}
                    `DIM`.`{{ col.name }}` + 1
                {% elif col.isSystemCurrentFlag %}
                    'Y'
                {% else %}
                   {{ get_source_transform(col) }}
                {% endif %}
                AS `{{ col.name }}`,
            {% endfor %}
                'INSERT_NEW_VERSION_ROWS' AS `DML_OPERATION`
            {{  get_clause(joinclause,'from')  }}
            INNER JOIN {{ ref_no_link(node.location.name, node.name) }} `DIM` ON
            {% for col in source.columns if col.isBusinessKey -%}
                {% if not loop.first %}
                    AND
                {% endif %}
                {{ get_source_transform(col) }} = `DIM`.`{{ col.name }}`
            {% endfor %}
            WHERE `DIM`.`{{ get_value_by_column_attribute("isSystemCurrentFlag") }}` = 'Y'
            {% for col in source.columns if (col.isChangeTracking == true) %}
                {% if loop.first %}
                    AND (
                {% else %}
                    OR
                {% endif %}
                ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST(`DIM`.`{{ col.name }}` as STRING), '**NULL**') )
                {% if loop.last %}
                    )
                {% endif %}
            {% endfor %}
            {{  get_clause(joinclause)  }}
            {% if config.groupByAll %}
            GROUP BY ALL
            {% endif %}
            {{ sortorder_by_colv() }}
            )
            UNION ALL
            /* Rows Needing To Be Expired Due To Type-2 Column Changes
            In this case, only two columns are merged (version and end date) */
            
            (SELECT
            {% if config.selectDistinct %}
            DISTINCT
            {% endif %}
            {%- for col in source.columns if not col.isSurrogateKey %}
                {% if col.isSystemEndDate %}
                    DATEADD(SECOND,-0.001,CAST(CURRENT_TIMESTAMP AS TIMESTAMP))
                {% elif col.isSystemCurrentFlag %}
                    'N'
                {% else %}
                    `DIM`.`{{ col.name }}`
                {% endif %}
                AS `{{ col.name }}`,
            {% endfor -%}
                'update_expired_version_rows' AS `DML_OPERATION`
            {{  get_clause(joinclause,'from')  }}
            INNER JOIN {{ ref_no_link(node.location.name, node.name) }} `DIM` ON
            {% for col in source.columns if col.isBusinessKey -%}
                {% if not loop.first %}
                    AND
                {% endif %}
                {{ get_source_transform(col) }} = `DIM`.`{{ col.name }}`
            {% endfor %}
            WHERE `DIM`.`{{ get_value_by_column_attribute("isSystemCurrentFlag") }}` = 'Y'
            {% for col in source.columns if (col.isChangeTracking == true) %}
                {% if loop.first %}
                    AND (
                {% else %}
                    OR
                {% endif %}
                ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST(`DIM`.`{{ col.name }}` as STRING), '**NULL**') )
                {% if loop.last %}
                    )
                {% endif %}
            {% endfor %}
            {{  get_clause(joinclause)  }}
            {% if config.groupByAll %}
            GROUP BY ALL
            {% endif %}
            {{ sortorder_by_colv() }}
            )
            {# The if-block below avoids unnecessary updates when no type 2 column changes are present #}
            {% if source.columns 
                | rejectattr('isSurrogateKey')
                | rejectattr('isBusinessKey')
                | rejectattr('isChangeTracking')
                | rejectattr('isSystemVersion')
                | rejectattr('isSystemCurrentFlag')
                | rejectattr('isSystemStartDate')
                | rejectattr('isSystemEndDate')
                | rejectattr('isSystemCreateDate')
                | rejectattr('isSystemUpdateDate') 
                | list | length == 0 
            %}
                {# Skip Section #}
            {% else %}
              UNION ALL
              /* Rows Needing To Be Updated Due To Changes To Non-Type-2 columns
              This case merges only when there are changes in non-type-2 column updates, but no changes in type-2 columns*/

              (SELECT
              {% if config.selectDistinct %}
              DISTINCT
              {% endif %}
              {%- for col in source.columns if not col.isSurrogateKey %}
                  {% if col.isSystemVersion or col.isSystemCreateDate or col.isSystemStartDate or col.isSystemEndDate %}
                      `DIM`.`{{ col.name }}`
                  {% elif col.isSystemCurrentFlag %}
                      'Y'
                  {% else %}
                      {{ get_source_transform(col) }}
                  {% endif %}
                  AS `{{ col.name }}`,
              {% endfor -%}
                  'UPDATE_NON_TYPE2_ROWS' AS `DML_OPERATION`
              {{  get_clause(joinclause,'from')  }}
              INNER JOIN {{ ref_no_link(node.location.name, node.name) }} `DIM` ON
              {% for col in source.columns if col.isBusinessKey -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  {{ get_source_transform(col) }} = `DIM`.`{{ col.name }}`
              {% endfor %}
              WHERE `DIM`.`{{ get_value_by_column_attribute("isSystemCurrentFlag") }}` = 'Y'
              AND (
              {% for col in source.columns if (col.isChangeTracking) -%}
                  {% if not loop.first %}
                      AND
                  {% endif %}
                  {{ get_source_transform(col) }} = `DIM`.`{{ col.name }}`
              {% endfor %} )
              
              {% for col in source.columns if not (   col.isBusinessKey or
                                                      col.isChangeTracking or
                                                      col.isSurrogateKey or
                                                      col.isSystemVersion or
                                                      col.isSystemCurrentFlag or
                                                      col.isSystemStartDate or
                                                      col.isSystemEndDate or
                                                      col.isSystemUpdateDate or
                                                      col.isSystemCreateDate) -%}
                  {% if loop.first %}
                      AND (
                  {% endif %}
                  {% if not loop.first %}
                      OR
                  {% endif %}
                  NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST(`DIM`.`{{ col.name }}` as STRING), '**NULL**')
                  {% if loop.last %}
                      )
                  {% endif %}
              {% endfor %}
            
               {{  get_clause(joinclause)  }}
               {% if config.groupByAll %}
                GROUP BY ALL
               {% endif %}
               {{ sortorder_by_colv() }} )      
            {% endif %}   
               {% if config.insertStrategy in ['UNION', 'UNION ALL'] and not loop.last %}
                {{config.insertStrategy}}
               {% endif %}            
        {% endfor %}  
                         
        ) AS `SRC`
        ON
        {% for col in columns if col.isBusinessKey -%}
            {% if not loop.first %}
                AND
            {% endif %}
            `TGT`.`{{ col.name }}` = `SRC`.`{{ col.name }}`
        {% endfor %}
        AND `TGT`.`{{ get_value_by_column_attribute("isSystemVersion") }}` = `SRC`.`{{ get_value_by_column_attribute("isSystemVersion") }}`
        WHEN MATCHED THEN UPDATE SET
        {%- for col in columns if not (col.isBusinessKey or col.isSurrogateKey or col.isSystemCreateDate) %}
            `TGT`.`{{ col.name }}` = `SRC`.`{{ col.name }}`
            {% if not loop.last %}, {% endif %}
        {% endfor -%}
        WHEN NOT MATCHED THEN INSERT (
        {%- for col in columns if not col.isSurrogateKey %}
            `{{ col.name }}`
            {% if not loop.last %}, {% endif %}
        {% endfor -%}
        )
        VALUES (
        {%- for col in columns if not col.isSurrogateKey %}
            `SRC`.`{{ col.name }}`
            {% if not loop.last %}, {% endif %}
        {% endfor -%}
        )

    {% else %}
        
        {# SCD-Type 1 Dimension == #}

            {{ stage('MERGE ' + ' Sources' | string) }}
            MERGE INTO {{ ref_no_link(node.location.name, node.name) }} `TGT`
            USING (

        {% for source in sources %}
               ( SELECT
                {% if config.selectDistinct %}
                DISTINCT
                {% endif %}
                {% for col in source.columns if not col.isSurrogateKey %}
                    {% if col.isSystemVersion %}
                        1
                    {% elif col.isSystemCurrentFlag %}
                        'Y'
                    {% else %}
                        {{ get_source_transform(col) }}
                    {% endif %}
                    AS `{{ col.name }}`
                    {%- if not loop.last %}, {% endif %}
                {% endfor %}
                {{ source.join }}
                 {% if config.groupByAll %}
                 GROUP BY ALL
                 {% endif %}
                 {{ sortorder_by_colv() }} )     
                 {% if config.insertStrategy in ['UNION', 'UNION ALL'] and not loop.last %}
                     {{config.insertStrategy}}
                 {% endif %}
        {% endfor %}
                 )               
                AS `SRC`             
            ON
            {% for col in columns if col.isBusinessKey -%}
                {% if not loop.first %}
                    AND
                {% endif %}
                `SRC`.`{{ col.name }}` = `TGT`.`{{ col.name }}`
            {% endfor %}
            WHEN MATCHED
            {% for col in columns if not (   col.isBusinessKey or
                                                    col.isSurrogateKey or
                                                    col.isSystemVersion or
                                                    col.isSystemCurrentFlag or
                                                    col.isSystemStartDate or
                                                    col.isSystemEndDate or
                                                    col.isSystemUpdateDate or
                                                    col.isSystemCreateDate) %}
                {% if loop.first %}
                    AND (
                {% else %}
                    OR
                {% endif %}
                NVL( CAST(`SRC`.`{{ col.name }}` as STRING), '**NULL**') <> NVL( CAST(`TGT`.`{{ col.name }}` as STRING), '**NULL**')
                {% if loop.last %}
                    )
                {% endif %}
            {% endfor %}
            THEN UPDATE SET
            {%- for col in columns if not (  col.isBusinessKey or
                                                    col.isSurrogateKey or
                                                    col.isSystemVersion or
                                                    col.isSystemCurrentFlag or
                                                    col.isSystemStartDate or
                                                    col.isSystemEndDate or
                                                    col.isSystemCreateDate) %}
                    `TGT`.`{{ col.name }}` = `SRC`.`{{ col.name }}`
                {% if not loop.last %}, {% endif %}
            {% endfor %}
            WHEN NOT MATCHED THEN
            INSERT (
            {%- for col in columns if not col.isSurrogateKey %}
                `{{ col.name }}`
                {% if not loop.last %}, {% endif %}
            {% endfor -%}
            )
            VALUES (
            {%- for col in columns if not col.isSurrogateKey %}
                `SRC`.`{{ col.name }}`
                {% if not loop.last %}, {% endif %}
            {% endfor -%}
            )
    {% endif %}

{# == Queries to be executed post data insertion  == #}

    {% if config.postSQL %}            
        {{ stage('Post-SQL') }}
        {{ config.postSQL }}
    {% endif %}
{% endif %}

{# == To run data quality tests after data insertion == #}

{% if config.testsEnabled %}
    {% for test in node.tests %}
        {% if test.runOrder == 'After' %}
            {{ test_stage(test.name, test.continueOnFailure) }}
            {{ test.templateString }}
        {% endif %}
    {% endfor %}

    {% for column in columns %}
        {% for test in column.tests %}
            {{ test_stage(column.name + ": " + test.name) }}
            {{ test.templateString }}
        {% endfor %}
    {% endfor %}
{% endif %}
```

</details>

Below is a simple (static) example **Run Template** that corresponds to the previous simple **Create Template** and adds one value to each column.

<Tabs>
  <TabItem value="snowflake-insert-table" label="Snowflake" default>

```sql
{{ stage('Insert Into Table') }}
INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
(
  "MY_COLUMN", 
  "MY_COLUMN2"
)
VALUES(
  3, 
  4
)
```

  </TabItem>
  <TabItem value="databricks-insert-table" label="Databricks">

```sql
{{ stage('Insert Into Table') }}
INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
(
  `MY_COLUMN`, 
  `MY_COLUMN2`
)
VALUES(
  3, 
  4
)
```

  </TabItem>

</Tabs>

Below is a more complex (dynamic) example, looping through all the sources, and source columns, applying transformations, and ultimately creating a DML statement.

<Tabs>
  <TabItem value="snowflake-loop-sources" label="Snowflake" default>

```sql
{% for source in sources %}
  {{ stage('Insert into Table') }}
  INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
  (
     {% for col in source.columns %}
       "{{ col.name }}"
        {%- if not loop.last -%},{% endif %}
     {% endfor %}
  )
  SELECT
  {% for col in source.columns %}
    {{ get_source_transform(col) }} AS "{{ col.name }}"
    {%- if not loop.last -%}, {% endif %}
  {% endfor %}
  {{ source.join }}
{% endfor %}
```

  </TabItem>
  <TabItem value="databricks-loop-sources" label="Databricks">

```sql
{% for source in sources %}
  {{ stage('Insert into Table') }}
  INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
  (
     {% for col in source.columns %}
       `{{ col.name }}`
        {%- if not loop.last -%},{% endif %}
     {% endfor %}
  )
  SELECT
  {% for col in source.columns %}
    {{ get_source_transform(col) }} AS `{{ col.name }}`
    {%- if not loop.last -%}, {% endif %}
  {% endfor %}
  {{ source.join }}
{% endfor %}
```

  </TabItem>
</Tabs>

---

## Deployment Strategies for Custom Node Types

Coalesce uses a `default` deployment strategy by default, which can be changed by adding `deployStrategy: transient` to a custom Node definition. If a user desires an out-of-the-box Node type with transient strategy, they can make a copy of the Node Type and edit it to change its deployment strategy. Example formatting and where to add the YAML in a UDN's definition.

```yaml Node Definition
capitalized: My New Node
short: NN
plural: My New Nodes
tagColor: '#2EB67D'

deployStrategy: transient # <-- 
```

## Default Strategy

`default` deploy strategy will be used, unless otherwise specified. When a change is made to a Node, Coalesce will attempt to CLONE and ALTER a copy of the Node, then swap/rename the cloned Node with the original Node. Notice the column's Description field was edited and the change took place over four stages.

## Transient Strategy

`transient` is an alternate strategy, and only used if specified. When a change is made to a Node, Coalesce will attempt to DROP and execute the CREATE TEMPLATE of the Node Type. Notice the same column's Description field was edited, notice there's only two stages.

---

## Hydrated Metadata Reference

## How To Use This Reference

This reference is made of three parts.

The Hydrated Metadata objects that includes definitions and examples. It's meant to show the structure of the Hydrated Metadata and provide a quick reference to look up information.

The Example of Hydrated Metadata provides two Nodes. One with one source, and another with multiple sources. They are an example of what a Node looks like when it's contains data.

The Dot Notation contains a Node with the dot notation required to return the value. Use it as a starting point.

Your Node may have different data or structure depending on the type and config options.

<!-- vale off -->

## Hydrated Metadata Object

  <Hydrated_Metadata />

## Hydrated Metadata Dot Notation

This is a Node that has the dot notation on each line as a comment to return the value. This is not representative of every possible value. See the [Hydrated Metadata Object][]. How you access each value will depend on your use, such as writing a loop.

<details>
<summary>**Hydrated Node Columns**</summary>

```yaml
columns:  # columns
  - id: ea286ca2-d281-4bec-b0ff-669cb90d266d  # columns[0].id
    name: DIM_REGION_KEY  # columns[0].name
    dataType: NUMBER  # columns[0].dataType
    description: ""  # columns[0].description
    nullable: true  # columns[0].nullable
    defaultValue: ""  # columns[0].defaultValue
    tests:  # columns[0].tests
      - name: "Null"  # columns[0].tests[0].name
        description: checks for any null values  # columns[0].tests[0].description
        continueOnFailure: true  # columns[0].tests[0].continueOnFailure
        runOrder: After  # columns[0].tests[0].runOrder
        templateString: |2-  # columns[0].tests[0].templateString
                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "DIM_REGION_KEY" IS NULL
    isSurrogateKey: true  # columns[0].isSurrogateKey
  - id: adfb932a-f209-471a-8c24-5b032f5092b0  # columns[1].id
    name: R_REGIONKEY  # columns[1].name
    dataType: NUMBER(38,0)  # columns[1].dataType
    description: ""  # columns[1].description
    nullable: false  # columns[1].nullable
    defaultValue: ""  # columns[1].defaultValue
    tests:  # columns[1].tests
      - name: "Null"  # columns[1].tests[0].name
        description: checks for any null values  # columns[1].tests[0].description
        continueOnFailure: true  # columns[1].tests[0].continueOnFailure
        runOrder: After  # columns[1].tests[0].runOrder
        templateString: |2-  # columns[1].tests[0].templateString
                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "R_REGIONKEY" IS NULL
    myColumnSelector: true  # columns[1].myColumnSelector
  - id: aea6abd5-f3dd-4d3d-92f1-4f315a106096  # columns[2].id
    name: R_NAME  # columns[2].name
    dataType: VARCHAR(25)  # columns[2].dataType
    description: ""  # columns[2].description
    nullable: false  # columns[2].nullable
    defaultValue: ""  # columns[2].defaultValue
    tests: []  # columns[2].tests
    isChangeTracking: true  # columns[2].isChangeTracking
    isBusinessKey: true  # columns[2].isBusinessKey
  - id: 76b8bc1f-4914-4062-8dd9-310aedd2912b  # columns[3].id
    name: R_COMMENT  # columns[3].name
    dataType: VARCHAR(152)  # columns[3].dataType
    description: ""  # columns[3].description
    nullable: true  # columns[3].nullable
    defaultValue: ""  # columns[3].defaultValue
    tests: []  # columns[3].tests
  - id: 4b07200a-2939-4b38-9cb2-f91732518a61  # columns[4].id
    name: SYSTEM_VERSION  # columns[4].name
    dataType: NUMBER  # columns[4].dataType
    description: ""  # columns[4].description
    nullable: true  # columns[4].nullable
    defaultValue: ""  # columns[4].defaultValue
    tests: []  # columns[4].tests
    isSystemVersion: true  # columns[4].isSystemVersion
  - id: fc72d526-2955-4c43-beb1-c0c218b0fb64  # columns[5].id
    name: SYSTEM_CURRENT_FLAG  # columns[5].name
    dataType: VARCHAR  # columns[5].dataType
    description: ""  # columns[5].description
    nullable: true  # columns[5].nullable
    defaultValue: ""  # columns[5].defaultValue
    tests: []  # columns[5].tests
    isSystemCurrentFlag: true  # columns[5].isSystemCurrentFlag
  - id: a1ddbf63-b36e-44c3-98ed-54d8f3fee3bf  # columns[6].id
    name: SYSTEM_START_DATE  # columns[6].name
    dataType: TIMESTAMP  # columns[6].dataType
    description: ""  # columns[6].description
    nullable: true  # columns[6].nullable
    defaultValue: ""  # columns[6].defaultValue
    tests: []  # columns[6].tests
    isSystemStartDate: true  # columns[6].isSystemStartDate
  - id: fec02e47-0d57-4af8-b1ef-9078c8991354  # columns[7].id
    name: SYSTEM_END_DATE  # columns[7].name
    dataType: TIMESTAMP  # columns[7].dataType
    description: ""  # columns[7].description
    nullable: true  # columns[7].nullable
    defaultValue: ""  # columns[7].defaultValue
    tests: []  # columns[7].tests
    isSystemEndDate: true  # columns[7].isSystemEndDate
  - id: 050b1e6f-2617-4366-a916-90a5af99062e  # columns[8].id
    name: SYSTEM_CREATE_DATE  # columns[8].name
    dataType: TIMESTAMP  # columns[8].dataType
    description: ""  # columns[8].description
    nullable: true  # columns[8].nullable
    defaultValue: ""  # columns[8].defaultValue
    tests: []  # columns[8].tests
    isSystemCreateDate: true  # columns[8].isSystemCreateDate
  - id: 9f8224ce-8742-472a-8831-65be08057f63  # columns[9].id
    name: SYSTEM_UPDATE_DATE  # columns[9].name
    dataType: TIMESTAMP  # columns[9].dataType
    description: ""  # columns[9].description
    nullable: true  # columns[9].nullable
    defaultValue: ""  # columns[9].defaultValue
    tests: []  # columns[9].tests
    isSystemUpdateDate: true  # columns[9].isSystemUpdateDate
```

</details>

<details>
<summary>**Hydrated Node Storage Locations**</summary>

```yaml
storageLocations:  # storageLocations
  - database: SNOWFLAKE_SAMPLE_DATA  # storageLocations[0].database
    schema: TPCH_SF100  # storageLocations[0].schema
    name: SAMPLE  # storageLocations[0].name
  - database: TATIANA_DOCS_DYNAMIC_TABLES  # storageLocations[1].database
    schema: HYDRATED  # storageLocations[1].schema
    name: WORK  # storageLocations[1].name
  - database: TATIANA_DOCS_DYNAMIC_TABLES  # storageLocations[2].database
    schema: QA  # storageLocations[2].schema
    name: DEV  # storageLocations[2].name
```

</details>

<details>
<summary>**Hydrated Node Config**</summary>

```yaml
config:  # config
  postSQL: ""  # config.postSQL
  myTabularConfig:  # config.myTabularConfig
    items:  # config.myTabularConfig.get('items')
      - myColumnDropdown2:  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']
          id: ea286ca2-d281-4bec-b0ff-669cb90d266d  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['id']
          name: DIM_REGION_KEY  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['name']
          dataType: NUMBER  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['dataType']
          description: ""  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['description']
          nullable: true  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['nullable']
          defaultValue: ""  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['defaultValue']
          tests:  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests']
            - name: "Null"  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests'][0]['name']
              description: checks for any null values  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests'][0]['description']
              continueOnFailure: true  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests'][0]['continueOnFailure']
              runOrder: After  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests'][0]['runOrder']
              templateString: |2-  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['tests'][0]['templateString']
                            SELECT * FROM {{ ref(node.location.name, node.name)}} 
                            WHERE "DIM_REGION_KEY" IS NULL
          isSurrogateKey: true  # config.myTabularConfig.get('items')[0]['myColumnDropdown2']['isSurrogateKey']
        myToggleButton2: true  # config.myTabularConfig.get('items')[0]['myToggleButton2']
        tabconfigdropdown: option2  # config.myTabularConfig.get('items')[0]['tabconfigdropdown']
      - myColumnDropdown2:  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']
          id: 4b07200a-2939-4b38-9cb2-f91732518a61  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['id']
          name: SYSTEM_VERSION  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['name']
          dataType: NUMBER  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['dataType']
          description: ""  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['description']
          nullable: true  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['nullable']
          defaultValue: ""  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['defaultValue']
          tests: []  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['tests']
          isSystemVersion: true  # config.myTabularConfig.get('items')[1]['myColumnDropdown2']['isSystemVersion']
        tabconfigdropdown: option1  # config.myTabularConfig.get('items')[0]['tabconfigdropdown']
        myToggleButton2: true  # config.myTabularConfig.get('items')[1]['myToggleButton2']
  myDropdown: option1  # config.myDropdown
  preSQL: ""  # config.preSQL
  insertStrategy: INSERT  # config.insertStrategy
  testsEnabled: true  # config.testsEnabled
  myButton: true  # config.myButton
  dropdownSelectorExample: option2  # config.dropdownSelectorExample
  myColumnDropdown:  # config.myColumnDropdown
    id: 76b8bc1f-4914-4062-8dd9-310aedd2912b  # config.myColumnDropdown.id
    name: R_COMMENT  # config.myColumnDropdown.name
    dataType: VARCHAR(152)  # config.myColumnDropdown.dataType
    description: ""  # config.myColumnDropdown.description
    nullable: true  # config.myColumnDropdown.nullable
    defaultValue: ""  # config.myColumnDropdown.defaultValue
    tests: []  # config.myColumnDropdown.tests
  myTextbox: Hello  # config.myTextbox
  dropdownSelectorExample: option1 # config.dropdownSelectorExample
```

</details>

<details>
<summary>**Hydrated Node Sources**</summary>

```yaml
sources: # sources
  - name: DIM_REGION  # sources[0].name
    columns:  # sources[0].columns
      - id: ea286ca2-d281-4bec-b0ff-669cb90d266d  # sources[0].columns[0].id
        name: DIM_REGION_KEY  # sources[0].columns[0].name
        dataType: NUMBER  # sources[0].columns[0].dataType
        description: ""  # sources[0].columns[0].description
        nullable: true  # sources[0].columns[0].nullable
        defaultValue: ""  # sources[0].columns[0].defaultValue
        tests:  # sources[0].columns[0].tests
          - name: "Null"  # sources[0].columns[0].tests[0].name
            description: checks for any null values  # sources[0].columns[0].tests[0].description
            continueOnFailure: true  # sources[0].columns[0].tests[0].continueOnFailure
            runOrder: After  # sources[0].columns[0].tests[0].runOrder
            templateString: |2-  # sources[0].columns[0].tests[0].templateString

                          SELECT * FROM {{ ref(node.location.name, node.name)}} 
                          WHERE "DIM_REGION_KEY" IS NULL
        isSurrogateKey: true  # sources[0].columns[0].isSurrogateKey
        transform: ""  # sources[0].columns[0].transform
        sourceColumns:  # sources[0].columns[0].sourceColumns
          - {}  # sources[0].columns[0].sourceColumns[0]
      - id: adfb932a-f209-471a-8c24-5b032f5092b0  # sources[0].columns[1].id
        name: R_REGIONKEY  # sources[0].columns[1].name
        dataType: NUMBER(38,0)  # sources[0].columns[1].dataType
        description: ""  # sources[0].columns[1].description
        nullable: false  # sources[0].columns[1].nullable
        defaultValue: ""  # sources[0].columns[1].defaultValue
        tests:  # sources[0].columns[1].tests
          - name: "Null"  # sources[0].columns[1].tests[0].name
            description: checks for any null values  # sources[0].columns[1].tests[0].description
            continueOnFailure: true  # sources[0].columns[1].tests[0].continueOnFailure
            runOrder: After  # sources[0].columns[1].tests[0].runOrder
            templateString: |2-  # sources[0].columns[1].tests[0].templateString

                          SELECT * FROM {{ ref(node.location.name, node.name)}} 
                          WHERE "R_REGIONKEY" IS NULL
        myColumnSelector: true  # sources[0].columns[1].myColumnSelector
        transform: ""  # sources[0].columns[1].transform
        sourceColumns:  # sources[0].columns[1].sourceColumns
          - node:  # sources[0].columns[1].sourceColumns[0].node
              id: 247d6d02-f042-4756-af5d-8034a1cb6fb2  # sources[0].columns[1].sourceColumns[0].node.id
              name: STG_REGION  # sources[0].columns[1].sourceColumns[0].node.name
              nodeType: Stage  # sources[0].columns[1].sourceColumns[0].node.nodeType
              location:  # sources[0].columns[1].sourceColumns[0].node.location
                name: WORK  # sources[0].columns[1].sourceColumns[0].node.location.name
              description: Region data as defined by TPC-H  # sources[0].columns[1].sourceColumns[0].node.description
              materializationType: table  # sources[0].columns[1].sourceColumns[0].node.materializationType
              isMultisource: false  # sources[0].columns[1].sourceColumns[0].node.isMultisource
              override:  # sources[0].columns[1].sourceColumns[0].node.override
                create:  # sources[0].columns[1].sourceColumns[0].node.override.create
                  enabled: false  # sources[0].columns[1].sourceColumns[0].node.override.create.enabled
                  script: ""  # sources[0].columns[1].sourceColumns[0].node.override.create.script
              tests: []  # sources[0].columns[1].sourceColumns[0].node.tests
            column:  # sources[0].columns[1].sourceColumns[0].column
              id: 2f63b9c9-937c-4fd2-a1ef-8fb41f596e8e  # sources[0].columns[1].sourceColumns[0].column.id
              name: R_REGIONKEY  # sources[0].columns[1].sourceColumns[0].column.name
              dataType: NUMBER(38,0)  # sources[0].columns[1].sourceColumns[0].column.dataType
              description: ""  # sources[0].columns[1].sourceColumns[0].column.description
              nullable: false  # sources[0].columns[1].sourceColumns[0].column.nullable
              defaultValue: ""  # sources[0].columns[1].sourceColumns[0].column.defaultValue
              tests: []  # sources[0].columns[1].sourceColumns[0].column.tests
      - id: aea6abd5-f3dd-4d3d-92f1-4f315a106096  # sources[0].columns[2].id
        name: R_NAME  # sources[0].columns[2].name
        dataType: VARCHAR(25)  # sources[0].columns[2].dataType
        description: ""  # sources[0].columns[2].description
        nullable: false  # sources[0].columns[2].nullable
        defaultValue: ""  # sources[0].columns[2].defaultValue
        tests: []  # sources[0].columns[2].tests
        isChangeTracking: true  # sources[0].columns[2].isChangeTracking
        isBusinessKey: true  # sources[0].columns[2].isBusinessKey
        transform: ""  # sources[0].columns[2].transform
        sourceColumns:  # sources[0].columns[2].sourceColumns
          - node:  # sources[0].columns[2].sourceColumns[0].node
              id: 247d6d02-f042-4756-af5d-8034a1cb6fb2  # sources[0].columns[2].sourceColumns[0].node.id
              name: STG_REGION  # sources[0].columns[2].sourceColumns[0].node.name
              nodeType: Stage  # sources[0].columns[2].sourceColumns[0].node.nodeType
              location:  # sources[0].columns[2].sourceColumns[0].node.location
                name: WORK  # sources[0].columns[2].sourceColumns[0].node.location.name
              description: Region data as defined by TPC-H  # sources[0].columns[2].sourceColumns[0].node.description
              materializationType: table  # sources[0].columns[2].sourceColumns[0].node.materializationType
              isMultisource: false  # sources[0].columns[2].sourceColumns[0].node.isMultisource
              override:  # sources[0].columns[2].sourceColumns[0].node.override
                create:  # sources[0].columns[2].sourceColumns[0].node.override.create
                  enabled: false  # sources[0].columns[2].sourceColumns[0].node.override.create.enabled
                  script: ""  # sources[0].columns[2].sourceColumns[0].node.override.create.script
              tests: []  # sources[0].columns[2].sourceColumns[0].node.tests
            column:  # sources[0].columns[2].sourceColumns[0].column
              id: e700d0b7-e5f1-4fd5-8c29-3d8c56124aef  # sources[0].columns[2].sourceColumns[0].column.id
              name: R_NAME  # sources[0].columns[2].sourceColumns[0].column.name
              dataType: VARCHAR(25)  # sources[0].columns[2].sourceColumns[0].column.dataType
              description: ""  # sources[0].columns[2].sourceColumns[0].column.description
              nullable: false  # sources[0].columns[2].sourceColumns[0].column.nullable
              defaultValue: ""  # sources[0].columns[2].sourceColumns[0].column.defaultValue
              tests: []  # sources[0].columns[2].sourceColumns[0].column.tests
      - id: 76b8bc1f-4914-4062-8dd9-310aedd2912b  # sources[0].columns[3].id
        name: R_COMMENT  # sources[0].columns[3].name
        dataType: VARCHAR(152)  # sources[0].columns[3].dataType
        description: ""  # sources[0].columns[3].description
        nullable: true  # sources[0].columns[3].nullable
        defaultValue: ""  # sources[0].columns[3].defaultValue
        tests: []  # sources[0].columns[3].tests
        transform: ""  # sources[0].columns[3].transform
        sourceColumns:  # sources[0].columns[3].sourceColumns
          - node:  # sources[0].columns[3].sourceColumns[0].node
              id: 247d6d02-f042-4756-af5d-8034a1cb6fb2  # sources[0].columns[3].sourceColumns[0].node.id
              name: STG_REGION  # sources[0].columns[3].sourceColumns[0].node.name
              nodeType: Stage  # sources[0].columns[3].sourceColumns[0].node.nodeType
              location:  # sources[0].columns[3].sourceColumns[0].node.location
                name: WORK  # sources[0].columns[3].sourceColumns[0].node.location.name
              description: Region data as defined by TPC-H  # sources[0].columns[3].sourceColumns[0].node.description
              materializationType: table  # sources[0].columns[3].sourceColumns[0].node.materializationType
              isMultisource: false  # sources[0].columns[3].sourceColumns[0].node.isMultisource
              override:  # sources[0].columns[3].sourceColumns[0].node.override
                create:  # sources[0].columns[3].sourceColumns[0].node.override.create
                  enabled: false  # sources[0].columns[3].sourceColumns[0].node.override.create.enabled
                  script: ""  # sources[0].columns[3].sourceColumns[0].node.override.create.script
              tests: []  # sources[0].columns[3].sourceColumns[0].node.tests
            column:  # sources[0].columns[3].sourceColumns[0].column
              id: 12f27257-4353-4dd9-86f5-cd3a74b56db2  # sources[0].columns[3].sourceColumns[0].column.id
              name: R_COMMENT  # sources[0].columns[3].sourceColumns[0].column.name
              dataType: VARCHAR(152)  # sources[0].columns[3].sourceColumns[0].column.dataType
              description: ""  # sources[0].columns[3].sourceColumns[0].column.description
              nullable: true  # sources[0].columns[3].sourceColumns[0].column.nullable
              defaultValue: ""  # sources[0].columns[3].sourceColumns[0].column.defaultValue
              tests: []  # sources[0].columns[3].sourceColumns[0].column.tests
      - id: 4b07200a-2939-4b38-9cb2-f91732518a61  # sources[0].columns[4].id
        name: SYSTEM_VERSION  # sources[0].columns[4].name
        dataType: NUMBER  # sources[0].columns[4].dataType
        description: ""  # sources[0].columns[4].description
        nullable: true  # sources[0].columns[4].nullable
        defaultValue: ""  # sources[0].columns[4].defaultValue
        tests: []  # sources[0].columns[4].tests
        isSystemVersion: true  # sources[0].columns[4].isSystemVersion
        transform: ""  # sources[0].columns[4].transform
        sourceColumns:  # sources[0].columns[4].sourceColumns
          - {}  # sources[0].columns[4].sourceColumns[0]
      - id: fc72d526-2955-4c43-beb1-c0c218b0fb64  # sources[0].columns[5].id
        name: SYSTEM_CURRENT_FLAG  # sources[0].columns[5].name
        dataType: VARCHAR  # sources[0].columns[5].dataType
        description: ""  # sources[0].columns[5].description
        nullable: true  # sources[0].columns[5].nullable
        defaultValue: ""  # sources[0].columns[5].defaultValue
        tests: []  # sources[0].columns[5].tests
        isSystemCurrentFlag: true  # sources[0].columns[5].isSystemCurrentFlag
        transform: ""  # sources[0].columns[5].transform
        sourceColumns:  # sources[0].columns[5].sourceColumns
          - {}  # sources[0].columns[5].sourceColumns[0]
      - id: a1ddbf63-b36e-44c3-98ed-54d8f3fee3bf  # sources[0].columns[6].id
        name: SYSTEM_START_DATE  # sources[0].columns[6].name
        dataType: TIMESTAMP  # sources[0].columns[6].dataType
        description: ""  # sources[0].columns[6].description
        nullable: true  # sources[0].columns[6].nullable
        defaultValue: ""  # sources[0].columns[6].defaultValue
        tests: []  # sources[0].columns[6].tests
        isSystemStartDate: true  # sources[0].columns[6].isSystemStartDate
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)  # sources[0].columns[6].transform
        sourceColumns:  # sources[0].columns[6].sourceColumns
          - {}  # sources[0].columns[6].sourceColumns[0]
      - id: fec02e47-0d57-4af8-b1ef-9078c8991354  # sources[0].columns[7].id
        name: SYSTEM_END_DATE  # sources[0].columns[7].name
        dataType: TIMESTAMP  # sources[0].columns[7].dataType
        description: ""  # sources[0].columns[7].description
        nullable: true  # sources[0].columns[7].nullable
        defaultValue: ""  # sources[0].columns[7].defaultValue
        tests: []  # sources[0].columns[7].tests
        isSystemEndDate: true  # sources[0].columns[7].isSystemEndDate
        transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)  # sources[0].columns[7].transform
        sourceColumns:  # sources[0].columns[7].sourceColumns
          - {}  # sources[0].columns[7].sourceColumns[0]
      - id: 050b1e6f-2617-4366-a916-90a5af99062e  # sources[0].columns[8].id
        name: SYSTEM_CREATE_DATE  # sources[0].columns[8].name
        dataType: TIMESTAMP  # sources[0].columns[8].dataType
        description: ""  # sources[0].columns[8].description
        nullable: true  # sources[0].columns[8].nullable
        defaultValue: ""  # sources[0].columns[8].defaultValue
        tests: []  # sources[0].columns[8].tests
        isSystemCreateDate: true  # sources[0].columns[8].isSystemCreateDate
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)  # sources[0].columns[8].transform
        sourceColumns: 
          - {}  # sources[0].columns[8].sourceColumns[0]
      - id: 9f8224ce-8742-472a-8831-65be08057f63  # sources[0].columns[9].id
        name: SYSTEM_UPDATE_DATE  # sources[0].columns[9].name
        dataType: TIMESTAMP  # sources[0].columns[9].dataType
        description: ""  # sources[0].columns[9].description
        nullable: true  # sources[0].columns[9].nullable
        defaultValue: ""  # sources[0].columns[9].defaultValue
        tests: []  # sources[0].columns[9].tests
        isSystemUpdateDate: true  # sources[0].columns[9].isSystemUpdateDate
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)  # sources[0].columns[9].transform
        sourceColumns:  # sources[0].columns[9].sourceColumns
          - {}  # sources[0].columns[9].sourceColumns[0]
    join: FROM {{ ref('WORK', 'STG_REGION') }} "STG_REGION"  # sources[0].join
    dependencies:  # sources[0].dependencies
      - node:  # sources[0].dependencies[0].node
          id: 247d6d02-f042-4756-af5d-8034a1cb6fb2  # sources[0].dependencies[0].node.id
          name: STG_REGION  # sources[0].dependencies[0].node.name
          nodeType: Stage  # sources[0].dependencies[0].node.nodeType
          location:  # sources[0].dependencies[0].node.location
            name: WORK  # sources[0].dependencies[0].node.location.name
          description: Region data as defined by TPC-H  # sources[0].dependencies[0].node.description
          materializationType: table  # sources[0].dependencies[0].node.materializationType
          isMultisource: false  # sources[0].dependencies[0].node.isMultisource
          override:  # sources[0].dependencies[0].node.override
            create:  # sources[0].dependencies[0].node.override.create
              enabled: false  # sources[0].dependencies[0].node.override.create.enabled
              script: ""  # sources[0].dependencies[0].node.override.create.script
          tests: []  # sources[0].dependencies[0].node.tests
        columns:  # sources[0].dependencies[0].columns
          - id: 2f63b9c9-937c-4fd2-a1ef-8fb41f596e8e  # sources[0].dependencies[0].columns[0].id
            name: R_REGIONKEY  # sources[0].dependencies[0].columns[0].name
            dataType: NUMBER(38,0)  # sources[0].dependencies[0].columns[0].dataType
            description: ""  # sources[0].dependencies[0].columns[0].description
            nullable: false  # sources[0].dependencies[0].columns[0].nullable
            defaultValue: ""  # sources[0].dependencies[0].columns[0].defaultValue
            tests: []  # sources[0].dependencies[0].columns[0].tests
          - id: e700d0b7-e5f1-4fd5-8c29-3d8c56124aef  # sources[0].dependencies[0].columns[1].id
            name: R_NAME  # sources[0].dependencies[0].columns[1].name
            dataType: VARCHAR(25)  # sources[0].dependencies[0].columns[1].dataType
            description: ""  # sources[0].dependencies[0].columns[1].description
            nullable: false  # sources[0].dependencies[0].columns[1].nullable
            defaultValue: ""  # sources[0].dependencies[0].columns[1].defaultValue
            tests: []  # sources[0].dependencies[0].columns[1].tests
          - id: 12f27257-4353-4dd9-86f5-cd3a74b56db2  # sources[0].dependencies[0].columns[2].id
            name: R_COMMENT  # sources[0].dependencies[0].columns[2].name
            dataType: VARCHAR(152)  # sources[0].dependencies[0].columns[2].dataType
            description: ""  # sources[0].dependencies[0].columns[2].description
            nullable: true  # sources[0].dependencies[0].columns[2].nullable
            defaultValue: ""  # sources[0].dependencies[0].columns[2].defaultValue
            tests: []  # sources[0].dependencies[0].columns[2].tests
    customSQL: ""  # sources[0].customSQL
```

</details>

<details>
<summary>**Hydrated Node Data**</summary>

```yaml
node:
  id: scratch  # node.id
  name: DIM_REGION  # node.name
  nodeType: Dimension  # node.nodeType
  location:  # node.location
    name: WORK  # node.location.name
  description: Region data as defined by TPC-H  # node.description
  materializationType: table  # node.materializationType
  isMultisource: true  # node.isMultisource
  override:  # node.override
    create:  # node.override.create
      enabled: false  # node.override.create.enabled
      script: ""  # node.override.create.script
  tests:  # node.tests
    - name: Test  # node.tests[0].name
      description: ""  # node.tests[0].description
      continueOnFailure: true  # node.tests[0].continueOnFailure
      runOrder: After  # node.tests[0].runOrder
      templateString: SELECT*FROM COALESCE  # node.tests[0].templateString
```

</details>

<details>
<summary>**Hydrated Node This and Parameters**</summary>

```yaml
this: "{{ ref_no_link(node.location.name, node.name) }}"  # this
parameters:  # parameters
  hello: goodbye  # parameters.hello
  goodMorning: goodnight  # parameters.goodMorning
```

</details>

## Example of Hydrated Metadata

This is an example of a Node with Hydrated Metadata and Node Definition. It can be used to see how information will appear in the Hydrated Metadata and how it correlates to the Node Defintion.

<details>
<summary>**Fully Hydrated Node Example**</summary>

```yaml
columns:
  - id: 4c1e4719-b72b-41fe-82f6-7b6311475fdd
    name: system_column_name1
    dataType: STRING
    description: Some description
    nullable: true
    defaultValue: hello
    tests: []
    isSystemColumnName: true
  - id: 8791b268-3a95-4ac0-9929-1a2b328c60bd
    name: L_ORDERKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests:
      - name: "Null"
        description: checks for any null values
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "L_ORDERKEY" IS NULL
      - name: Unique
        description: check if value is unique
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT "L_ORDERKEY", COUNT(*)
                      FROM {{ ref(node.location.name, node.name)}} 
                      GROUP BY "L_ORDERKEY"
                      HAVING COUNT(*) > 1
  - id: 6f73c0e2-fe2b-45bc-b181-dcaa02256b38
    name: L_PARTKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 72636ad8-6c64-4983-a222-f9ac533c4eed
    name: L_SUPPKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests:
      - name: "Null"
        description: checks for any null values
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "L_SUPPKEY" IS NULL
      - name: Unique
        description: check if value is unique
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT "L_SUPPKEY", COUNT(*)
                      FROM {{ ref(node.location.name, node.name)}} 
                      GROUP BY "L_SUPPKEY"
                      HAVING COUNT(*) > 1
  - id: 99710860-2240-42ac-bcad-d5011ce50909
    name: L_LINENUMBER
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 6dc793eb-8f7d-4f13-a315-8a1cacdf541a
    name: L_QUANTITY
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests:
      - name: "Null"
        description: checks for any null values
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "L_QUANTITY" IS NULL
  - id: 9fc64e2f-fd37-42e6-8f40-bc78a21680ed
    name: C_CUSTKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: eb70b19f-dc09-4b42-833b-c9c5d9d4897a
    name: C_NAME
    dataType: VARCHAR(25)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 4a062af3-1777-474a-b6c8-635adb540959
    name: C_ADDRESS
    dataType: VARCHAR(40)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 7dbe0df9-10b8-4c6b-8376-eae71c25632e
    name: C_NATIONKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 59c6ddb9-e8bd-4608-b011-6377c8906534
    name: C_PHONE
    dataType: VARCHAR(15)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 3d1a6245-0bef-4fa3-b2c9-273b24a0c275
    name: C_ACCTBAL
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: a4b64e99-eb5f-406b-9c5f-baddf3c99103
    name: C_MKTSEGMENT
    dataType: VARCHAR(10)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: bd37ac6b-ce51-485a-9e52-8a3e4b79475a
    name: C_COMMENT
    dataType: VARCHAR(117)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: a96cd9cb-8201-4c54-8f04-c41e46c41115
    name: L_EXTENDEDPRICE
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 7ba1fe2a-6ff6-44ce-a76c-831dc01d0b59
    name: L_DISCOUNT
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 2dc17e49-4013-445b-81e1-9ec64a25f472
    name: L_TAX
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: c2823c69-a131-4c5c-93b5-1630b1385595
    name: L_RETURNFLAG
    dataType: VARCHAR(1)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: fe7dc273-d716-45a6-92d6-d311a5355440
    name: L_LINESTATUS
    dataType: VARCHAR(1)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 54e1b35c-4676-46da-a371-7c7da356f2bb
    name: L_SHIPDATE
    dataType: DATE
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 8959011e-a95b-4673-860c-a8f48e1d6b48
    name: L_COMMITDATE
    dataType: DATE
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 591df848-a979-4cd8-8bb2-4c88204aba2d
    name: L_RECEIPTDATE
    dataType: DATE
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: f2fd738f-ca2b-4845-bb7f-4d18703ecb23
    name: L_SHIPINSTRUCT
    dataType: VARCHAR(25)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: 602fa59d-4697-4e27-b9a1-bbb2d6400e67
    name: L_SHIPMODE
    dataType: VARCHAR(10)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
  - id: ee257834-ee7f-4da5-8689-4e1b22500fad
    name: L_COMMENT
    dataType: VARCHAR(44)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
storageLocations:
  - database: SNOWFLAKE_SAMPLE_DATA
    schema: TPCH_SF100
    name: SAMPLE
  - database: TATIANA_DOCS_DYNAMIC_TABLES
    schema: HYDRATED
    name: WORK
  - database: TATIANA_DOCS_DYNAMIC_TABLES
    schema: QA
    name: DEV
config:
  testsEnabled: true
  myDropdown: option1
  insertStrategy: INSERT
  myButton: false
  preSQL: ""
  postSQL: ""
  truncateBefore: true
  dropdownSelectorExample: option1
  myTextbox: ""
  myTabularConfig:
    items:
      - myToggleButton2: false
  myColumnDropdown: null
sources:
  - name: DOCS_DIM_LINE_ITEM
    columns:
      - id: 4c1e4719-b72b-41fe-82f6-7b6311475fdd
        name: system_column_name1
        dataType: STRING
        description: Some description
        nullable: true
        defaultValue: hello
        tests: []
        isSystemColumnName: true
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
        sourceColumns:
          - {}
      - id: 8791b268-3a95-4ac0-9929-1a2b328c60bd
        name: L_ORDERKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests:
          - name: "Null"
            description: checks for any null values
            continueOnFailure: true
            runOrder: After
            templateString: |2-

                          SELECT * FROM {{ ref(node.location.name, node.name)}} 
                          WHERE "L_ORDERKEY" IS NULL
          - name: Unique
            description: check if value is unique
            continueOnFailure: true
            runOrder: After
            templateString: |2-

                          SELECT "L_ORDERKEY", COUNT(*)
                          FROM {{ ref(node.location.name, node.name)}} 
                          GROUP BY "L_ORDERKEY"
                          HAVING COUNT(*) > 1
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 6bf537d0-c46a-4179-8a7d-0e1b0a1d1108
              name: L_ORDERKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 6f73c0e2-fe2b-45bc-b181-dcaa02256b38
        name: L_PARTKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 4108767a-2457-4ee5-8eac-7681326d1a88
              name: L_PARTKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 72636ad8-6c64-4983-a222-f9ac533c4eed
        name: L_SUPPKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests:
          - name: "Null"
            description: checks for any null values
            continueOnFailure: true
            runOrder: After
            templateString: |2-

                          SELECT * FROM {{ ref(node.location.name, node.name)}} 
                          WHERE "L_SUPPKEY" IS NULL
          - name: Unique
            description: check if value is unique
            continueOnFailure: true
            runOrder: After
            templateString: |2-

                          SELECT "L_SUPPKEY", COUNT(*)
                          FROM {{ ref(node.location.name, node.name)}} 
                          GROUP BY "L_SUPPKEY"
                          HAVING COUNT(*) > 1
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 2c9b638c-360f-4b86-8752-52d1120d445e
              name: L_SUPPKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 99710860-2240-42ac-bcad-d5011ce50909
        name: L_LINENUMBER
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: edf4d472-6aad-4a41-8f5a-1da1dfe6bfc8
              name: L_LINENUMBER
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 6dc793eb-8f7d-4f13-a315-8a1cacdf541a
        name: L_QUANTITY
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests:
          - name: "Null"
            description: checks for any null values
            continueOnFailure: true
            runOrder: After
            templateString: |2-

                          SELECT * FROM {{ ref(node.location.name, node.name)}} 
                          WHERE "L_QUANTITY" IS NULL
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 1a2db895-c5d1-49c3-9677-09b6d1f6f742
              name: L_QUANTITY
              dataType: NUMBER(12,2)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 9fc64e2f-fd37-42e6-8f40-bc78a21680ed
        name: C_CUSTKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: e9cfcbf7-766b-4acc-90c5-c5f4801a64e1
              name: C_CUSTKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: eb70b19f-dc09-4b42-833b-c9c5d9d4897a
        name: C_NAME
        dataType: VARCHAR(25)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 133f3952-bca1-4ad0-a748-3d9e4c62a4f9
              name: C_NAME
              dataType: VARCHAR(25)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 4a062af3-1777-474a-b6c8-635adb540959
        name: C_ADDRESS
        dataType: VARCHAR(40)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: aaef39a1-da9b-437a-b9ff-5a6def36c1ec
              name: C_ADDRESS
              dataType: VARCHAR(40)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 7dbe0df9-10b8-4c6b-8376-eae71c25632e
        name: C_NATIONKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: e938c2bf-fb1e-4ff0-b319-5d26ef7c67d8
              name: C_NATIONKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 59c6ddb9-e8bd-4608-b011-6377c8906534
        name: C_PHONE
        dataType: VARCHAR(15)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: dbeaf294-04ab-4b14-98b9-1be0c6d8e441
              name: C_PHONE
              dataType: VARCHAR(15)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 3d1a6245-0bef-4fa3-b2c9-273b24a0c275
        name: C_ACCTBAL
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: a7ba5727-5e81-4ecb-b76a-f55ec0382458
              name: C_ACCTBAL
              dataType: NUMBER(12,2)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: a4b64e99-eb5f-406b-9c5f-baddf3c99103
        name: C_MKTSEGMENT
        dataType: VARCHAR(10)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 79cdb302-ebb4-472b-8e20-091310dc3e87
              name: C_MKTSEGMENT
              dataType: VARCHAR(10)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: bd37ac6b-ce51-485a-9e52-8a3e4b79475a
        name: C_COMMENT
        dataType: VARCHAR(117)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 2998976d-8ac6-4c60-b816-3694225928b8
              name: C_COMMENT
              dataType: VARCHAR(117)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: a96cd9cb-8201-4c54-8f04-c41e46c41115
        name: L_EXTENDEDPRICE
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 31b65bef-42f8-4f73-bd93-5cdcf785f490
              name: L_EXTENDEDPRICE
              dataType: NUMBER(12,2)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 7ba1fe2a-6ff6-44ce-a76c-831dc01d0b59
        name: L_DISCOUNT
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 332bfaf3-7b6b-440d-a4e9-d8c5cd2c64db
              name: L_DISCOUNT
              dataType: NUMBER(12,2)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 2dc17e49-4013-445b-81e1-9ec64a25f472
        name: L_TAX
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 1838f7d7-f0b7-4359-b69a-d4b80bd9370c
              name: L_TAX
              dataType: NUMBER(12,2)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: c2823c69-a131-4c5c-93b5-1630b1385595
        name: L_RETURNFLAG
        dataType: VARCHAR(1)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: c2f5a9e8-dda3-467c-ae2b-f48335e87a73
              name: L_RETURNFLAG
              dataType: VARCHAR(1)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: fe7dc273-d716-45a6-92d6-d311a5355440
        name: L_LINESTATUS
        dataType: VARCHAR(1)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: aaaa9052-9c2e-4212-a79e-eb302677e3d0
              name: L_LINESTATUS
              dataType: VARCHAR(1)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 54e1b35c-4676-46da-a371-7c7da356f2bb
        name: L_SHIPDATE
        dataType: DATE
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 9cad565a-7c63-42cd-a3c1-0f79fee13102
              name: L_SHIPDATE
              dataType: DATE
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 8959011e-a95b-4673-860c-a8f48e1d6b48
        name: L_COMMITDATE
        dataType: DATE
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 7b339c3a-fe69-42cc-99ec-b59a4a648b33
              name: L_COMMITDATE
              dataType: DATE
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 591df848-a979-4cd8-8bb2-4c88204aba2d
        name: L_RECEIPTDATE
        dataType: DATE
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 3813fedb-aa3c-4d28-a2aa-3915556def9d
              name: L_RECEIPTDATE
              dataType: DATE
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: f2fd738f-ca2b-4845-bb7f-4d18703ecb23
        name: L_SHIPINSTRUCT
        dataType: VARCHAR(25)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 59b4d78a-5719-4c4e-b564-c02057badb5f
              name: L_SHIPINSTRUCT
              dataType: VARCHAR(25)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: 602fa59d-4697-4e27-b9a1-bbb2d6400e67
        name: L_SHIPMODE
        dataType: VARCHAR(10)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 1c0e32ea-70b3-446c-8c0b-77c7fab2c701
              name: L_SHIPMODE
              dataType: VARCHAR(10)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
      - id: ee257834-ee7f-4da5-8689-4e1b22500fad
        name: L_COMMENT
        dataType: VARCHAR(44)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
              name: STG_LINEITEM
              nodeType: Stage
              location:
                name: WORK
              description: Lineitem data as defined by TPC-H
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 031c8ddc-9fe6-4754-bc54-eba44190d9a7
              name: L_COMMENT
              dataType: VARCHAR(44)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
    join: FROM {{ ref('WORK', 'STG_LINEITEM') }} "STG_LINEITEM"
    dependencies:
      - node:
          id: 9fcf37c9-c8b2-4b86-9009-5cda886df1ba
          name: STG_LINEITEM
          nodeType: Stage
          location:
            name: WORK
          description: Lineitem data as defined by TPC-H
          materializationType: table
          isMultisource: false
          override:
            create:
              enabled: false
              script: ""
          tests: []
        columns:
          - id: 6bf537d0-c46a-4179-8a7d-0e1b0a1d1108
            name: L_ORDERKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 4108767a-2457-4ee5-8eac-7681326d1a88
            name: L_PARTKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 2c9b638c-360f-4b86-8752-52d1120d445e
            name: L_SUPPKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: edf4d472-6aad-4a41-8f5a-1da1dfe6bfc8
            name: L_LINENUMBER
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 1a2db895-c5d1-49c3-9677-09b6d1f6f742
            name: L_QUANTITY
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 31b65bef-42f8-4f73-bd93-5cdcf785f490
            name: L_EXTENDEDPRICE
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 332bfaf3-7b6b-440d-a4e9-d8c5cd2c64db
            name: L_DISCOUNT
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 1838f7d7-f0b7-4359-b69a-d4b80bd9370c
            name: L_TAX
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: c2f5a9e8-dda3-467c-ae2b-f48335e87a73
            name: L_RETURNFLAG
            dataType: VARCHAR(1)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: aaaa9052-9c2e-4212-a79e-eb302677e3d0
            name: L_LINESTATUS
            dataType: VARCHAR(1)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 9cad565a-7c63-42cd-a3c1-0f79fee13102
            name: L_SHIPDATE
            dataType: DATE
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 7b339c3a-fe69-42cc-99ec-b59a4a648b33
            name: L_COMMITDATE
            dataType: DATE
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 3813fedb-aa3c-4d28-a2aa-3915556def9d
            name: L_RECEIPTDATE
            dataType: DATE
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 59b4d78a-5719-4c4e-b564-c02057badb5f
            name: L_SHIPINSTRUCT
            dataType: VARCHAR(25)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 1c0e32ea-70b3-446c-8c0b-77c7fab2c701
            name: L_SHIPMODE
            dataType: VARCHAR(10)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
          - id: 031c8ddc-9fe6-4754-bc54-eba44190d9a7
            name: L_COMMENT
            dataType: VARCHAR(44)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    customSQL: ""
node:
  id: scratch
  name: DOCS_DIM_LINE_ITEM
  nodeType: "16"
  location:
    name: WORK
  description: Lineitem data as defined by TPC-H
  materializationType: table
  isMultisource: true
  override:
    create:
      enabled: false
      script: ""
  tests:
    - name: Test
      description: ""
      continueOnFailure: true
      runOrder: After
      templateString: SELECT * FROM COALESCE
this: "{{ ref_no_link(node.location.name, node.name) }}"
parameters:
  hello: goodbye
  goodMorning: goodnight
```

</details>

<details>
<summary>**Multi Source Node Example**</summary>

```yaml
columns:
  - id: f8457ba3-fda2-4141-bcb5-857140b7724b
    name: N_CREWKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: f158fed0-78b2-400f-a24b-1fb5dd385f2b
    name: N_FIRSTNAME
    dataType: VARCHAR(50)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: fd76799a-f594-4668-8d13-52372bd125b1
    name: N_LASTNAME
    dataType: VARCHAR(50)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: 8cb29d5c-eac4-4bc5-a3e1-ac9069adab6f
    name: N_COMMENT
    dataType: VARCHAR(152)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
  - id: 4f4ed987-12e5-4d7f-b051-ba2ba6e75a83
    name: N_MIDDLEINITIAL
    dataType: VARCHAR(1)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
storageLocations:
  - database: TATIANA_GENERAL
    schema: DEV
    name: SAMPLE
  - database: TATIANA_GENERAL
    schema: WORK
    name: WORK
config:
  preSQL: ""
  truncateBefore: true
  insertStrategy: INSERT
  testsEnabled: true
  postSQL: ""
sources:
  - name: SRC1
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
              name: NOSTROMO_SRC1
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 32991b79-632f-4729-9c84-61f53e53069c
              name: N_CREWKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: f158fed0-78b2-400f-a24b-1fb5dd385f2b
        name: N_FIRSTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
              name: NOSTROMO_SRC1
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 7772191a-c470-4b20-b90b-79fbfcd41072
              name: N_FIRSTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: fd76799a-f594-4668-8d13-52372bd125b1
        name: N_LASTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
              name: NOSTROMO_SRC1
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: f7d3b0ac-8ea4-4cc6-8299-54443242a70f
              name: N_LASTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 8cb29d5c-eac4-4bc5-a3e1-ac9069adab6f
        name: N_COMMENT
        dataType: VARCHAR(152)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
              name: NOSTROMO_SRC1
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 24cc3381-cd9d-491a-8ec8-166d29769610
              name: N_COMMENT
              dataType: VARCHAR(152)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 4f4ed987-12e5-4d7f-b051-ba2ba6e75a83
        name: N_MIDDLEINITIAL
        dataType: VARCHAR(1)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - {}
    join: FROM {{ ref('SAMPLE', 'NOSTROMO_SRC1') }} "NOSTROMO_SRC1"
    dependencies:
      - node:
          id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
          name: NOSTROMO_SRC1
          nodeType: Source
          location:
            name: SAMPLE
          description: ""
          materializationType: table
          isMultisource: false
          override:
            create:
              enabled: false
              script: ""
          tests: []
        columns:
          - id: 32991b79-632f-4729-9c84-61f53e53069c
            name: N_CREWKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 7772191a-c470-4b20-b90b-79fbfcd41072
            name: N_FIRSTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: f7d3b0ac-8ea4-4cc6-8299-54443242a70f
            name: N_LASTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 24cc3381-cd9d-491a-8ec8-166d29769610
            name: N_COMMENT
            dataType: VARCHAR(152)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    customSQL: ""
  - name: SRC2
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
              name: NOSTROMO_SRC2
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: d0d3a1be-4068-4f5d-b61f-c74bf4952025
              name: N_CREWKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: f158fed0-78b2-400f-a24b-1fb5dd385f2b
        name: N_FIRSTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
              name: NOSTROMO_SRC2
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 11611358-8d45-4284-ad77-d377ceb117dc
              name: N_FIRSTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: fd76799a-f594-4668-8d13-52372bd125b1
        name: N_LASTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
              name: NOSTROMO_SRC2
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 30774bd2-455e-4211-95f0-2b9e45363dde
              name: N_LASTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 8cb29d5c-eac4-4bc5-a3e1-ac9069adab6f
        name: N_COMMENT
        dataType: VARCHAR(152)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
              name: NOSTROMO_SRC2
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 01d323c6-5958-4e47-a521-219bad4ac172
              name: N_COMMENT
              dataType: VARCHAR(152)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 4f4ed987-12e5-4d7f-b051-ba2ba6e75a83
        name: N_MIDDLEINITIAL
        dataType: VARCHAR(1)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
              name: NOSTROMO_SRC2
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 8c45b2ed-1e76-4c1f-802c-8f39dd75de0c
              name: N_MIDDLEINITIAL
              dataType: VARCHAR(1)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
    join: FROM {{ ref('SAMPLE', 'NOSTROMO_SRC2') }} "NOSTROMO_SRC2"
    dependencies:
      - node:
          id: 9ee5c8a0-b962-48ef-96c5-2e5e3e2516bc
          name: NOSTROMO_SRC2
          nodeType: Source
          location:
            name: SAMPLE
          description: ""
          materializationType: table
          isMultisource: false
          override:
            create:
              enabled: false
              script: ""
          tests: []
        columns:
          - id: d0d3a1be-4068-4f5d-b61f-c74bf4952025
            name: N_CREWKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 11611358-8d45-4284-ad77-d377ceb117dc
            name: N_FIRSTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 8c45b2ed-1e76-4c1f-802c-8f39dd75de0c
            name: N_MIDDLEINITIAL
            dataType: VARCHAR(1)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 30774bd2-455e-4211-95f0-2b9e45363dde
            name: N_LASTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 01d323c6-5958-4e47-a521-219bad4ac172
            name: N_COMMENT
            dataType: VARCHAR(152)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    customSQL: ""
  - name: SRC3
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: e49d528e-dbbc-48c5-82d7-408a5c44f05f
              name: NOSTROMO_SRC3
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 9cf87c06-3016-44e7-ac34-efd58c83cab4
              name: N_CREWKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: f158fed0-78b2-400f-a24b-1fb5dd385f2b
        name: N_FIRSTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: e49d528e-dbbc-48c5-82d7-408a5c44f05f
              name: NOSTROMO_SRC3
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 8f1f539f-1077-4740-b71e-2cd3d08ba760
              name: N_FIRSTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: fd76799a-f594-4668-8d13-52372bd125b1
        name: N_LASTNAME
        dataType: VARCHAR(50)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: e49d528e-dbbc-48c5-82d7-408a5c44f05f
              name: NOSTROMO_SRC3
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: b28e03e2-a8c2-4a97-a676-399b5ddd0258
              name: N_LASTNAME
              dataType: VARCHAR(50)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 8cb29d5c-eac4-4bc5-a3e1-ac9069adab6f
        name: N_COMMENT
        dataType: VARCHAR(152)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: e49d528e-dbbc-48c5-82d7-408a5c44f05f
              name: NOSTROMO_SRC3
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: a32323d1-705e-4bb0-8318-9fb105f32f65
              name: N_COMMENT
              dataType: VARCHAR(152)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
      - id: 4f4ed987-12e5-4d7f-b051-ba2ba6e75a83
        name: N_MIDDLEINITIAL
        dataType: VARCHAR(1)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - {}
    join: FROM {{ ref('SAMPLE', 'NOSTROMO_SRC3') }} "NOSTROMO_SRC3"
    dependencies:
      - node:
          id: e49d528e-dbbc-48c5-82d7-408a5c44f05f
          name: NOSTROMO_SRC3
          nodeType: Source
          location:
            name: SAMPLE
          description: ""
          materializationType: table
          isMultisource: false
          override:
            create:
              enabled: false
              script: ""
          tests: []
        columns:
          - id: 9cf87c06-3016-44e7-ac34-efd58c83cab4
            name: N_CREWKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: 8f1f539f-1077-4740-b71e-2cd3d08ba760
            name: N_FIRSTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: b28e03e2-a8c2-4a97-a676-399b5ddd0258
            name: N_LASTNAME
            dataType: VARCHAR(50)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
          - id: a32323d1-705e-4bb0-8318-9fb105f32f65
            name: N_COMMENT
            dataType: VARCHAR(152)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    customSQL: ""
node:
  id: scratch
  name: STG_NOSTROMO_COMBINED
  nodeType: Stage
  location:
    name: SAMPLE
  description: ""
  materializationType: table
  isMultisource: true
  override:
    create:
      enabled: false
      script: ""
  tests: []
this: "{{ ref_no_link(node.location.name, node.name) }}"
parameters: {}
```

</details>

[Hydrated Metadata Object]:/docs/build-your-pipeline/user-defined-nodes/hydrated-metadata-reference#hydrated-metadata-object

---

## Hydrated Metadata

Hydrated Metadata is the object that you have access to when creating custom Nodes. It contains information like the available columns, storage location information, and other Node information. The metadata is populated from the Node data and user inputs defined in the Node Definition. For example, the Node Name and column names are from the Node data. Fields like tests, business key, and custom fields are populated when the user makes a choice.

This guide goes over some of the available Node Metadata.

Review the [Node Metadata Reference][] for a full list of available Node metadata information.

<details>
<summary>**Hydrated Metadata Example**</summary>

```yaml
columns:
- id: ae9488da-ecb0-4c01-bc76-e7bbe9f4f916
    name: DIM_CUSTOMER1_KEY
    dataType: NUMBER
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSurrogateKey: true
- id: 7223f867-6939-439a-82b5-160b6cefd94f
    name: C_CUSTKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: 3f7491c3-d4b0-4af9-ba3f-1195abce361d
    name: C_NAME
    dataType: VARCHAR(25)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: d731b559-6967-45cf-a988-9645d78057c5
    name: C_ADDRESS
    dataType: VARCHAR(40)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: 2ecc8d8a-cafb-41b6-b251-0f6bd9466dd0
    name: C_NATIONKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: 30dd7333-7d8e-4b71-8468-23b455e03678
    name: C_PHONE
    dataType: VARCHAR(15)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: 27138b8e-063a-4b66-9e5c-6f0f62ebdc92
    name: C_ACCTBAL
    dataType: NUMBER(12,2)
    description: ""
    nullable: false
    defaultValue: ""
    tests: []
- id: bfb86085-716f-4761-b3db-6612799db3d1
    name: C_MKTSEGMENT
    dataType: VARCHAR(10)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
- id: c2dd9319-03a8-438f-b2b8-552db628403e
    name: C_COMMENT
    dataType: VARCHAR(117)
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
- id: d6376c0f-52fe-4b07-99d6-447c4c114ce0
    name: SYSTEM_VERSION
    dataType: NUMBER
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemVersion: true
- id: 7c0345be-e5e9-40f5-8fa4-48775973dc14
    name: SYSTEM_CURRENT_FLAG
    dataType: VARCHAR
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemCurrentFlag: true
- id: fa6f3d17-7c30-4711-a1a0-345bba6fe4f3
    name: SYSTEM_START_DATE
    dataType: TIMESTAMP
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemStartDate: true
- id: c7823b90-b12c-4132-a0a2-99a0cd407116
    name: SYSTEM_END_DATE
    dataType: TIMESTAMP
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemEndDate: true
- id: df8534c7-cb55-4898-a845-01a1aa53b634
    name: SYSTEM_CREATE_DATE
    dataType: TIMESTAMP
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemCreateDate: true
- id: f5f0ce1f-d4b0-4c8a-b259-cd54da4ed4c8
    name: SYSTEM_UPDATE_DATE
    dataType: TIMESTAMP
    description: ""
    nullable: true
    defaultValue: ""
    tests: []
    isSystemUpdateDate: true
storageLocations:
- database: SNOWFLAKE_SAMPLE_DATA
    schema: TPCH_SF100
    name: SAMPLE
- database: TATIANA_DOCS_DYNAMIC_TABLES
    schema: HYDRATED
    name: WORK
config:
testsEnabled: true
postSQL: ""
preSQL: ""
myDropdown: option1
sources:
- name: DIM_CUSTOMER1
    columns:
    - id: ae9488da-ecb0-4c01-bc76-e7bbe9f4f916
        name: DIM_CUSTOMER1_KEY
        dataType: NUMBER
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSurrogateKey: true
        transform: ""
        sourceColumns:
        - {}
    - id: 7223f867-6939-439a-82b5-160b6cefd94f
        name: C_CUSTKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: e9cfcbf7-766b-4acc-90c5-c5f4801a64e1
            name: C_CUSTKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: 3f7491c3-d4b0-4af9-ba3f-1195abce361d
        name: C_NAME
        dataType: VARCHAR(25)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: 133f3952-bca1-4ad0-a748-3d9e4c62a4f9
            name: C_NAME
            dataType: VARCHAR(25)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: d731b559-6967-45cf-a988-9645d78057c5
        name: C_ADDRESS
        dataType: VARCHAR(40)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: aaef39a1-da9b-437a-b9ff-5a6def36c1ec
            name: C_ADDRESS
            dataType: VARCHAR(40)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: 2ecc8d8a-cafb-41b6-b251-0f6bd9466dd0
        name: C_NATIONKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: e938c2bf-fb1e-4ff0-b319-5d26ef7c67d8
            name: C_NATIONKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: 30dd7333-7d8e-4b71-8468-23b455e03678
        name: C_PHONE
        dataType: VARCHAR(15)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: dbeaf294-04ab-4b14-98b9-1be0c6d8e441
            name: C_PHONE
            dataType: VARCHAR(15)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: 27138b8e-063a-4b66-9e5c-6f0f62ebdc92
        name: C_ACCTBAL
        dataType: NUMBER(12,2)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: a7ba5727-5e81-4ecb-b76a-f55ec0382458
            name: C_ACCTBAL
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
    - id: bfb86085-716f-4761-b3db-6612799db3d1
        name: C_MKTSEGMENT
        dataType: VARCHAR(10)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: 79cdb302-ebb4-472b-8e20-091310dc3e87
            name: C_MKTSEGMENT
            dataType: VARCHAR(10)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    - id: c2dd9319-03a8-438f-b2b8-552db628403e
        name: C_COMMENT
        dataType: VARCHAR(117)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
        - node:
            id: d9d2dca5-7ec1-4401-b893-443a19419c36
            name: STG_CUSTOMER
            nodeType: Stage
            location:
                name: SAMPLE
            description: Hello
            materializationType: table
            isMultisource: false
            override:
                create:
                enabled: false
                script: ""
            tests: []
            column:
            id: 2998976d-8ac6-4c60-b816-3694225928b8
            name: C_COMMENT
            dataType: VARCHAR(117)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    - id: d6376c0f-52fe-4b07-99d6-447c4c114ce0
        name: SYSTEM_VERSION
        dataType: NUMBER
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemVersion: true
        transform: ""
        sourceColumns:
        - {}
    - id: 7c0345be-e5e9-40f5-8fa4-48775973dc14
        name: SYSTEM_CURRENT_FLAG
        dataType: VARCHAR
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemCurrentFlag: true
        transform: ""
        sourceColumns:
        - {}
    - id: fa6f3d17-7c30-4711-a1a0-345bba6fe4f3
        name: SYSTEM_START_DATE
        dataType: TIMESTAMP
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemStartDate: true
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
        sourceColumns:
        - {}
    - id: c7823b90-b12c-4132-a0a2-99a0cd407116
        name: SYSTEM_END_DATE
        dataType: TIMESTAMP
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemEndDate: true
        transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
        sourceColumns:
        - {}
    - id: df8534c7-cb55-4898-a845-01a1aa53b634
        name: SYSTEM_CREATE_DATE
        dataType: TIMESTAMP
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemCreateDate: true
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
        sourceColumns:
        - {}
    - id: f5f0ce1f-d4b0-4c8a-b259-cd54da4ed4c8
        name: SYSTEM_UPDATE_DATE
        dataType: TIMESTAMP
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        isSystemUpdateDate: true
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
        sourceColumns:
        - {}
    join: FROM {{ ref('SAMPLE', 'STG_CUSTOMER') }} "STG_CUSTOMER"
    dependencies:
    - node:
        id: d9d2dca5-7ec1-4401-b893-443a19419c36
        name: STG_CUSTOMER
        nodeType: Stage
        location:
            name: SAMPLE
        description: Hello
        materializationType: table
        isMultisource: false
        override:
            create:
            enabled: false
            script: ""
        tests: []
        columns:
        - id: e9cfcbf7-766b-4acc-90c5-c5f4801a64e1
            name: C_CUSTKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: 133f3952-bca1-4ad0-a748-3d9e4c62a4f9
            name: C_NAME
            dataType: VARCHAR(25)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: aaef39a1-da9b-437a-b9ff-5a6def36c1ec
            name: C_ADDRESS
            dataType: VARCHAR(40)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: e938c2bf-fb1e-4ff0-b319-5d26ef7c67d8
            name: C_NATIONKEY
            dataType: NUMBER(38,0)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: dbeaf294-04ab-4b14-98b9-1be0c6d8e441
            name: C_PHONE
            dataType: VARCHAR(15)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: a7ba5727-5e81-4ecb-b76a-f55ec0382458
            name: C_ACCTBAL
            dataType: NUMBER(12,2)
            description: ""
            nullable: false
            defaultValue: ""
            tests: []
        - id: 79cdb302-ebb4-472b-8e20-091310dc3e87
            name: C_MKTSEGMENT
            dataType: VARCHAR(10)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
        - id: 2998976d-8ac6-4c60-b816-3694225928b8
            name: C_COMMENT
            dataType: VARCHAR(117)
            description: ""
            nullable: true
            defaultValue: ""
            tests: []
    customSQL: ""
node:
id: scratch
name: DIM_CUSTOMER1
nodeType: "19"
location:
    name: WORK
description: Hello
materializationType: table
isMultisource: false
override:
    create:
    enabled: false
    script: ""
tests: []
this: "{{ ref_no_link(node.location.name, node.name) }}"
parameters:
hello: goodbye
goodMorning: goodnight
```

</details>

<figure>
  
  <figcaption>Node Type Editor with Metadata</figcaption>
</figure>

## Columns

Columns define the structure of your metadata. Every node will come with:

* `id` - An auto-generated column ID. This can’t be used when creating Nodes.
* `name` - The column identifier.
* `dataType` - A SQL data type.
* `description` - Description of the column.
* `nullable` - Boolean that indicates if NULL values are allowed.
* `defaultValue` - Default value of the column.
* `tests` - An array of column level tests.

```yaml title="Node netadata column object example"
columns:
 - id: e6841805-ab8d-46a5-93cb-5b8528400db2
   name: DIM_CUSTOMER_KEY
   dataType: NUMBER
   description: ""
   nullable: true
   defaultValue: ""
   tests: []
   isSurrogateKey: true
   myMappingColumn: Only available with data
 - id: ddef2ed3-0fae-4133-9012-23e12c880685
   name: C_CUSTKEY
   dataType: NUMBER(38,0)
   description: ""
   nullable: false
   defaultValue: ""
   tests: []
   isBusinessKey: true
....

 - id: 32568624-8008-466f-b62a-672cb6e101be
   name: SYSTEM_UPDATE_DATE
   dataType: TIMESTAMP
   description: ""
   nullable: true
   defaultValue: ""
   tests: []
   isSystemUpdateDate: true
 - id: 5356f829-0de5-43a1-bb40-065d4086f196
   name: GH
   dataType: STRING
   description: ""
   nullable: true
   defaultValue: ""
   tests: []
```

<figure>
  
  <figcaption>Node Type Editor with columns</figcaption>
</figure>

### Dynamic Column Fields

Depending on the Node Definition or user input, there can be fields that don't have data, some that aren’t populated unless they contain data, or only populate on initialization. For example, `tests: []` shows as an empty object since there are no tests configured on the Node. To add a test, go to the **Node Editor > Testing**. Then go back to the Node Type Editor and reload the Node. You’ll see the test array populate. The following example has column test added. Node tests will appear in the Node object.

<figure>
  
  <figcaption>Tests in the Node Editor</figcaption>
</figure>

```yaml title="Metadata with tests array" showLineNumbers
    tests:
      - name: Unique
        description: check if value is unique
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT "DIM_LINEITEM_KEY", COUNT(*)
                      FROM {{ ref(node.location.name, node.name)}} 
                      GROUP BY "DIM_LINEITEM_KEY"
                      HAVING COUNT(*) > 1
...
  - id: 5f46755f-cad3-48af-a91e-72e2bf030390
    name: L_SUPPKEY
    dataType: NUMBER(38,0)
    description: ""
    nullable: false
    defaultValue: ""
    tests:
      - name: "Null"
        description: checks for any null values
        continueOnFailure: true
        runOrder: After
        templateString: |2-

                      SELECT * FROM {{ ref(node.location.name, node.name)}} 
                      WHERE "L_SUPPKEY" IS NULL
```

:::info[Removing Tests]

You can disable testing on the Node by using the Enable Tests option in the Node Definition. Users will still be able to add tests, but the tests will not run.

:::

### Mapping Columns

Some fields only appear in the column metadata if they are applicable to the Node or have data added. For example, [Mapping Columns][]. You can test this out by adding `mappingColumns` to the Node Definition then adding some data to the field, then reloading the Node Type Editor.

```yaml title="Add to Node Definition"
mappingColumns:
- headerName: My Mapping Column Name
  attributeName: myMappingColumn
  type: textBox
```

```yaml title="myMappingColumn is invisible until data is added" showLineNumbers {19}
columns:
 - id: e6841805-ab8d-46a5-93cb-5b8528400db2
   name: DIM_CUSTOMER_KEY
   dataType: NUMBER
   description: ""
   nullable: true
   defaultValue: ""
   tests:
     - name: "Null"
       description: checks for any null values
       continueOnFailure: true
       runOrder: After
       templateString: |2-

                     SELECT * FROM {{ ref(node.location.name, node.name)}}
                     WHERE "DIM_CUSTOMER_KEY" IS NULL
   isSurrogateKey: true
   myMappingColumn: Only available with data
```

### System Columns

System Columns only appear as a column on Node initialization. They are given custom attributes which show in the metadata and are set to true to indicate it's a system column. In this example, the custom attribute name is `isHelloGoodbye` and the column name is `system_column_name1`.

```yaml
  - id: 4c1e4719-b72b-41fe-82f6-7b6311475fdd
    name: system_column_name1
    dataType: STRING
    description: Some description
    nullable: true
    defaultValue: hello
    tests: []
    isHelloGoodbye: true
```

## Storage Locations

Storage Locations list out all Storage Locations added to the Node. To know where the Node is currently stored, you can review the [Node object][].

Learn more about [Storage Locations][].

```yaml title="Storage Locations"
storageLocations:
  - database: SNOWFLAKE_SAMPLE_DATA
    schema: TPCH_SF100
    name: SAMPLE
  - database: TATIANA_DOCS_DYNAMIC_TABLES
    schema: HYDRATED
    name: WORK
  - database: TATIANA_DOCS_DYNAMIC_TABLES
    schema: QA
    name: DEV
```

This is where the node is located, `node.location.name`.

```yaml title="Node location information" {6}
node:
  id: scratch
  name: DIM_LINEITEM
  nodeType: "16"
  location:
    name: WORK
```

<figure>
  
  <figcaption>Storage Locations in the Node Editor</figcaption>
</figure>

## Config

The config is where customizable settings are located. There are some that are standard such as `postSQL`, `preSQL` and `testEnabled`. Custom elements created in Node Definition will also appear here. For example, you can create a [custom `dropdown`][] and it will appear here.

```yaml title="Config"
config:
  testsEnabled: true
  postSQL: ""
  preSQL: ""
  myDropdown: option1
```

<figure>
  
  <figcaption>Config in the Node Type Editor</figcaption>
</figure>

## Sources

Sources map your columns to their immediate parent nodes. Each source shows:

* Which Node the data comes from.
* Which column in that Node maps to your column.
* Sources only tracks the direct predecessor and not the full Node lineage.

In this example:

* The Node is `DIM_CUSTOMER_EXAMPLE`
* The column is `C_CUSTKEY`
* The source is `STG_CUSTOMER`

The column `C_CUSTKEY` in the Node `DIM_CUSTOMER_EXAMPLE`. The column named `C_CUSTKEY` has a source node of `STG_CUSTOMER` and a source column in `STG_CUSTOMER` of `C_CUSTKEY`.

The `sources > name > columns > sourceColumns > node` contains some information about the source Node.

The `sources > name > columns > sourceColumns > column` contains some information about the source column.

```yaml title="Sources example DIM_CUSTOMER_EXAMPLE" {14,28}
sources:
  - name: DIM_CUSTOMER_EXAMPLE
    columns:
      - id: 7223f867-6939-439a-82b5-160b6cefd94f
        name: C_CUSTKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: false
        defaultValue: ""
        tests: []
        isBusinessKey: true
        transform: ""
        sourceColumns:
          - node:
              id: d9d2dca5-7ec1-4401-b893-443a19419c36
              name: STG_CUSTOMER
              nodeType: Stage
              location:
                name: SAMPLE
              description: Hello
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: e9cfcbf7-766b-4acc-90c5-c5f4801a64e1
              name: C_CUSTKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: false
              defaultValue: ""
              tests: []
```

<figure>
  
  <figcaption>DAG DIM_CUSTOMER node whose source is STG_CUSTOMER.</figcaption>
</figure>

### Columns With No Sources

There are some that won’t have source columns. For example, system columns. These are created on initialization and don’t have a source.

```yaml title="System columns don't have a source" {16-26}
sources:
 - name: DIM_CUSTOMER1
   columns:
     - id: ae9488da-ecb0-4c01-bc76-e7bbe9f4f916
       name: DIM_CUSTOMER1_KEY
       dataType: NUMBER
       description: ""
       nullable: true
       defaultValue: ""
       tests: []
       isSurrogateKey: true
       transform: ""
       sourceColumns:
         - {}
....
     - id: d6376c0f-52fe-4b07-99d6-447c4c114ce0
       name: SYSTEM_VERSION
       dataType: NUMBER
       description: ""
       nullable: true
       defaultValue: ""
       tests: []
       isSystemVersion: true
       transform: ""
       sourceColumns:
         - {}
```

### Multi Source Nodes

If a Node has multiple sources, there will be multiple source columns with their sources listed. In this example, the node has three sources.

* `SRC1`
* `SRC2`
* `SRC3`

```yaml title="Multi Source" {2,35,40}
sources:
  - name: SRC1
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
        dataType: NUMBER(38,0)
        description: ""
        nullable: true
        defaultValue: ""
        tests: []
        transform: ""
        sourceColumns:
          - node:
              id: 82023b9f-eafe-4a8b-8f9b-b31bd3323d2c
              name: NOSTROMO_SRC1
              nodeType: Source
              location:
                name: SAMPLE
              description: ""
              materializationType: table
              isMultisource: false
              override:
                create:
                  enabled: false
                  script: ""
              tests: []
            column:
              id: 32991b79-632f-4729-9c84-61f53e53069c
              name: N_CREWKEY
              dataType: NUMBER(38,0)
              description: ""
              nullable: true
              defaultValue: ""
              tests: []
  - name: SRC2
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
      .....
  - name: SRC3
    columns:
      - id: f8457ba3-fda2-4141-bcb5-857140b7724b
        name: N_CREWKEY
```

## Node

This contains core information about the Node itself, such as the Storage Location. Use this object when accessing Node data.

```yaml title="Node"
node:
 id: scratch
 name: DIM_CUSTOMER1
 nodeType: "19"
 location:
   name: WORK
 description: Hello
 materializationType: table
 isMultisource: false
 override:
   create:
     enabled: false
     script: ""
 tests: []
```

### Scratch ID

When a Node is selected dropdown menu in the Node Type Editor the system makes a copy of the Node. This copy is referred to as the `scratch` Node. When working with Nodes, you are working with a `scratch copy` of the Node. `id: scratch` can't be used for anything or be accessed. The `nodeType` is the type of Node you are working on. For scratch nodes, or nodes you are building this is a random number.

```yaml title="Scratch"
node:
  id: scratch
  name: DOCS_DIM_LINE_ITEM
  nodeType: "16"
```

## This

This refers to the Node. `this` refers to a templating reference that provides a shorthand way to reference the fully qualified name of the Node being worked on. Instead of writing out the full reference with location name and node name, you can use `this` to reference the current Node in templates. Learn more about [Ref Functions][].

```yaml title="This"
this: "{{ ref_no_link(node.location.name, node.name) }}"
```

For example, in a Create Template you might have `{{ ref_no_link(node.location.name, node.name) }}`, which can be shortened to `{{this}}`.

<Tabs>
  <TabItem value="create-template-with-this" label="Create template with this" default>

```sql {5}
{% if node.materializationType == 'table' %}
{{ stage('Create Dimension Table') }}

CREATE OR REPLACE TABLE {{this}}
(
    {% for col in columns %}
        "{{ col.name }}" {{ col.dataType }}
        {% if col.isSurrogateKey %}
            identity
        {% endif %}
        {%- if not col.nullable %} NOT NULL
            {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
        {% endif %}
```

  </TabItem>
  <TabItem value="create-template-orignal" label="Create template using ref_no_link">

```sql {5}
{% if node.materializationType == 'table' %}
{{ stage('Create Dimension Table') }}

CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
(
    {% for col in columns %}
        "{{ col.name }}" {{ col.dataType }}
        {% if col.isSurrogateKey %}
            identity
        {% endif %}
```

  </TabItem>
</Tabs>

## Parameters

Parameters in the metadata represent runtime parameters that can be passed to templates. They can contain arbitrary key-pair values. They act as a way to pass in variables without having to hard code them into the template.

```yaml title="Paramters" {7-9}
columns:...
storageLocations:...
config:...
sources:...
node:...
this:...
parameters:{}
```

Using JSON add a parameter, for example:

```json
{
    "hello": "goodbye",
    "goodMorning": "goodnight"
}
```

<figure>
  
  <figcaption>Node metadata with parameters data</figcaption>
</figure>

## What's Next

* Learn how to [access][] Hydrated Metadata in templates
* Review the [Hydrated Metadata Reference][]

[Node Metadata Reference]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata-reference
[Mapping Columns]: /docs/build-your-pipeline/user-defined-nodes/node-definition#mapping-columns
[Storage Locations]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[Ref Functions]: /docs/reference/ref-functions/
[Node object]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata#node
[custom `dropdown`]: /docs/build-your-pipeline/user-defined-nodes/node-config-options#dropdown-selector
[access]: /docs/build-your-pipeline/user-defined-nodes/access-hyrdated-metadata#dot-notation-in-jinja-templates
[Hydrated Metadata Reference]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata-reference

---

## Custom Node Types

Coalesce comes with built-in [Node types][] and [Packages][] for data transformations, you can also create your own custom Nodes or UDN(user-defined Nodes) using YAML, Jinja 2, and SQL.

To make your own **Node Type**, go to **Build Settings**>**Node Types** and select **New Node Type** to start from scratch or **Duplicate** on any Node Type to extend existing functionality.

## Learn About the Components of a Node

Before building a Node, review our guides on each part of a Node and how they work together.

A Node consists of these components:

1. [**Node Definition**][] - Written in YAML, specifies UI elements and configurations like node color, materialization options, and business keys.
2. [**Create Template**][] - Defines the table's structure (DDL).
3. [**Run Template**][] - Defines how the table will be populated with data (DML).
4. [**Hydrated Metadata**][] -   An object that contains all the structured information needed to define a Node in Coalesce.
5. [**Macros**][] - A block of code you can define once to use multiple times. They can be used in the Create or Run template.

## What's Next

* Video - [What Are Custom Node Types in Coalesce?][]

<DocCardList />

[Node types]: /docs/get-started/coalesce-fundamentals/nodes/
[Packages]: /docs/marketplace
[What Are Custom Node Types in Coalesce?]:https://www.youtube.com/watch?v=I-A_mjTgbAA
[**Node Definition**]: /docs/build-your-pipeline/user-defined-nodes/node-definition
[**Create Template**]: /docs/build-your-pipeline/user-defined-nodes/create-run-template
[**Run Template**]: /docs/build-your-pipeline/user-defined-nodes/create-run-template
[**Hydrated Metadata**]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata
[**Macros**]: /docs/reference/macros/

---

## Join Templates

The Coalesce platform generates a Join string by default when a node is initialized or when a user triggers a Generate Join. This uses the dependencies of the node to generate a generic `JOIN` string using `INNER JOIN`. However, for more complex use cases, users may wish to customize the behavior by using a Join Template that generates the JOIN string based on the metadata of the node.

## Working With Refs

To ensure that Coalesce captures dependencies from the compiled Join string, the `ref` syntax must be left un-rendered as part of the Join Template. This presents a challenge since we need to render part of the Jinja, but exclude the `ref`. To address this challenge, the Join template will only render to a depth of 1, which is inconsistent with other Jinja templates found elsewhere in the app that recursively render Jinja until no Jinja is left.

The following macros can be utilized to render un-rendered `ref` calls. To use these macros they must be added to your Workspace under **Build Settings** > **Macros**.

```sql
{%- macro ref_raw(location_name, node_name) -%}
    {% raw %}{{ ref('{% endraw %}{{ location_name }}{% raw %}', '{% endraw %}{{ node_name }}{% raw %}') }}{% endraw %}
{%- endmacro -%}

{%- macro ref_no_link_raw(location_name, node_name) -%}
    {% raw %}{{ ref_no_link('{% endraw %}{{ location_name }}{% raw %}', '{% endraw %}{{ node_name }}{% raw %}') }}{% endraw %}
{%- endmacro -%}

{%- macro ref_link_raw(location_name, node_name) -%}
    {% raw %}{{ ref_link('{% endraw %}{{ location_name }}{% raw %}', '{% endraw %}{{ node_name }}{% raw %}') }}{% endraw %}
{%- endmacro -%}
```

Here's an example of using `ref_raw` to generate a very simple Join string using the Join Template.

```sql
capitalized: 'My Node Name'
short: 'MNN'
plural: 'My Node Names'
tagColor: '#FF5A5F'

joinTemplate: |
  {%- for dep in sources[0].dependencies -%}
    {%- if loop.first %} FROM {% endif -%}
    {%- if not loop.first %}LEFT JOIN {% endif -%}
    {{- ref_raw(dep.node.location.name, dep.node.name) }} {{ dep.node.name }}
    {% if not loop.first %} ON {{ sources[0].dependencies[loop.index0].node.name }}./*COLUMN*/ = {{ sources[0].dependencies[loop.index0 - 1].node.name }}./*COLUMN*/ {%- endif %}
  {% endfor -%}
  
  WHERE 1=1
```

## Stage-Based Templates

Node Type templates typically generate one or many SQL execution queries, which Coalesce refers to as Stages. Stages allow Coalesce to run multiple queries against the data warehouse while providing additional runtime behavior and tracking metrics about individual SQL executions.

For example, you may decide to output a `TRUNCATE` and a `MERGE` statements as individual stages, using Jinja to render stages at the appropriate time conditionally.

Coalesce then reads the stage output and executes each of the stages one by one, tracking each statement's progress and data results.

A Stage only needs to be declared at the beginning of the query. A Stage ends either at the beginning of the next declared Stage or at the end of the template.

To declare a **Stage**, use the following syntax. `{{ stage('My Stage Name') }}`

---

## Node Definition Reference

<!-- vale off -->

Node configuration options determine the behavior of an individual node.

<details>
    <summary>**Node Definition Example**</summary>

```yaml
capitalized: 'My Node Name'                      # Common Name (string, required)
short: 'MNN'                                     # Node prefix. A '_' delimiter will be added automatically. (string, required)
plural: 'My Node Names'                          # plural name of common name (string, required)
tagColor: '#FF5A5F'                              # Node color. CSS colors or hex colors (string, required)
deployStrategy: default              # Deployment strategy - see separate article for details.
isColumnResyncEnabled: true          # Allow non-source columns to be re-synced.

config:                                          # Array of the following config items
- groupName: 'Core UI Elements'                  # Name of config group (string, required)
  description: 'Core UI Elements'                # Description of group. Displayed in the GUI (string)
  enableIf: 'true'                               # If true, display this config group, else hide this config group. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  items:                                         # This will be followed by all your config items/UI elements for the given group (array, required)

                                                 ## Core UI Elements ##
  - type: businessKeyColumns                     # Selector for business key columns
    displayName: 'Business Key'                  # displayName for type businessKeyColumns (cannot be modified by user) 
    attributeName: 'isBusinessKey'               # attributeName for type businessKeyColumns (cannot be modified by user) 
    isRequired: false                            # Require input from user for this config element (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
  
  - type: note                                   # Add a text element anywhere in the configuration.
    attributeName: 'note.attribute'               # Can be user set
    displayName: 'Add text here'
    enableIf: "{% if node.materializationType == 'view' %} 
               false {% else %} true {% endif %}"             # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 

  - type: materializationSelector                # Dropdown selector for materialization options
    default: 'table'                             # Default selection (string)
    options:                                     # Materialization options allowed for this node. (array of strings 'table' | 'view')
    - 'table'  
    - 'view'  
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: multisourceToggle  
    displayName: 'Multi Source'                  # displayName for type multisourceToggle (cannot be modified by user) 
    attributeName: 'node.isMultisource'          # attributeName for type multisourceToggle (cannot be modified by user) 
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: overrideSQLToggle                      # Enables GUI input for DDL SQL 
    displayName: 'Override Create SQL'           # displayName for type overrideSQLToggle (cannot be modified by user) 
    attributeName: 'node.override.create'        # attributeName for type overrideSQLToggle (cannot be modified by user) 
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
   
  - type: changeTrackingColumns                  # Selector for change tracking columns
    displayName: 'Change Tracking'               # displayName for type changeTrackingColumns (cannot be modified by user) 
    attributeName: 'isChangeTracking'            # attributeName for type changeTrackingColumns (cannot be modified by user) 
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

- groupName: 'User-defined UI Elements'          # Name of config group (string, required)
  description: 'User-defined UI Elements'        # Description of group. Displayed in the GUI (string)
  enableIf: 'true'                               # If true, display this config group, else hide this config group
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
  items:                                         # This will be followed by all your config items/UI elements for the given group (array, required)
  
                                                 # User-defined UI Elements
  - type: toggleButton                           # Toggle button
    displayName: 'My Button'                     # GUI display name (string, required) 
    attributeName: 'button'                      # Name of element used in templates (string, required)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: dropdownSelector                       # Dropdown of options
    displayName: 'My Dropdown Selector'          # GUI display name (string, required) 
    attributeName: 'dropdown'                    # Name of element used in templates (string, required)
    default: 'option 1'                          # Default selection (string)
    options:                                     # Dropdown options (array of strings, required)
    - 'option 1'
    - 'option 2'
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: columnSelector                         # Transfer config element for selecting multiple columns from the mapping grid
    displayName: 'My Multi Column Selector'      # GUI display name (string, required) 
    attributeName: 'columnSelector'              # Name of element used in templates (string, required)
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: textBox                                # Generic textbox
    displayName: 'My Textbox'                    # GUI display name (string, required) 
    attributeName: 'textbox'                     # Name of element used in templates (string, required)
    syntax: 'none'                               # Syntax highlighting in text box (string, defaults 'none', 'none' | 'sql')
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
  
  - type: tabular                                # Tabular config option. Combines multiple config elements into a single table-like structure
    displayName: 'My Tabular Config'             # GUI display name (string, required) 
    attributeName: 'myTabularConfig'             # Name of element used in templates (string, required)
    columns:                                     # Array of config element types
                                                 # (array, toggleButton | dropdownSelector | columnDropdownSelector | textBox, required) 
    - type: toggleButton                         # Toggle button
      displayName: 'My Button'                   # GUI display name (string, required) 
      attributeName: 'tabToggleButton'           # Name of element used in templates (string, required)
      enableIf: 'true'                           # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
    
    - type: dropdownSelector                     # Dropdown of options
      displayName: 'My Dropdown Selector'        # GUI display name (string, required) 
      attributeName: 'tabDropdown'               # Name of element used in templates (string, required)
      default: 'option 1'                        # Default selection (string)
      options:                                   # Dropdown options (array of strings, required)
      - 'option 1'
      - 'option 2'
      isRequired: false                          # If a config item is required (boolean, defaults false)
      enableIf: 'true'                           # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
    
    - type: textBox                              # Generic textbox
      displayName: 'My Textbox'                  # GUI display name (string, required) 
      attributeName: 'tabTextbox'                # Name of element used in templates (string, required)
      syntax: 'none'                             # Syntax highlighting in text box (string, defaults 'none', 'none' | 'sql')
      isRequired: false                          # If a config item is required (boolean, defaults false)
      enableIf: 'true'                           # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted booelan may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
    
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted booelan may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')                                              
    
- groupName: 'Customizable Core UI Elements'     # Name of config group (string, required)
  description: 'Customizable Core UI Elements'   # Description of group. Displayed in the GUI (string)
  enableIf: 'true'                               # If true, display this config group, else hide this config group. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
  items:                                         # This will be followed by all your config items/UI elements for the given group (array, required)
         
                                                 ## Customizable Core UI Elements ##
  - type: dropdownSelector                       # Select the insert strategy for multisource nodes. Leveraged by Coalesce default Node Types
    displayName: 'Multisource Strategy'          # Config element name (string, required) 
    attributeName: 'insertStrategy'              # Name of element used in templates (string, required)
    default: 'INSERT'                            # Default selection (string)
    options:                                     # Dropdown options (array of strings, required)
    - 'INSERT'
    - 'UNION'
    - 'UNION ALL'

    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: toggleButton                           # Enable or disable truncate before running. Leveraged by Coalesce default Stage Node Type
    displayName: 'Truncate Before'               # GUI display name (string, required) 
    attributeName: 'truncateBefore'              # Name of element used in templates (string, required)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')
    
  - type: toggleButton                           # Enable tests to be run. Leveraged by Coalesce default Node Types
    displayName: 'Enable Tests'                  # GUI display name (string, required) 
    attributeName: 'testsEnabled'                # Name of element used in templates (string, required)
    default: true                                # Element default state (boolean)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: textBox                                # SQL to be executed before run
    displayName: 'Pre-SQL'                       # GUI display name (string, required) 
    attributeName: 'preSQL'                      # Name of element used in templates (string, required)
    syntax: 'sql'                                # Syntax highlighting in text box (string, defaults 'none', 'none' | 'sql')
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

  - type: textBox                                # SQL to be executed after run
    displayName: 'Post-SQL'                      # GUI display name (string, required) 
    attributeName: 'postSQL'                     # Name of element used in templates (string, required)
    syntax: 'sql'                                # Syntax highlighting in text box (string, defaults 'none', 'none' | 'sql')
    isRequired: false                            # If a config item is required (boolean, defaults false)
    enableIf: 'true'                             # If true, display this element, else hide this element. 
                                                 # Jinja expression resulting in a quoted boolean may be used for dynamic behavior 
                                                 # (boolean string, 'true' | 'false', defaults 'true')

    
systemColumns:                                   # Additional columns (rows in mapping grid) to initialize the node with (array of columns objects)
- displayName: 'SYSTEM_COLUMN_1'                 # Column display name (string, required) 
                                                 # '{{NODE_NAME}}' token can be used in the displayName to reference the name of the current node
  attributeName: 'systemColumn1'                 # Name of element used in templates (string, required)
  transform: '1'                                 # Initialized transform value of system column (string, required)
  dataType: 'NUMBER'                             # SQL data type (string, required)
  placement: 'beginning'                         # Determines placement of system column in relation to normal columns.  (string, 'beginning' | 'end')

mappingColumns:                                  # Adds additional columns to the mapping grid for additional metadata per node-column
- type: textBox                                  # Generic textbox in mapping grid
  headerName: 'My Header'                        # GUI display name (string, required) 
  attributeName: 'myMappingColumn'               # Name of element used in templates (string, required)
```

</details>

## Node Properties

|**Name** | **Description** | **Type** |
|---|---|---|
|**Location** | Logical storage location representing the database and schema which are mapped in the Workspace/Environment's Storage Mappings. | String |
|**Node Type** | The name of the type of node. This could be user-defined or a core option. | String |
|**Description** | Description text about this specific node. | String |
|**Deploy Enabled** | This toggle determines if a node will be included in a deployment. By default, all nodes in a workspace are included in a deployment, this allows changing that behavior | Boolean |

:::warning[Disabling Deploy]

If a Node exists on the target environment and deploy is disabled, the next deployment will cause that node's table to be dropped.

:::

### Re-Sync Non Source Columns

`isColumnResyncEnabled: true`

When set to true for a node, it enables a "Re-Sync Columns" button in the top-right corner of the Node’s mapping grid. It allows users to refresh column definitions.

- If a user deletes the columns in the mapping grid, then clicks "Re-Sync Columns," they will be prompted to add the deleted columns.
- If a user deletes a column in their data platform, then tries to re-sync the columns, they will be asked to delete the columns that were removed in the data platform.

Multi source nodes will not display the "Re-Sync Columns" button, even if the flag is enabled.

## Core UI Elements

Core UI elements can't be edited or customized, but they can be combined with customizable elements for more functionality.

### Materialization Selector or Create As

Determines if the Node will be materialized as a table or as a view. This influences how Coalesce will deploy and refresh this node.

```yaml title="materializationSelector"
config:
- groupName: 'My Config Group'
  items:  
  - type: materializationSelector
    default: table
    options:
    - table
    - view
    isRequired: false
  enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **displayName** | The display name can't be changed. It will show as Create As. This field is **read-only** and doesn't need to be included in the configuration. | String |
| **attributeName** | `node.materializationType`. This field is **read-only**. The name used to access the data in this attribute.| String |
| **isRequired** | If this is required. | Boolean |
| **default** | The default selection. `table` or `view` | Boolean |
| **options** | Only `table` or `view` are valid unless using advancedDeployment strategies.  | Object |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>materializationSelector</figcaption>
</figure>

### Multi-Source Toggle

Toggle that allows a Node to accept data from multiple sources by virtue of a SQL union, union all, or individual inserts. When this toggle is true, the mapping grid will allow for multiple source columns to be mapped to a single column in this Node. Combine the Multi-Source toggle with [Multi-Source Strategy][] dropdown.

```yaml title="multisourceToggle"
config:
- groupName: 'My Config Group'
  items: 
  - type: multisourceToggle
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **displayName** | The display name can't be changed. It will show as Multi Source. This field is **read-only** and doesn't need to be included in the configuration. | String |
| **attributeName** | `node.isMultisource`. This field is **read-only**. The attribute name is used to access the data in this attribute.| String |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>multisourceToggle</figcaption>
</figure>

### View DDL Override or Override Create SQL Toggle

This option is used when the `materializationType` is view. Override Create SQL allows a user to override the DDL of a view, enabling greater flexibility in view creation. This option enables a text box in the GUI for the user to write a custom view DDL. Jinja templating is supported in this text box using all of the Node's metadata.

```yaml title="overrideSQLToggle"
config:
- groupName: 'My Config Group'
  items:  
  - type: overrideSQLToggle
    enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **displayName** | The display name can't be changed. It will show as Multi Source. This field is **read-only** and doesn't need to be included in the configuration. | String |
| **attributeName** | `node.override.create.enabled`. This field is **read-only**. The attribute name is used to access the data in this attribute.| String |
| **enableIf** | `{% if node.materializationType == 'view' %} true {% else %} false {% endif %}`| String |

<figure>
  
  <figcaption>overrideSQLToggle</figcaption>
</figure>

### Business Key Selector

Selector that allows the user to specify the business or primary key columns in the Node. It will appear on column-level metadata.

```yaml title="businessKeyColumns"
config:
- groupName: 'My Config Group'
  items:
  - type: businessKeyColumns
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **displayName** | The display name can't be changed. It will show as Business Key. This field is **read-only** and doesn't need to be included in the configuration. | String |
| **attributeName** | `isBusinessKey`. This field is **read-only**. The attribute name is to access the data in this attribute.| String |
| **isRequired** | If the configuration is required.| Boolean|
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>businessKeyColumns</figcaption>
</figure>

### Change Tracking Selector

Selector that allows the user to specify the columns for which change tracking should be enabled. Typically this is used for slowly changing dimensions, but can be used for other applications.

```yaml title="changeTrackingColumns"
config:
- groupName: 'My Config Group'
  items: 
  - type: changeTrackingColumns
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **displayName** | The display name can't be changed. It will show as Change Tracking. This field is **read-only** and doesn't need to be included in the configuration. | String |
| **attributeName** | `isChangeTracking`. This field is **read-only**. The attribute name is to access the data in this attribute.| String |
| **isRequired** | If the configuration is required.| Boolean|
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>changeTrackingColumns</figcaption>
</figure>

## Customizable Core UI Elements

Customizable core UI elements are generic elements that Coalesce uses in our [Node Types][]. These can be customized by altering the `displayName` and `attributeName`.

### Multi-Source Strategy Dropdown Selector

This is a dropdown selector that provides a choice of 3 strategies for combining multiple data sources. This is used when `multisourceToggle` is enabled.

```yaml title="insertStrategy"
config:
- groupName: 'My Config Group'
  items:
  - type: dropdownSelector
    displayName: Multisource Strategy
    attributeName: insertStrategy
    default: INSERT
    options:
    - INSERT
    - UNION
    - UNION ALL
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `dropdownSelector` | String|
| **displayName** | Name of the dropdown selector. Multisource Strategy | String |
| **attributeName** | `insertStrategy`. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **default** | `INSERT` The default selection.| String |
| **options** | `INSERT`, `UNION`, `UNION ALL` Options for the dropdown| String|
| **isRequired** | If the configuration is required.| Boolean|
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>insertStrategy</figcaption>
</figure>

### Truncate Before Toggle

This is a toggle button that determines if the data will be truncated before running this Node's `INSERT` script.

```yaml title="truncateBefore"
config:
- groupName: 'My Config Group'
  items:
  - displayName: Truncate Before
    attributeName: truncateBefore
    type: toggleButton
    default: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `toggleButton` | String|
| **displayName** | Name of the toggle. Truncate Before | String |
| **attributeName** | `truncateBefore`. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **default** | `true` The default selection.| String |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>truncateBefore</figcaption>
</figure>

### Enable Tests Toggle

This toggle determines if user created tests will be run as part of the Node's script. This does not remove Testing from the Node. Users can still enter in testing information, they won't be run in the script.

```yaml title="testsEnabled"
config:
- groupName: 'My Config Group'
  items:  
  - type: toggleButton
    displayName: Enable Tests
    attributeName: testsEnabled
    default: true
    enableIf: 'true' 
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `toggleButton` | String|
| **displayName** | Name of the toggle. Enable Tests | String |
| **attributeName** | `testsEnabled`. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **default** | `true` The default selection.| String |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>testsEnabled</figcaption>
</figure>

### Pre and Post SQL Text Boxes

SQL text boxes that run before or after of the main DML operation.

```yaml title="preSQL and postSQL"
config:
- groupName: 'My Config Group'
  items:  
  - displayName: Pre-SQL
    attributeName: preSQL
    type: textBox
    syntax: sql
    isRequired: false
    enableIf: 'true'  

  - displayName: Post-SQL
    attributeName: postSQL
    type: textBox
    syntax: sql
    isRequired: false
    enableIf: 'true' 
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `textBox` | String|
| **displayName** | Name of the toggle. Pre-SQL and Post-SQL | String |
| **attributeName** | `preSQL`, `postSQL`. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **syntax**| `sql` only. The syntax highlighting in text box. | String|
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>preSQL/postSQL</figcaption>
</figure>

## Generic UI Elements

Below is a comprehensive list of all user-defined items and their possible attributes. These are highly customizable, using both YAML either Jinja2.

### Toggle Button

Defines a toggle button and its options for the node type. The metadata can be referenced in the `config` object using the `attributeName`.

```yaml title="toggleButton"
- groupName: 'My Config Group'
  items:
  - type: toggleButton
    displayName: My Button
    attributeName: myButton
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `toggleButton` | String|
| **displayName** | Name of the toggle. | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>toggleButton</figcaption>
</figure>

### Dropdown Selector

Defines a dropdown selector and its options for the node type. The metadata can be referenced in the `config` object using the `attributeName`.

```yaml title="dropdownSelector"
config:
- groupName: 'My Config Group'
  items:
  - type: dropdownSelector
    displayName: My Dropdown Selector
    attributeName: myDropdown
    default: option1
    options:
    - option1
    - option2
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `dropdownSelector` | String|
| **displayName** | Name of the dropdown selector. | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **default**| The default selection| String|
| **options** | The available dropdown options | String|
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>dropdownSelector</figcaption>
</figure>

### Column Dropdown Selector

Defines a dropdown selector using that Node's columns and its options for the Node type. The metadata can be referenced in the `config` object using the `attributeName`.

```yaml title="columnDropdownSelector"
config:
- groupName: 'My Config Group'
  items:
  - type: columnDropdownSelector 
    displayName: Column Name
    attributeName: myColumnDropdown
    isRequired: true
    enableIf: 'true'

```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `columnDropdownSelector` | String|
| **displayName** | Name of the column selector. | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>columnDropdownSelector</figcaption>
</figure>

### Tabular Configuration Component

A tabular structure for collecting metadata. Each column in the component can be one of several types including:

- `dropdownSelector`
- `toggleButton`
- `columnDropdownSelector`
- `textBox`

The metadata can be referenced in the `config` object using the `attributeName`.

```yaml title="tabular"
config:
- groupName: 'My Config Group'
  items:
  - type: tabular
    displayName: My Tabular Config
    attributeName: myTabularConfig
    isRequired: false
    enableIf: 'true'

    columns:
    - type: dropdownSelector
      displayName: My Dropdown Selector
      attributeName: myDropdown
      options:
        - option1
        - option2
      isRequired: false
      enableIf: 'true'

    - type: toggleButton
      displayName: My Toggle Button 2
      attributeName: myToggleButton2
      isRequired: false
      enableIf: 'true'

    - type: columnDropdownSelector 
      displayName: Column Name
      attributeName: myColumnDropdown
      isRequired: true
      enableIf: 'true'

    - type: textBox
      displayName: My Textbox
      attributeName: myTextbox
      syntax: sql
      isRequired: true
      enableIf: 'true'
    
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `tabular` | String|
| **displayName** | Name of the tabular config. | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>tabular</figcaption>
</figure>

### Column Selector

Defines a column selector transfer component. The metadata can be referenced at the column-level using the `attributeName`.

```yaml title="columnSelector"
config:
- groupName: 'My Config Group'
  items:
  - type: columnSelector
    displayName: My Column Selector
    attributeName: myColumnSelector
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `columnSelector` | String|
| **displayName** | Name of the columnSelector config. | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>columnSelector</figcaption>
</figure>

### Text Box

Defines a text box and its options, with optional support for SQL syntax highlighting. The metadata can be referenced in the `config` object using the `attributeName`.

```yaml title="textbox"
config:
- groupName: 'My Config Group'
  items:
  - type: textBox
    displayName: My Textbox
    attributeName: myTextbox
    syntax: sql
    isRequired: true
    enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **type**| `textBox` | String|
| **displayName** | Name of the textbox. Pre-SQL and Post-SQL | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **syntax**| `sql` only. The syntax highlighting in text box. | String|
| **isRequired** | If this attribute is required. | Boolean |
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>textBox</figcaption>
</figure>

### Note

Add text anywhere in the config object. This can't be nested in other elements.

| **Name** | **Description** | **Type** |
|---|---|---|
| **type** | `note` | String |
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **displayName** | Where you'll enter your custom text. | String |
| **enableIf** | Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

```yaml title="note"
  - type: note
    attributeName: note.attribute
    displayName: Add text here
    enableIf: "{% if node.materializationType == 'view' %} false {% else %} true {% endif %}"
```

<figure>
  
  <figcaption>note</figcaption>
</figure>

## System Columns

Append additional columns to a node on initialization.

```yaml title="systemColumns"
systemColumns:
- displayName: system_column_name1
  attributeName: isSurrogateKey
  transform: "my sql transform"
  dataType: STRING
  placement: beginning
  nullable: true
  defaultValue: hello
  description: Some description
  enableIf: 'true'
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **systemColumns**| Name of the array| Array|
| **displayName** | Name of the system column. `{{NODE_NAME}}` can be used to reference the name of the Node the column is generated for.| String|
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **transform** | The initial transformation value. For example `CAST(CURRENT_TIMESTAMP AS TIMESTAMP)`.| String|
| **dataType**| SQL data type | String|
| **placement** | `beginning`, `end` Determines placement of system column in relation to normal columns. | String|
| **nullable** | If null is allowed | Boolean|
| **defaultValue**| The default value | String |
| **description** | What goes in the field description| String|
| **enableIf** | If true, display this config group, else hide this config group. Jinja expressions can be used in a quoted boolean for dynamic behavior. For example. `(boolean string, 'true' \| 'false', defaults 'true'\)` | String |

<figure>
  
  <figcaption>systemColumns</figcaption>
</figure>

:::warning[Adding/Modifying System Columns]

System columns are only added to a node when the node is first created on the graph. When adding new system columns to a UDN, these system columns will not immediately be available in the metadata of the current node you are editing.

In order for these changes to take effect, you must create a new node of this type on your graph. You must select this new node in the Node Type Editor to continue authoring it.

:::

## Mapping Columns

Append additional metadata columns to the mapping grid.

```yaml title="mappingColumns"
mappingColumns:
- headerName: My Mapping Column Name
  attributeName: myMappingColumn
  type: textBox
- headerName: Tech Description
  attributeName: techDesc
  type: textBox
- headerName: Ghost Record
  attributeName: ghostRecord
  type: textBox
```

| **Name** | **Description** | **Type** |
|---|---|---|
| **mappingColumns**| Name of the array| Array|
| **headerName** | Name of the system column. | String|
| **attributeName** | A custom name. This is name that will show in the Node metadata and the name you'll use writing code. | String |
| **type** | `textbox` is the only option.| String|

<figure>
  
  <figcaption>mappingColumns</figcaption>
</figure>

[Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Multi-Source Strategy]: /docs/build-your-pipeline/user-defined-nodes/node-config-options#multi-source-strategy-dropdown-selector

---

## Node Definition

Written in YAML, the Node Definition is where you specify the UI elements and other configurations that should be available for Nodes. This includes the Node's color, its materialization options (table or view), business keys, and more.

For a full list, see the [Node Definition Reference][].

<details>
    <summary>**Node Definition Example**</summary>

```yaml
capitalized: Hydrated Dimension
short: DIM
tagColor: '#1E339A'
plural: Dimensions

config:
- groupName: Options
  items:
  - type: materializationSelector
    isRequired: true
    default: table
    options:
    - table
    - view

  - type: multisourceToggle
    enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}"

  - type: businessKeyColumns
    isRequired: true

  - type: changeTrackingColumns
    isRequired: false

  - type: overrideSQLToggle
    enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"

  - displayName: Enable Tests
    attributeName: testsEnabled
    type: toggleButton
    default: true
    
  - displayName: Pre-SQL
    attributeName: preSQL
    type: textBox
    syntax: sql
    isRequired: false

  - displayName: Post-SQL
    attributeName: postSQL
    type: textBox
    syntax: sql
    isRequired: false

- groupName: Dropdown Selector Example
  items:
  - type: dropdownSelector
    displayName: My Dropdown Selector
    attributeName: myDropdown
    default: option1
    options:
    - option1
    - option2
    isRequired: true

systemColumns:

- displayName: '{{NODE_NAME}}_KEY'
  transform: ''
  dataType: NUMBER
  placement: beginning
  attributeName: isSurrogateKey

- displayName: SYSTEM_VERSION
  transform: ''
  dataType: NUMBER
  placement: end
  attributeName: isSystemVersion

- displayName: SYSTEM_CURRENT_FLAG
  transform: ''
  dataType: VARCHAR
  placement: end
  attributeName: isSystemCurrentFlag

- displayName: SYSTEM_START_DATE
  transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
  dataType: TIMESTAMP
  placement: end
  attributeName: isSystemStartDate

- displayName: SYSTEM_END_DATE
  transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
  dataType: TIMESTAMP
  placement: end
  attributeName: isSystemEndDate

- displayName: SYSTEM_CREATE_DATE
  transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
  dataType: TIMESTAMP
  placement: end
  attributeName: isSystemCreateDate

- displayName: SYSTEM_UPDATE_DATE
  transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
  dataType: TIMESTAMP
  placement: end
  attributeName: isSystemUpdateDate

mappingColumns:
- headerName: My Mapping Column Name
  attributeName: myMappingColumn
  type: textBox
```

</details>

## Node Properties

Nodes will always have an auto-generated Node Properties section. It contains:

* **Storage Location** - Logical storage location representing the database and schema which are mapped in the Workspace/Environment's Storage Mappings.
* **Node Type** - The name of the type of node. This could be user-defined or a core option. This can be changed by clicking the pencil next to the Node Type.
* **Deploy Enabled** - This toggle determines if a Node will be included in a deployment. By default, all Nodes in a Workspace are included in a deployment, this allows changing that behavior.

:::warning[Disabling Deploy]

If a Node exists on the target environment and deploy is disabled, the next deployment will cause that Node's table to be dropped.

:::

## Node Definition Config

The `config` section is where you’ll add in UI elements for the user such as a dropdown or business key toggle and then use it in [Macros][], [Create Template][], and [Run Template][].

In the config for the Node below, there is `groupName:Options` which contains the `materializationSelector`, `multisourceToggle`, `businessKeyColumns`, `changeTrackingColumns`, and other options. These correlate to the elements shown in the options. These are pre-built elements you can use.

<figure>
  
  <figcaption>Node definition config</figcaption>
</figure>

### Adding Groups

Groups are where you can organize Node options. For example, the after the group labeled Options, there is a new group called `Dropdown Selector Example`. This adds a new group with a generic drop down selector that requires the user to select information. Using generic elements is a good way to get custom input.

<!-- markdownlint-disable MD046 -->

```yaml title="Dropdown Selector Example"
- groupName: Dropdown Selector Example
  items:
  - type: dropdownSelector
    displayName: My Dropdown Selector
    attributeName: myDropdown
    default: option1
    options:
    - option1
    - option2
    isRequired: true
```

<figure>
  
  <figcaption>Node definition with new group added</figcaption>
</figure>

## System Columns

System columns add additional columns to a Node when first initialized. System columns can be customized using transforms. In the following image you can see multiple system columns in the Node Definition. System columns can be customized such as placement and the display name accepts Jinja.

<figure>
  
  <figcaption>System Columns</figcaption>
</figure>

<figure>
  
  <figcaption>Coalesce Node Editor showing system columns</figcaption>
</figure>

Here is an example of a system column that will appended to the beginning of the columns, so it appears as the first column. These only appear on initialization, so you might need to add a new Node to see the `systemColumns`.

You can use Jinja templating in the Node Definition, here it’s being used to get the Node Name. The `attributeName` is used in the Create or Run template to access the data.

```yaml title="systemColumns"
systemColumns:

- displayName: '{{NODE_NAME}}_CREATED_BY_COALESCE'
  transform: ''
  dataType: NUMBER
  placement: beginning
  attributeName: isCustomName
```

<figure>
  
  <figcaption>System column added with Jinja templating</figcaption>
</figure>

:::warning[Adding and Modifying System Columns]

System columns are only added to a Node when the Node is first created on the graph. When adding new system columns to a user defined Node, these system columns will not immediately be available in the metadata of the current node you are editing.

For these changes to take effect, you must create a new Node of this type on your graph. You must select this new Node in the Node Type Editor to continue authoring it.

:::

## Mapping Columns

Mapping columns append additional metadata to the mapping grid. They are added as a column and are always a textbox.

```yaml title="mappingColumns"
mappingColumns:
- headerName: My Mapping Column Name
  attributeName: myMappingColumn
  type: textBox
```

<figure>
  
  <figcaption>Node Definition with  `My Mapping Column Name` added</figcaption>
</figure>

[Node Definition Reference]: /docs/build-your-pipeline/user-defined-nodes/node-config-options
[Macros]: /docs/reference/macros/
[Create Template]: /docs/build-your-pipeline/user-defined-nodes/create-run-template
[Run Template]: /docs/build-your-pipeline/user-defined-nodes/create-run-template

---

## Using Subgraphs to Build Your Pipeline

Subgraphs help you focus on part of your Node graph without changing how Nodes are defined or how refreshes run. This page covers how to create and edit subgraphs, how they relate to Jobs and Git commits, when to add another subgraph, and naming patterns that keep large Projects manageable. When you are ready to deploy and refresh, follow [Creating and Run Jobs][] from creating the Job through commit, deploy, and run. For a video on subgraphs with filters, extra screenshots, and the Main Graph overview, see [Organizing Your DAG][].

## Subgraphs and Jobs

Subgraphs and Jobs solve different problems. A **Subgraph** is a view in the Build Interface: you work on a subset of Nodes, and each placement is a **reference** to the same Node definition. A **Job** defines which Nodes to include when you deploy and refresh, using selector queries, dragged subgraphs, or both.

They differ in purpose:

- **Subgraphs** - Organize the graph, reduce noise on screen, and make it easier to find related Nodes while you edit.
- **Jobs** - Group Nodes for deployment and scheduled or on-demand refreshes. Jobs are stored in version control like the rest of your Project.

A practical pattern is to shape your graph with subgraphs first, then pull those subgraphs into Jobs when you are ready to run a consistent slice of the pipeline. The Job still refreshes the Nodes you selected, not a separate copy of the data logic.

Use [Creating and Run Jobs][] for the end-to-end procedure: create the Job, add Nodes with selectors or subgraphs (see also [Managing Jobs][] for drag-and-drop details), commit, configure the environment, deploy, and run the refresh. That guide is the place to start whenever you need to create and run Jobs.

## How to Create a Subgraph

You define subgraphs yourself. Creating one adds an empty canvas where you place **references** to Nodes that already exist on the main graph (or you add Nodes from the list, which still points at the same definitions).

1. In the Build Interface, click the `+` plus sign to create a new Subgraph or right click on a Node.
2. Give it a name.
3. Then you can right click to add other Nodes or click and drag Nodes into the subgraph.

Adding a Node to a subgraph does not copy it. The same Node can appear in multiple subgraphs, dependencies stay consistent across views, and edits to that Node apply everywhere it appears.

For more on adding subgraphs, including a video, see [Organizing Your DAG][].

:::info[Large graphs]

[Organizing Your DAG][] shows how to combine filters in the Node list with drag-and-drop when the list is long. Filter syntax is in [Selector Reference][].

:::

## Naming Subgraphs

Clear names make subgraphs easy to find in menus and when you add them to Jobs. Use this guidance:

- **Short, specific labels** - Prefer names that state the scope, for example a domain, mart, or milestone, instead of generic titles.
- **Stable over time** - Rename sparingly if teammates bookmark or reference subgraph names in runbooks.
- **One obvious name per purpose** - Avoid near-duplicate names that differ only by punctuation or spelling so selectors and manual picks stay unambiguous.

## How to Edit a Subgraph

Editing a subgraph means changing which Nodes appear in that view, how you inspect them, or the Node definitions themselves. The underlying Nodes stay shared with the main graph and any other subgraphs that reference them.

1. Open the subgraph from the **Subgraph** list when you want to change membership or layout on the canvas.
2. Rename the subgraph if the scope of work changes. Apply the same patterns as in Naming Subgraphs.
3. Add more Nodes with drag-and-drop from **Nodes**, or choose **Add to Subgraph** from a Node's context menu. If your UI shows **Add to a Subgraph** instead, choose that option. Drag-and-drop works in **Graph**, **Node Grid**, and **Column Grid** views.
4. Switch **View As** inside the subgraph when you want a different layout for the same Nodes:
   - **Graph** - Visual DAG of Nodes and edges.
   - **Node Grid** - Table-oriented editing across Nodes.
   - **Column Grid** - Column-level view for dependency checks across Nodes.
5. Edit a Node by double-clicking it on the canvas or opening it from the sidebar. The [Node Editor][] opens the same way from the main graph or from within a subgraph.

For dependency lines that look wrong in the subgraph view, read [When Visual Dependencies Look Wrong][] and [Fix Un-Linked References in Sub-Graphs in Coalesce][].

## When to Add a Subgraph

Use this section when you are deciding whether to create another subgraph.

### You Want a Stable Bundle for Job Selection

Use a subgraph as a reusable selection when you build or update Jobs.

You can drag subgraphs into the Job tab or express the same scope with selector queries. Either way, what runs in an environment depends on the committed Project, Job configuration, and deployment, not on which subgraph is open in the UI. See [What Gets Committed][] for how subgraph metadata is stored in your repository.

After you pick subgraphs or selectors, open [Creating and Run Jobs][] and work through each step so the Job is committed, deployed, and ready to refresh.

### Your Main Graph Is Hard to Read or Navigate

Use a named subgraph and filters when scrolling the full DAG slows you down.

Focus a subgraph on 1 subject area, layer, or release slice. Combine it with filters in the Node list and **View As** options so you can switch between graph, grid, and column views without losing the same Node set. Step-by-step UI guidance lives in [Organizing Your DAG][].

### You Share a Project Across Teams or Parallel Work Streams

Use Subgraphs named for ownership or feature areas when several people edit the same Workspace.

The same Node can appear in more than one subgraph. Edits always apply to the underlying Node, so everyone sees the same definition regardless of which subgraph they used to open it. Subgraphs don't create duplicate Nodes or isolated copies of logic.

## How Edits, Commits, and Refreshes Interact

A subgraph is a lens on the graph. Adding a Node to a subgraph adds a reference; it doesn't fork the Node. Changes you make to SQL, metadata, or settings apply everywhere that Node appears.

Deployments and refreshes follow the branch and commit you use for your Project, plus the Jobs and environments you configure. Organizing work in subgraphs doesn't bypass version control or change which Nodes belong to the pipeline. To run a refresh after deploy, use the flow in [Creating and Run Jobs][] (scheduler, API, or CLI).

If you adjust a subgraph and that change should affect what a Job refreshes, commit the Job to version control and deploy to the target environment. The [Coalesce Scheduler][] only runs deployed Jobs, so scheduled runs do not pick up subgraph or Job updates until after deploy.

## Filters, Views, and Workspace Tabs

Selectors support a subgraph filter so you can target Nodes that belong to a named subgraph in queries and tooling that accept selector syntax. See [Selector Reference][] for `{subgraph:"..."}` and related options.

:::info[Workspace tabs]

When you use multiple tabs in the Build Interface, the workspace can retain scroll position, filters, and subgraph zoom as you move between tabs, so you spend less time resetting the canvas.

:::

## When Visual Dependencies Look Wrong

Sometimes a subgraph doesn't draw edges between Nodes even though SQL references them. In those cases, use `ref_link()` and `ref_no_link()` or the commented-reference pattern described in [Fix Un-Linked References in Sub-Graphs in Coalesce][]. Background on reference helpers is in [Ref Functions][].

## What's Next?

- [Creating and Run Jobs][] - Create a Job, add Nodes with selectors or subgraphs, commit, deploy, and run refreshes (scheduler, API, or CLI). Start here when you are moving from graph organization to execution.
- [Organizing Your DAG][] - Main graph overview, video, screenshots, filters, and **View As** modes alongside subgraph steps.
- [Node Editor][] - Open and edit Node definitions from the graph or a subgraph.
- [Graphs and Subgraphs][] - Core concepts for the main graph and subgraph references.
- [Managing Jobs][] - Adjust Jobs after creation, including drag-and-drop behavior on the Job tab.
- [Selector Reference][] - Filter syntax including subgraph filters.
- [What Gets Committed][] - Repository layout for subgraphs and other metadata.
- [Fix Un-Linked References in Sub-Graphs in Coalesce][] - Resolve missing lineage edges in subgraph views.

---

[Creating and Run Jobs]: /docs/deploy-and-refresh/refresh/refreshing-an-environment
[Fix Un-Linked References in Sub-Graphs in Coalesce]: /docs/build-your-pipeline/build-how-to/how-to-fix-un-linked-references-in-sub-graphs-in-coalesce
[Graphs and Subgraphs]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs
[Managing Jobs]: /docs/deploy-and-refresh/refresh/managing-jobs
[Node Editor]: /docs/build-your-pipeline/the-build-interface/node-editor
[Organizing Your DAG]: /docs/build-your-pipeline/organizing-your-dag
[Ref Functions]: /docs/reference/ref-functions
[Selector Reference]: /docs/reference/selector
[What Gets Committed]: /docs/git-integration/what-gets-committed
[When Visual Dependencies Look Wrong]: /docs/build-your-pipeline/using-subgraphs#when-visual-dependencies-look-wrong
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce

---

## SQL Annotations Reference

SQL annotations are how you set options on each V2 Node. Instead of using only the mapping grid config panel on every Node, you write `@name` or `@name("value")` directives in the Node SQL file alongside your `SELECT`. When the Node type exposes matching **Options**, you can also set those values in the **Config** tab, and Coalesce keeps the panel and the annotations in sync.

For an annotation to change deployed SQL, the Node type must declare it in the YAML definition, **Create** and **Run** templates must consume the hydrated metadata, and the Node SQL or **Options** must set the value. See [Declare Annotations in the Node Type Definition][] for the full checklist.

The annotation parser accepts `@name` syntax for any identifier you choose. A small number of names are [Reserved Annotations][] with built-in system behavior. Everything else is a [Custom Annotations][]. Custom names work the same way as `@truncateBefore` or `@preSQL`: declare the name in the node type definition, read it in templates, then set it in Node SQL or **Options**.

## Syntax

A boolean annotation has no parameters and produces a `true` value:

```sql
@truncateBefore
@testsEnabled
@isBusinessKey
```

A parameterized annotation takes one or more quoted values in parentheses:

```sql
@materializationType("table")
@insertStrategy("MERGE")
@preSQL("ALTER SESSION SET TIMEZONE = 'UTC'")
```

### Placement

#### Node-Level Placement

Put these annotations before the `SELECT` clause. They set Node configuration in `config.*`:

```sql
@materializationType("table")
@truncateBefore
SELECT ...
```

#### Column-Level Placement

Add these after a column expression in the `SELECT` list. They set column properties in `column.*`:

```sql
SELECT
    ID AS id @isBusinessKey,
    NAME AS name @isChangeTracking,
    STATUS AS status
FROM {{ ref('WORK', 'STG_CUSTOMERS') }}
```

A column can have multiple annotations:

```sql
SELECT
    ID AS id @isBusinessKey @notNull,
    NAME AS name @isChangeTracking @notNull,
    ...
```

## Reserved Annotations

These annotations have built-in system behavior. `@id` and `@nodeType` are auto-populated by Coalesce when a Node is created. You never need to write them manually, and you should not edit them. `@materializationType` is a reserved keyword.

| Annotation | Level | Routes to | Behavior |
| --- | --- | --- | --- |
| `@id("...")` | Node | `node.id` | **Auto-populated. Never change.**Modifying it breaks the Node's identity, version control history, and any references to it. |
| `@nodeType("...")` | Node | `node.nodeType` | **Auto-populated.**Only change this if you are intentionally pointing the Node at a different valid Node type ID in your Workspace. |
| `@materializationType("...")` | Node | `node.materializationType` | Controls how the Node is materialized using `"table"` or `"view"`.Also available as `config.materializationType.parameters[0]` in templates. |

## Custom Annotations

Everything beyond the three reserved annotations is custom. That includes familiar names such as `@truncateBefore` and `@preSQL`, and author-defined names such as `@myLoadMode("full")` or `@regionCode("US")`.

Coalesce does not ship special-case behavior for custom names. If you add `@somename` in SQL but omit `somename` from the node type definition, Coalesce does not treat it as a registered option. The annotation may not hydrate into `config.*` or column metadata, and templates cannot rely on it. The same rule applies to built-in-sounding names like `truncateBefore` and `preSQL`, and to names you invent for your Node type.

See [Declare Annotations in the Node Type Definition][] for the config item and template steps.

### Node-Level Fields in Config

Top-level annotations land in `config.*`:

```sql
@truncateBefore          -- config.truncateBefore = true
@insertStrategy("MERGE") -- config.insertStrategy = { parameters: ['MERGE'] }
@preSQL("ALTER SESSION SET TIMEZONE = 'UTC'")  -- config.preSQL = { parameters: ['ALTER SESSION...'] }
```

### Column-Level Fields on the Column Object

Column annotations are flattened onto the column object at the root level and are accessible directly as `column.isBusinessKey`:

```sql
SELECT
    ID AS id @isBusinessKey,        -- column.isBusinessKey = true
    NAME AS name @isChangeTracking, -- column.isChangeTracking = true
    STATUS AS status @notNull       -- column.notNull = true
FROM {{ ref('WORK', 'STG_CUSTOMERS') }}
```

Parameterized column annotations produce `{ parameters: [...] }` on the column metadata object:

```sql
SELECT
    ID AS id @partitionBy('HASH', '16'),   -- column.partitionBy = { parameters: ['HASH', '16'] }
    AMOUNT AS amount @precision('12', '2') -- column.precision = { parameters: ['12', '2'] }
FROM {{ ref('WORK', 'STG_TRANSACTIONS') }}
```

`@isBusinessKey`, `@isChangeTracking`, `@notNull`, and `@isUnique` are custom annotations. The system does not enforce constraints automatically. They work when the node type definition declares the matching column config or mapping column and your templates check for them.

### Author-Defined Annotation Names

You can introduce new annotation names for a custom V2 Node type. Pick any valid identifier, declare it in the node type YAML, wire it in templates, then use it in Node SQL.

Node-level example: a dropdown in the definition with `attributeName: myLoadMode` maps to `@myLoadMode("full")` in SQL and to `config.myLoadMode` in templates.

```yaml
  - displayName: Load Mode
    attributeName: myLoadMode
    type: dropdownSelector
    default: incremental
    options:
    - incremental
    - full
```

```sql
@myLoadMode("full")

SELECT
    id AS id
FROM {{ ref('STAGING', 'STG_ORDERS') }}
```

```jinja
{% if config.myLoadMode is defined and config.myLoadMode.parameters[0] == 'full' %}
    {{ stage('Truncate Table') }}
    TRUNCATE IF EXISTS {{ ref_no_link(node.location.name, node.name) }};
{% endif %}
```

Column-level custom names follow the same rule: add a `mappingColumns` entry or column selector in the definition, then reference `column.yourAttributeName` in the **Create** template. See [Node Config Options][] for config item types you can bind to annotations.

### Common Annotation Examples

| Annotation | Level | Type | Hydrated to | Purpose |
| --- | --- | --- | --- | --- |
| `@isBusinessKey` | Column | Boolean | `column.isBusinessKey` | Business/primary key columns |
| `@isChangeTracking` | Column | Boolean | `column.isChangeTracking` | SCD change detection columns |
| `@notNull` | Column | Boolean | `column.notNull` | NOT NULL constraint in DDL |
| `@isUnique` | Column | Boolean | `column.isUnique` | Unique constraint columns |
| `@isSurrogateKey` | Column | Boolean | `column.isSurrogateKey` | Surrogate key columns |
| `@truncateBefore` | Node | Boolean | `config.truncateBefore` | Truncate table before INSERT |
| `@testsEnabled` | Node | Boolean | `config.testsEnabled` | Enable test execution |
| `@insertStrategy("...")` | Node | Parameterized | `config.insertStrategy.parameters` | Insert strategy: `INSERT`, `UNION`, `UNION ALL`, `MERGE` |
| `@preSQL("...")` | Node | Parameterized | `config.preSQL.parameters` | SQL to run before the main query |
| `@postSQL("...")` | Node | Parameterized | `config.postSQL.parameters` | SQL to run after the main query |
| `@nodeTests("...")` | Node | Parameterized | `config.nodeTests.parameters` | Node-level data quality tests |
| `@columnTests("...")` | Column | Parameterized | `column.columnTests.parameters` | Column-level data quality tests |

## Sync Annotations With Options

When a Node type declares config items under **Options**, those controls appear on the Node **Config** tab and stay aligned with the matching SQL annotations. You can author from either side:

- Add or edit an annotation in the Node SQL file. After the SQL parses, the matching **Options** control reflects the new value.
- Change a toggle, dropdown, or text field in **Options**. Coalesce inserts, replaces, or removes the matching annotation in the `.sql` file without reformatting the rest of your SQL.
- When the value matches the Node type default, Coalesce omits the annotation from the file. Missing annotations in that case still use the declared default at run and deploy time.

Column-level annotations such as `@isBusinessKey` belong after the column alias in the `SELECT` list. Tabular config items map to repeated annotations with positional parameters, for example one `@hashColumns("COL", "SHA256")` line per row when your Node type defines that shape.

Only declared fields sync. If you write `@somename` in SQL but omit `somename` from the Node type definition, Coalesce does not treat it as a registered **Options** control. See [The V2 Editor][] for how **Config** and **Storage Location** appear in the UI.

## Declare Annotations in the Node Type Definition

Before you use an annotation in Node SQL, add a matching entry to the node type YAML `config` block in **Build Settings > Node Types**. The `attributeName` must match the annotation name without `@`. That rule applies to every annotation you use, including names you define yourself: `@myLoadMode` requires `attributeName: myLoadMode`. Coalesce uses the definition to validate annotations, apply defaults, sync **Options** in the UI, and hydrate `config.*` or column fields for templates.

This applies to node-level options such as `truncateBefore`, `preSQL`, `insertStrategy`, and `testsEnabled`, and to column-level flags your templates read, such as `isBusinessKey`, `notNull`, or custom mapping columns. Reserved annotations `@id` and `@nodeType` are auto-populated; `@materializationType` uses the `materializationSelector` config item when you expose it.

Work through this checklist for each option:

1. Add the config item to the node type definition (toggle, dropdown, text box, column selector, or other type from [Node Config Options][]).
2. Add Jinja in the **Create** and/or **Run** template that reads the hydrated field.
3. Set the value in Node SQL with `@annotationName`, or in **Options** when the definition exposes the control.

The [Getting Started with Node Type V2][] guide includes a full definition plus **Create** and **Run** templates that wire **Truncate Before**, **Pre-SQL**, **Post-SQL**, and related options.

Example fragment for `@truncateBefore`:

```yaml
  - displayName: Truncate Before
    attributeName: truncateBefore
    type: toggleButton
    default: true
```

Example fragment for `@preSQL`:

```yaml
  - displayName: Pre-SQL
    attributeName: preSQL
    type: textBox
    syntax: sql
    isRequired: false
```

`@insertStrategy` accepts `INSERT`, `UNION`, `UNION ALL`, and `MERGE` only. Full replace loads use `@truncateBefore`, or **Truncate Before** in **Options**, not `@insertStrategy("TRUNCATE")`.

## Using Annotations in Templates

You can define any annotation name and reference it in your Create and Run templates. Below are complete templates that consume the annotations from the [Full Example][] below.

### Create Template

This template uses the reserved `node.materializationType` field to branch between table and view DDL, and the custom `col.notNull` field to add NOT NULL constraints:

```sql
{% if node.materializationType == 'table' %}
    {{ stage('Create Table') }}

    CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
    (
        {% for col in columns %}
            "{{ col.name }}" {{ col.dataType }}
            {%- if not col.nullable or col.notNull %} NOT NULL
                {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
            {% endif %}
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%}, {% endif %}
        {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

{% elif node.materializationType == 'view' %}
    {{ stage('Create View') }}

    CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
    (
        {% for col in columns %}
            "{{ col.name }}"
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%}, {% endif %}
        {% endfor %}
    )
    {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
    AS
    {% for source in sources %}
        SELECT
        {% for col in source.columns %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"
            {%- if not loop.last -%}, {% endif %}
        {% endfor %}
        {{ source.join }}
    {% endfor %}

{% endif %}
```

### Run Template

This template consumes `@preSQL`, `@postSQL`, `@truncateBefore`, `@insertStrategy`, `@isBusinessKey`, `@isChangeTracking`, `@testsEnabled`, and `@tests`. It assumes the node type definition declares the matching config items. See [Declare Annotations in the Node Type Definition][].

```sql
{# --- Pre-SQL --- #}
{% if config.preSQL is defined and config.preSQL.parameters is defined -%}
    {% for sql in config.preSQL.parameters -%}
        {{ stage('Pre-SQL ' + loop.index|string) }}
        {{ sql }}
    {% endfor %}
{%- endif %}

{# --- Truncate before insert --- #}
{% if config.truncateBefore %}
    {{ stage('Truncate Table') }}
    TRUNCATE IF EXISTS {{ ref_no_link(node.location.name, node.name) }};
{% endif %}

{# --- Insert Strategy --- #}
{% if config.insertStrategy is defined and config.insertStrategy.parameters is defined %}
    {% if config.insertStrategy.parameters[0] == 'MERGE' %}
        {{ stage('Merge Data') }}
        MERGE INTO {{ ref_no_link(node.location.name, node.name) }} TGT
        USING (
        {% for source in sources %}
            {{ source.cteString }}
            SELECT
            {% for col in source.columns %}
                {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
            {{ source.join }}
        {% endfor %}
        ) SRC
        ON {% for col in columns if col.isBusinessKey %}{% if not loop.first %} AND {% endif %}TGT."{{ col.name }}" = SRC."{{ col.name }}"{% endfor %}
        WHEN MATCHED THEN UPDATE SET
            {% for col in columns if col.isChangeTracking %}
                TGT."{{ col.name }}" = SRC."{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
        WHEN NOT MATCHED THEN INSERT (
            {% for col in columns %}
                "{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
        ) VALUES (
            {% for col in columns %}
                SRC."{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
        );

    {% else %}
        {{ stage('Insert Data') }}
        INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
        (
            {% for col in columns %}
                "{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
        )
        {% for source in sources %}
            {{ source.cteString }}
            SELECT
            {% for col in source.columns %}
                {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
            {% endfor %}
            {{ source.join }}
            {% if not loop.last %}UNION ALL{% endif %}
        {% endfor %};

    {% endif %}
{% else %}
    {{ stage('Insert Data') }}
    INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
    (
        {% for col in columns %}
            "{{ col.name }}"{% if not loop.last %}, {% endif %}
        {% endfor %}
    )
    {% for source in sources %}
        {{ source.cteString }}
        SELECT
        {% for col in source.columns %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
        {% endfor %}
        {{ source.join }}
        {% if not loop.last %}UNION ALL{% endif %}
    {% endfor %};

{% endif %}

{# --- Post-SQL --- #}
{% if config.postSQL is defined and config.postSQL.parameters is defined -%}
    {% for sql in config.postSQL.parameters -%}
        {{ stage('Post-SQL ' + loop.index|string) }}
        {{ sql }}
    {% endfor %}
{%- endif %}

{# --- Tests --- #}
{% if config.testsEnabled -%}
    {% if config.tests is defined and config.tests.parameters is defined -%}
        {% for test in config.tests.parameters -%}
            {{ test_stage(test) }}
            {{ test }}
        {% endfor %}
    {%- endif %}

    {% for column in columns -%}
        {% if column.tests is mapping and column.tests.parameters is defined -%}
            {% for test in column.tests.parameters -%}
                {{ test_stage(column.name + ": Test " + loop.index|string) }}
                {{ test }}
            {%- endfor %}
        {%- endif %}
    {%- endfor %}
{%- endif %}
```

## Full Example

A complete SQL file using reserved and custom annotations.

:::warning[Reserved Node ID and Node Type Fields]

- `@id` and `@nodeType` are managed by Coalesce automatically. Never modify `@id`, because changing it breaks the Node's identity. Don't change `@nodeType` unless you're pointing the Node at a different valid Node type ID in your Workspace.
- `@materializationType` is a reserved annotation that controls how the Node is deployed.
- All other annotations are custom. Declare each one in the node type definition, reference it in **Create** and **Run** templates, then set it in Node SQL. Custom annotations have no effect unless a template consumes them.

:::

```sql
@id("98245936-8e90-468e-a1ed-e3a18e3ec941")
@nodeType("fd252eae-7b6d-4061-91cd-291ceaa52be1")
@materializationType("table")
@insertStrategy("MERGE")
@truncateBefore
@preSQL("ALTER SESSION SET TIMEZONE = 'UTC'")
@postSQL("ALTER SESSION UNSET TIMEZONE")
@testsEnabled
@tests("SELECT * FROM {{ this }} WHERE C_NAME IS NULL", "SELECT * FROM {{ this }} WHERE C_ACCTBAL < 0")

SELECT
    "C_CUSTKEY" AS C_CUSTKEY @isBusinessKey @notNull,
    "C_NAME" AS C_NAME @isChangeTracking @notNull,
    "C_ADDRESS" AS C_ADDRESS @isChangeTracking,
    "C_NATIONKEY" AS C_NATIONKEY,
    "C_PHONE" AS C_PHONE @tests('SELECT * FROM {{ this }} WHERE C_PHONE IS NULL') @notNull,
    "C_ACCTBAL" AS C_ACCTBAL,
    "C_MKTSEGMENT" AS C_MKTSEGMENT,
    "C_COMMENT" AS C_COMMENT
FROM {{ ref('SRC', 'CUSTOMER') }} "CUSTOMER"
```

This produces:

- `node.materializationType = "table"` - Reserved; controls deployment
- `config.insertStrategy = { parameters: ['MERGE'] }` - Custom; consumed by templates
- `config.truncateBefore = true` - Custom; consumed by templates
- `config.preSQL = { parameters: ['ALTER SESSION SET TIMEZONE = ...'] }` - Custom; consumed by templates
- `config.postSQL = { parameters: ['ALTER SESSION UNSET TIMEZONE'] }` - Custom; consumed by templates
- `config.testsEnabled = true` - Custom; consumed by templates
- `config.tests = { parameters: ['SELECT * FROM ...', 'SELECT * FROM ...'] }` - Custom; consumed by templates
- `column.isBusinessKey = true` on `C_CUSTKEY`
- `column.isChangeTracking = true` on `C_NAME`, `C_ADDRESS`
- `column.notNull = true` on `C_CUSTKEY`, `C_NAME`, `C_PHONE`
- `column.tests = { parameters: ['SELECT * FROM ...'] }` on `C_PHONE`

## Accessing Annotation Values in Templates

The format for accessing annotation values depends on the annotation type:

- **Boolean annotations** using names such as `@testsEnabled` and `@truncateBefore` produce `true`. Use directly: `{% if config.testsEnabled %}`.
- **Parameterized annotations** such as `@insertStrategy("MERGE")` produce `{ parameters: ['MERGE'] }`. Access the value with `config.insertStrategy.parameters[0]`.

## Quoting and Escaping

Annotation values can be wrapped in either double quotes or single quotes. Both styles are valid:

```sql
@materializationType("table")
@insertStrategy('MERGE')
```

You can also pass unquoted numbers and the Boolean literals `true` and `false`:

```sql
@threshold(100)
@enabled(true)
```

When your annotation value contains quotes, use the opposite quote style to wrap it:

```sql
@preSQL("ALTER SESSION SET TIMEZONE = 'UTC'")
```

Complex quoting scenarios, including nested quotes and multi-statement `@preSQL` strings with semicolons, may have edge cases. Test your specific use case before you rely on it in production.

## What's Next?

- [Node Type V2 Overview][] for the hub page on SQL-first Nodes.
- [Getting Started with Node Type V2][] to create your first V2 Node type and Node.
- [The V2 Editor][] for a tour of the V2 editing experience.
- [Upgrading from V1 to V2 Node Types][] if you are migrating from deprecated `inputMode: 'sql'`.
- [Troubleshooting and FAQ][] for parse, column, and template issues.

[Reserved Annotations]: #reserved-annotations
[Custom Annotations]: #custom-annotations
[Declare Annotations in the Node Type Definition]: #declare-annotations-in-the-node-type-definition
[Full Example]: #full-example
[Node Config Options]: /docs/build-your-pipeline/user-defined-nodes/node-config-options
[Node Type V2 Overview]: /docs/build-your-pipeline/v2-node-types/
[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes
[The V2 Editor]: /docs/build-your-pipeline/v2-node-types/node-editor
[Upgrading from V1 to V2 Node Types]: /docs/build-your-pipeline/v2-node-types/upgrading-v1-v2
[Troubleshooting and FAQ]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq

---

## Getting Started with Node Type V2

This guide walks you through creating your first V2 Node: you create a V2 Node type, add its definition and **Create** and **Run** templates, add a Node to your pipeline and write SQL, configure behavior with annotations, then validate and deploy.

## Prerequisites

You need the following:

- A Coalesce Workspace connected to Snowflake.
- At least one source Node in your pipeline to `SELECT` from.

## Step 1: Create a V2 Node Type

Before you can create a V2 Node, you need a V2 Node type. Node types are the templates that control how a Node deploys and runs, and they are shared across all Nodes of that type.

Go to **Build Settings > Node Types** and click the **Create Node Type** dropdown. Select the **V2** option. This creates a Node type with `version: 2` set. Give it a descriptive name, for example `SQL Stage`.

The Node type list includes a **Version** column so you can distinguish V1 and V2 types at a glance.

You can also duplicate an existing V2 Node type. The duplicate inherits `version: 2` and the original's templates.

:::info[Configure the node type before you add Nodes]

V2 Nodes are authored in SQL, but the Node type still needs a YAML definition and **Create** and **Run** templates. Declare each annotation you want to use, for example **Truncate Before**, **Pre-SQL**, or **Multi Source Strategy**, in the definition, handle it in templates, then set values in Node SQL or **Options**. Step 2 shows a full example.

:::

## Step 2: Add the Node Type Definition and Create and Run Templates

Open **Build Settings > Node Types**, select your V2 Node type, and configure the definition and templates before you rely on Nodes of that type in the graph. The YAML definition registers which annotations the type supports and which **Options** appear in the UI. The **Create** template emits DDL. The **Run** template emits load logic.

Each config item uses an `attributeName` that matches the annotation name in Node SQL, for example `truncateBefore` for `@truncateBefore`, or any custom name you define, such as `myLoadMode` for `@myLoadMode`. If you add an annotation in SQL but omit it from the definition, or omit the matching branch in a template, the option does not affect deployed SQL.

### Node Type Definition in YAML

Paste or sync a definition like this so the Node type uses SQL input mode and CTE support. Your Workspace assigns the Node type `id` when you save in the Coalesce App.

```yaml
capitalized: Copy of Stage
short: STG
plural: Stages
tagColor: '#2EB67D'

config:
- groupName: Options
  items:
  - type: materializationSelector
    default: table
    options:
    - table
    - view
    isRequired: true

  - type: multisourceToggle
    enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}" 

  - type: overrideSQLToggle
    enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"
    
  - displayName: Multi Source Strategy
    attributeName: insertStrategy
    type: dropdownSelector
    default: INSERT
    options:
    - "INSERT"
    - "UNION"
    - "UNION ALL"
    isRequired: true
    enableIf: "{% if node.isMultisource %} true {% else %} false {% endif %}"

  - displayName: Truncate Before
    attributeName: truncateBefore
    type: toggleButton
    default: true

  - displayName: Enable Tests
    attributeName: testsEnabled
    type: toggleButton
    default: true
    
  - displayName: Pre-SQL
    attributeName: preSQL
    type: textBox
    syntax: sql
    isRequired: false

  - displayName: Post-SQL
    attributeName: postSQL
    type: textBox
    syntax: sql
    isRequired: false
```

### Create Template

<details>
    <summary>Create Template</summary>

    ```jinja
    {#
        Copyright (c) 2026 Coalesce. All rights reserved.
    This script and its associated documentation are confidential and proprietary to Coalesce.
    Unauthorized reproduction, distribution, or disclosure of this material is strictly prohibited.
    Coalesce permits you to copy and modify this script for the purposes of using with Coalesce but
    does not permit copying or modification for any other purpose.
    #}
    {# == Node Type Name           : SQL Insert  == #}
    {# == Node Type Description    : This node creates work table or view  == #}
    {#---------------------------------------------------------------------------------------------#}

    {%- if node.materializationType | lower in ['table', 'transient table'] %}
        {{ stage('Create ' + node.materializationType ) }}

        {# CreateSQL for Table #}
        CREATE OR REPLACE {{ node.materializationType }} {{ ref_no_link(node.location.name, node.name) }}
        (
            {%- for col in columns %}

                {%- set nullParams = col.nullable.parameters | default([]) %}
                {%- set defaultValueParams = col.defaultValue.parameters | default([]) %}
                {%- set descriptionParams = col.description.parameters | default([]) %}

                "{{ col.name }}" {{ col.dataType }}
                {%- if nullParams and (nullParams[0] in ['false', false]) %} NOT NULL {%- endif %}
                {%- if defaultValueParams and defaultValueParams[0] is not none %} DEFAULT {{ get_default(col.dataType, defaultValueParams[0]) }} {%- endif %}
                {%- if descriptionParams and descriptionParams[0] | length > 0 %} COMMENT '{{ descriptionParams[0] | escape }}' {%- endif %}
                {%- if not loop.last -%}, {%- endif %}
            {%- endfor %}
        )
        {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}' {%- endif %}

    {%- elif node.materializationType | lower == 'view' %}

        {# CreateSQL for View #}
        {{ stage('Create ' + node.materializationType ) }}

        CREATE OR REPLACE {{ node.materializationType }} {{ ref_no_link(node.location.name, node.name) }}
        (
            {%- for col in columns %}
                {%- set descriptionParams = col.description.parameters | default([]) %}

                "{{ col.name }}"
                {%- if descriptionParams and descriptionParams[0] | length > 0 %} COMMENT '{{ descriptionParams[0] | escape }}' {%- endif %}
                {%- if not loop.last -%}, {%- endif %}
            {%- endfor %}
        )
        {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}' {%- endif %}
        AS
        {{ sources[0].cteString }}
        SELECT {%- if config.selectDistinct %} DISTINCT {%- endif %}
        {%- for col in sources[0].columns %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"
            {%- if not loop.last -%}, {%- endif %}
        {%- endfor %}

        {{ sources[0].join }}

    {%- endif %}
    ```
</details>

### Run Template

<details>
    <summary>Run Template</summary>

    ```sql
    {#
        Copyright (c) 2026 Coalesce. All rights reserved.
    This script and its associated documentation are confidential and proprietary to Coalesce.
    Unauthorized reproduction, distribution, or disclosure of this material is strictly prohibited.
    Coalesce permits you to copy and modify this script for the purposes of using with Coalesce but
    does not permit copying or modification for any other purpose.
    #}
    {# == Node Type Name           : SQL Insert  == #}
    {# == Node Type Description    : Creates a table with custom SQL logic  == #}
    {#---------------------------------------------------------------------------------------------#}

    {# == To run data quality tests before data insertion == #}

    {%- if node.materializationType | lower in ['table', 'transient table'] %}

        {# --- Pre-SQL --- #}
        {%- if config.preSQL is defined and config.preSQL.parameters is defined -%}
            {%- for sql in config.preSQL.parameters -%}
                {{ stage('Pre-SQL ' + loop.index|string) }}
                {{ sql }}
            {%- endfor %}
        {%- endif %}

        {# == Truncate data before data insertion  == #}
        {%- if config.truncateBefore %}
            {{ stage('Truncate ' + node.materializationType ) }}
            TRUNCATE IF EXISTS {{ ref_no_link(node.location.name, node.name) }}
        {%- endif %}

        {# == Insert data from sources into Work table  == #}

        {{ stage('Load ' + node.materializationType + ' Using Insert') }}
        INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
        (
            {%- for col in sources[0].columns %}
                "{{ col.name }}"
                {%- if not loop.last -%},{%- endif %}
            {%- endfor %}
        )
        {{ sources[0].cteString }}
        SELECT {%- if config.selectDistinct %} DISTINCT {%- endif %}
        {%- for col in sources[0].columns %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"
            {%- if not loop.last -%}, {%- endif %}
        {%- endfor %}

        {{ sources[0].join }}
        {% if config.groupByAll %} GROUP BY ALL {% endif %}

        {# --- Post-SQL --- #}
        {%- if config.postSQL is defined and config.postSQL.parameters is defined -%}
            {%- for sql in config.postSQL.parameters -%}
                {{ stage('Post-SQL ' + loop.index|string) }}
                {{ sql }}
            {%- endfor %}
        {%- endif %}

    {%- else %}
        {{ stage('Load Skipped for View') }}
        -- The node {{node.name}} is materialized as View. Therefore, a Load operation is not supported.
        SELECT 1 AS INFO_MESSAGE WHERE FALSE
    {%- endif %}

    {# == To run data quality tests after data insertion == #}
    ```
</details>

Coalesce does not apply annotations unless the node type declares them and your templates read the matching metadata. See [SQL Annotations Reference][] for the annotation list and the declare-and-wire checklist.

## Step 3: Add a Node and Write Your SQL

1. In the Build interface, add a new Node to your graph.
2. Select your V2 Node type as the Node type.
3. Use the SQL editor in the center panel instead of the mapping grid. If you attach downstream from an upstream Node, the editor can pre-populate with a starter `SELECT`.

V2 Nodes are stored as `.sql` files. The file name follows `location-name.sql`, where `location` is the storage location and `name` is the Node name. V1 Nodes use `.yml` files instead.

Write or paste a standard `SELECT` statement. Use `{{ ref('LOCATION', 'NODE_NAME') }}` to reference upstream Nodes so Coalesce can build the dependency graph and lineage.

```sql
SELECT
    o.O_ORDERKEY AS order_key,
    o.O_CUSTKEY AS customer_key,
    o.O_ORDERSTATUS AS order_status,
    o.O_TOTALPRICE::DECIMAL(12,2) AS order_total,
    o.O_ORDERDATE AS order_date
FROM {{ ref('STAGING', 'STG_ORDERS') }} o
```

When you save, Coalesce parses the `SELECT` and the **Columns** tab lists each column with its inferred name and data type.

Confirm on the **Columns** tab that every column you expect is present with the right data types. If parsing fails for a type, add an explicit `CAST` in your SQL.

:::tip[Reserved annotations]

Coalesce adds `@id` and `@nodeType` when you create the Node. Do not edit them. See [SQL Annotations Reference][].

:::

## Step 4: Configure the Node with Annotations

V2 Nodes do not use the mapping grid. You configure behavior with SQL annotations in the Node file and with **Options** when Step 2 declared the matching config items. Your **Create** and **Run** templates must read those fields, for example `config.truncateBefore` or `config.preSQL.parameters`.

```sql
@materializationType("table")
@truncateBefore
@preSQL("ALTER SESSION SET TIMEZONE = 'UTC'")

SELECT
    o.O_ORDERKEY AS order_key @isBusinessKey,
    o.O_CUSTKEY AS customer_key,
    o.O_ORDERSTATUS AS order_status
FROM {{ ref('STAGING', 'STG_ORDERS') }} o
```

Use `@truncateBefore` for full replace loads, not `@insertStrategy("TRUNCATE")`. Valid `@insertStrategy` values are `INSERT`, `UNION`, `UNION ALL`, and `MERGE`.

See [SQL Annotations Reference][] for the full annotation list and template wiring.

## Step 5: Validate Select, Create, and Run

1. **Validate Select** checks the Node SQL and shows compiled SQL before you deploy.
2. **Create** deploys the Node and runs the **Create** template DDL in your warehouse.
3. **Run** executes the **Run** template DML and loads the table.

After running, check that lineage and data look correct:

- **DAG view** - upstream dependencies from your `{{ ref() }}` calls appear in the graph.
- **Warehouse** - query the target table to confirm data.

## Full Example

This example uses sample data: it joins orders and line items, uses CTEs, and sets a business key with an annotation.

```sql

WITH orders_base AS (
    SELECT
        O_ORDERKEY AS order_key,
        O_CUSTKEY AS customer_key,
        O_ORDERSTATUS AS order_status,
        O_TOTALPRICE AS order_total_price,
        O_ORDERDATE AS order_date,
        TRIM(O_ORDERPRIORITY) AS order_priority
    FROM {{ ref('STAGING', 'STG_ORDERS') }} "ORDERS"
),

lineitem_base AS (
    SELECT
        L_ORDERKEY AS order_key,
        L_LINENUMBER::NUMBER AS line_number,
        L_QUANTITY AS quantity,
        L_EXTENDEDPRICE AS extended_price,
        L_DISCOUNT AS discount_percent,
        L_EXTENDEDPRICE * (1 - L_DISCOUNT) AS discounted_price
    FROM {{ ref('STAGING', 'STG_LINEITEM') }} "LINEITEM"
)

SELECT
    o.order_key @isBusinessKey,
    li.line_number,
    o.customer_key,
    o.order_date,
    o.order_status,
    o.order_priority,
    li.quantity,
    li.extended_price,
    li.discount_percent,
    li.discounted_price
FROM orders_base o
INNER JOIN lineitem_base li
    ON o.order_key = li.order_key
```

After saving, the **Columns** tab shows 10 columns with inferred types. The DAG lists dependencies on `STG_ORDERS` and `STG_LINEITEM`. **Create** builds the table and **Run** loads it.

## Best Practices

- **Alias your columns.** Explicit aliases give the parser the clearest signal for column names.
- **Cast ambiguous types.** If a column's data type shows as `UNKNOWN`, add an explicit cast in your SQL: `value::TIMESTAMP_NTZ` or `CAST(value AS DECIMAL(12,2))`.
- **Start simple.** Write a basic `SELECT` first, confirm columns parse correctly, then add CTEs, window functions, or annotations.

## What's Next?

- [SQL Annotations Reference][] for the full annotation list.
- [The V2 Editor][] for a walkthrough of every part of the interface.
- [Node Type V2][] for how V2 compares to V1, platform scope, and links across the section.
- [Troubleshooting and FAQ][] for common parse, column, and lineage issues.

[SQL Annotations Reference]: /docs/build-your-pipeline/v2-node-types/annotations-reference
[The V2 Editor]: /docs/build-your-pipeline/v2-node-types/node-editor
[Node Type V2]: /docs/build-your-pipeline/v2-node-types/
[Troubleshooting and FAQ]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq

---

## Node Type V2

Node Type V2 is a SQL-first authoring experience. You write a SQL `SELECT` in the Coalesce editor, in an IDE, or with an AI assistant. Coalesce infers columns, data types, and upstream dependencies. Per-Node options such as truncate-before-insert, Pre-SQL, and insert strategy are expressed as inline annotations in the SQL file.

Node types still use a YAML definition plus **Create** and **Run** templates. You declare which annotations a type supports in the definition, wire them in templates, then set values in each Node's SQL, or in **Options** when the definition exposes toggles and fields. See [How V2 Annotations Reach Deployed SQL][].

## When V2 Fits Your Workflow

The mapping grid still works well for visual, low-code modeling. Choose V2 when SQL is the natural starting point:

- **Migrating SQL into Coalesce** - Bring existing SQL, dbt models, or stored procedure logic without splitting it into per-column mappings.
- **CTE-heavy logic** - Represent CTEs inside a single Node, which the mapping grid does not support.
- **AI-generated SQL** - Check in full `SELECT` statements and keep lineage and governance in Coalesce.
- **Code-first workflows** - Version transformations as plain `.sql` files next to the rest of your code.

## How V1 and V2 Differ

Use this table to compare authoring and storage. Execution, templates, and pipeline behavior are covered in the next section.

| | V1 Node Type | Node Type V2 |
| --- | --- | --- |
| **Authoring** | Mapping grid | SQL editor |
| **File format** | `.yml` | `.sql` |
| **Column definition** | Manually mapped in the grid | Inferred from your `SELECT` clause; the column list is read-only in the UI |
| **Source dependencies** | Selected in the UI | Declared with `{{ ref() }}` in SQL |
| **Per-Node configuration** | Config panel on each Node | Inline SQL annotations and **Options**; see [SQL Annotations Reference][] |
| **CTE support** | Not available inside a single Node | Supported natively |
| **Node type specs** | Config items in the YAML definition | Config items in the YAML definition declare which annotations the type supports; templates consume them |

V1 and V2 coexist in the same Workspace and pipeline. You choose the version per Node type, and `{{ ref() }}` works the same whether the upstream Node is V1 or V2.

:::warning[V1 inputMode SQL setting is deprecated]

The older `inputMode: 'sql'` setting on V1 Node type definitions is deprecated and will be removed in a future release. Node types that use it show a deprecation warning in the Coalesce App. See [Upgrading from V1 to V2 Node Types][] to migrate.

:::

## How V2 Annotations Reach Deployed SQL

An annotation in Node SQL does not change warehouse behavior by itself. Work through these layers for every option you want to use, for example `@truncateBefore`, `@preSQL`, `@insertStrategy`, or column flags such as `@isBusinessKey`:

1. **Node type definition** - Declare the option as a config item in the YAML definition so Coalesce registers the annotation, applies defaults, and can show **Options** controls. Match `attributeName` to the annotation name, for example `truncateBefore` for `@truncateBefore`, or any name you define, such as `myLoadMode` for `@myLoadMode`. See [Node Config Options][] and [Getting Started with Node Type V2][].
2. **Create and Run templates** - Read the hydrated metadata in Jinja, for example `config.truncateBefore`, `config.preSQL.parameters`, or `column.isBusinessKey`, and emit the matching DDL or load stages.
3. **Node SQL** - Set the annotation on the Node, or use **Options** when the definition exposes the field.

If any layer is missing, the annotation may parse but the compiled plan can omit the stage you expect. That can surface as a successful run with no error, for example insert-only loads when truncate was intended.

## What Stays the Same in Your Pipeline

Templates (Create, Run, Join, Macro), deployment, execution, DAG and column-level lineage, testing, and governance policies work the same as for other Node types.

## Data Platform Support

Node Type V2 is supported on Snowflake.

## What's Next?

- [The V2 Editor][] for a tour of the interface.
- [Getting Started with Node Type V2][] to create your first V2 Node type and Node.
- [SQL Annotations Reference][] for syntax, node type definition, and template wiring.
- [Upgrading from V1 to V2 Node Types][] if you need to leave deprecated `inputMode: 'sql'`.
- [Troubleshooting and FAQ][] for common authoring issues.

[How V2 Annotations Reach Deployed SQL]: #how-v2-annotations-reach-deployed-sql
[Node Config Options]: /docs/build-your-pipeline/user-defined-nodes/node-config-options
[The V2 Editor]: /docs/build-your-pipeline/v2-node-types/node-editor
[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes
[SQL Annotations Reference]: /docs/build-your-pipeline/v2-node-types/annotations-reference
[Upgrading from V1 to V2 Node Types]: /docs/build-your-pipeline/v2-node-types/upgrading-v1-v2
[Troubleshooting and FAQ]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq

---

## The V2 Editor

When you open a Node built on a V2 Node type, you author in a SQL-first editor instead of the V1 mapping grid. This page explains each part of the interface and how it connects to your `.sql` file.

## Build Settings

When creating a Node, you'll have the option of creating a V1 Node or V2 Node. You'll also be able to see the type of other existing Nodes.

## SQL Editor

The center panel opens on the **SQL Editor** tab, where you write your `SELECT` statement. Switch to **Mapping** to review the columns Coalesce inferred from that SQL. The Node is stored as a `.sql` file. The file name combines the **Storage Location** and Node name, for example `WORK-STG_CUSTOMERS.sql`.

The first two lines of every V2 Node file are reserved annotations that Coalesce manages automatically:

```sql
@id("1d181c4d-ac7d-4721-bc39-5ac6be21ac79")
@nodeType("10")
```

:::warning[Reserved header annotations]

Do not edit `@id` or `@nodeType`. `@id` identifies the Node. Changing it breaks the Node's identity, version control history, and references to it. `@nodeType` must map to a valid Node Type ID in your Workspace. Change it only when you are intentionally pointing the Node at a different valid Node Type.

:::

Below the reserved annotations, you add your SQL: CTEs, the final `SELECT`, and any inline annotations for configuration:

```sql
@id("...")
@nodeType("...")
@materializationType("table")

WITH source_data AS (
    SELECT
        C_CUSTKEY,
        TRIM(C_NAME) AS C_NAME,
        TRIM(C_ADDRESS) AS C_ADDRESS
    FROM {{ ref('TARGET', 'STG_CUSTOMER') }}
)

SELECT
    C_CUSTKEY AS customer_key @isPrimaryKey,
    C_NAME AS customer_name @isChangeTracking @notNull,
    C_ADDRESS AS customer_address @isChangeTracking @isUnique,
    C_NATIONKEY AS nation_key @foreignKey('DIM_NATION'),
    C_ACCTBAL::DECIMAL(12,2) AS account_balance @isChangeTracking,
    C_MKTSEGMENT AS market_segment
FROM source_data
```

Node-level annotations such as `@materializationType("table")` go before the `SELECT`. Column-level annotations such as `@isPrimaryKey`, `@isChangeTracking`, and `@notNull` go directly after the column expression in the `SELECT` list.

For every reserved name, quoting rule, and template hook, see the [SQL Annotations Reference][].

## Config Panel

On V1 Nodes, the **Config** tab holds dropdowns, toggles, and text fields for materialization, insert strategy, `preSQL` and `postSQL`, and other settings.

On V2 Nodes, open **Config** on the right. **Node Properties** and **Options** group the controls for this Node. **Options** shows the fields your Node type declares, such as **Create As**, **Truncate Before**, **Enable Tests**, and **Pre-SQL** or **Post-SQL**. You can set the same values either in **Options** or as SQL annotations. Coalesce keeps them in sync:

- Edit an annotation in the SQL editor, such as `@truncateBefore(false)`. After the SQL parses, the matching **Options** control updates.
- Change a value in **Options**. Coalesce inserts, replaces, or removes the matching annotation in the SQL file and leaves the rest of your SQL formatting unchanged.
- When a value matches the Node type default, Coalesce omits the annotation from the `.sql` file so the SQL stays clean.

Only fields declared on the Node type participate in this sync. See [SQL Annotations Reference][] for the declare-and-wire checklist and how defaults affect which annotations appear.

<DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>V2 Node with SQL Editor and Mapping tabs. Config Options stay in sync with SQL annotations.</figcaption>
    </figure>
    <figure>
    
    <figcaption>V1 Node Config panel for comparison with the V2 SQL-first editor.</figcaption>
    </figure>
</DocCarousel>

## Mapping

Open the **Mapping** tab next to **SQL Editor** to review the columns Coalesce inferred from your `SELECT` clause. Each column appears with its name and data type.

Mapping is a read-only view of what your SQL defines. Your SQL remains the source of truth. To add, remove, or rename a column, edit the `SELECT` statement on the **SQL Editor** tab, then return to **Mapping** to confirm the updated list.

Scroll horizontally within a row to see that column's lineage alongside its name and data type.

:::info[UNKNOWN data types]

If a column's data type shows as `UNKNOWN`, Coalesce could not infer the type from the expression. Add an explicit cast in your SQL, for example `CAST(value AS DECIMAL(12,2))` or `value::TIMESTAMP_NTZ`.

:::

## Column Lineage

In **Mapping**, scroll within a column row to see how that column flows from its source through CTEs to the final `SELECT`.

For example, for `C_NAME` in the sample above, lineage can show how values move through each CTE step, from the upstream column through any functions such as `TRIM`, then into the final projection.

Column lineage behaves the same as for other Coalesce Nodes. For a V2 Node, the parser rebuilds the path from your CTE chain instead of from per-column transforms in the V1 mapping grid.

## Code and Preview Pane

The collapsible pane at the bottom of the V2 editor shows compiled SQL, run output, and a sample of table data. Use the toolbar above the pane to validate, create, and run the Node, then review results without leaving the editor.

### Save Status

Next to the action buttons, a status pill shows whether your latest SQL or Config edits have been saved:

| Status | Meaning |
| ------ | ------- |
| **Synced** | No pending edits since you opened the Node. What you see already matches the saved Node. |
| **Saving** | Coalesce is writing your changes. This can appear briefly after you stop typing while the save completes. |
| **Saved** | Your latest change finished saving. After the first save in a session, the pill toggles between **Saving** and **Saved**. |

**Validate Select**, **Create**, and **Run** wait for any in-progress save to finish before they run, so they always use your latest SQL and Options.

### Node Actions

These actions work like the V1 Results and Data Pane controls. For the full behavior of each mode, including how validation wraps SQL with `EXPLAIN`, see [The Node Editor][].

- **Validate Select** compiles a `SELECT` from your Node SQL and shows the compiled statement in the Results view without writing to the warehouse.
- **Create** runs the Node Type **Create** template to apply DDL in your warehouse. Use the menu on the button for **Validate Create** when you want to compile without executing.
- **Run** runs the Node Type **Run** template to apply DML and load data. Use the menu on the button for **Validate Run** when you want to compile without executing.

### Preview Data

After a successful run, the Data Viewer can show a sample of the target table. Click **Load Preview** to fetch sample rows without re-running the Node. Project and session Auto-Preview settings control whether that sample loads automatically. See [Auto-Preview][].

## What's Next?

- [Getting Started with Node Type V2][] for an end-to-end first Node.
- [SQL Annotations Reference][] for every supported annotation and template pattern.
- [Node Type V2][] for how V2 compares to V1, platform scope, and links to the rest of the section.
- [Troubleshooting and FAQ][] for common parse, column, and lineage issues.

[SQL Annotations Reference]: /docs/build-your-pipeline/v2-node-types/annotations-reference
[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes
[Node Type V2]: /docs/build-your-pipeline/v2-node-types/
[Troubleshooting and FAQ]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq
[The Node Editor]: /docs/build-your-pipeline/the-build-interface/node-editor
[Auto-Preview]: /docs/build-your-pipeline/the-build-interface/auto-preview

---

## Troubleshooting and FAQ for Node Type V2

Use this page when a V2 Node does not parse, columns look wrong, lineage is missing, DAG edges do not appear, or you need the current limits and answers to common questions.

## Troubleshooting

### Columns Not Appearing After Saving

You save your SQL but the **Mapping** tab shows no columns, or fewer columns than you expect. Work through these causes:

- **Missing aliases on expressions** - Coalesce infers column names from your `SELECT` clause. Complex expressions without an alias, for example `L_EXTENDEDPRICE * (1 - L_DISCOUNT)`, can produce an unexpected name or fail to parse. Add an explicit alias: `L_EXTENDEDPRICE * (1 - L_DISCOUNT) AS discounted_price`.
- **Parse error earlier in the file** - A syntax error above your final projection can stop the parser from reading downstream columns. Look for an error indicator in the editor, fix the issue, and save again.
- **`SELECT *` from a CTE or upstream Node** - `SELECT *` records Node-level lineage and does not populate **Mapping** with named columns the way explicit projections do. List column names when you need column-level lineage in **Mapping**.

### Column Data Type Shows as UNKNOWN

The **Mapping** tab shows `UNKNOWN` as the data type for one or more columns when Coalesce cannot infer a type from a complex expression. Add an explicit cast in your SQL:

```sql
value::TIMESTAMP_NTZ
CAST(value AS DECIMAL(12,2))
```

### Ref Helper Not Appearing as a DAG Dependency

An upstream Node you reference with `{{ ref() }}` does not show as a connected dependency in the DAG view, even though the warehouse runs the SQL.

`{{ ref() }}` calls inside deeply nested subqueries do not always register as DAG dependencies. The warehouse still runs the SQL, but the Coalesce parser does not detect the reference.

Restructure the query so the ref sits in a top-level CTE:

```sql
-- Harder for the parser to detect:
SELECT *
FROM (
    SELECT order_id FROM {{ ref('STAGING', 'STG_ORDERS') }}
) sub

-- Easier for the parser to detect:
WITH orders AS (
    SELECT order_id FROM {{ ref('STAGING', 'STG_ORDERS') }}
)
SELECT * FROM orders
```

### Parse Error on Save

Coalesce shows an error with a line and column number when you save. Your SQL stays in the editor, and a parse error does not roll back previously compiled output or the deployed table. Fix the issue and save again.

Common causes include mismatched parentheses, an unclosed CTE block, or Jinja outside the supported `ref` pattern. For what is supported today, read [Jinja Templates Beyond Ref][] in Known Limitations.

### Annotations Not Taking Effect

You added an annotation but the Node does not behave the way you expect. Examples include `@insertStrategy("MERGE")` without MERGE in the compiled SQL, `@preSQL` with no Pre-SQL stage, or `@truncateBefore` with insert-only runs and growing row counts.

Work through these layers in order:

1. **Node type definition** - Confirm the annotation appears as a config item with the correct `attributeName`, for example `truncateBefore` for `@truncateBefore`. See [Declare Annotations in the Node Type Definition][] in the annotations reference.
2. **Create and Run templates** - Confirm the template branches on the hydrated field, for example `{% if config.truncateBefore %}`, `config.preSQL.parameters`, `config.insertStrategy.parameters[0] == 'MERGE'`, or `column.isBusinessKey`.
3. **Node SQL or Options** - Set the annotation in the Node file or toggle the matching **Options** control when the definition exposes it.

Parsing writes values into metadata, but templates decide what SQL to emit. If the definition or template is missing, the run can succeed while omitting the stage you expected.

Watch for these common mistakes:

- Using `@insertStrategy("TRUNCATE")` for full replace loads. Valid values are `INSERT`, `UNION`, `UNION ALL`, and `MERGE`. Use `@truncateBefore` with `@insertStrategy("INSERT")` instead.
- Adding `@truncateBefore` or `@preSQL` in SQL when the node type definition never declared `truncateBefore` or `preSQL`.
- Using a custom `@somename` annotation in SQL without a matching config item in the node type definition (`attributeName: somename`).
- Expecting column flags such as `@isBusinessKey` to drive MERGE when the Run template does not reference `column.isBusinessKey`.

After you change the definition or templates, re-validate **Create** and **Run** and confirm the compiled plan lists the stages you expect, for example **Truncate** before **Insert Data**.

See [SQL Annotations Reference][] for template examples and [Getting Started with Node Type V2][] for a wired node type.

### Lineage Missing Through Subqueries

Column-level lineage is incomplete or missing for columns that flow through a subquery.

Use CTEs instead of deeply nested subqueries. The parser traces lineage through CTE steps and does not always follow lineage through nested subqueries. A CTE chain usually restores column-level lineage.

## Known Limitations

Sections note whether a limit is by design, a known limitation, or planned for a future release.

### Jinja Templates Beyond Ref

This capability is planned. The SQL editor supports `{{ ref() }}` for upstream references. General Jinja, including custom macros, package macros, environment variables, and conditional blocks, is not supported. If you rely on patterns such as `{{ env.database }}`, use a mapping-grid Node type for those Nodes.

### Column Propagation From Upstream Nodes

This behavior is by design. When columns are added to an upstream Node, they do not automatically flow into downstream V2 Nodes. Your SQL is the source of truth. Add the new columns to your `SELECT`, or use `SELECT *` for broader propagation with the trade-off of limited column-level lineage in the grid.

### Lineage and Refs Inside Subqueries

This is a known limitation. `{{ ref() }}` inside subqueries can fail to register as DAG dependencies, so upstream Nodes might not show as connected in lineage views even though execution succeeds. Prefer CTEs so refs resolve and lineage stays consistent.

### Copilot Integration

This capability is planned. Copilot does not support creating or editing V2 Nodes.

### Platform Support

Node Type V2 is supported on Snowflake. Databricks and BigQuery are not supported for V2.

### No Mapping To SQL Conversion

This behavior is by design. There is no automated conversion from a mapping-grid V1 Node to a V2 Node. Add a new Node that uses a V2 Node type and rewrite the transformation in SQL. See [Upgrading from V1 to V2 Node Types][].

## FAQ

### Can You Mix V1 and V2 Nodes in the Same Pipeline?

Yes. Node Type V2 is additive. It does not remove mapping-grid Node types. Both run in the same Workspace and pipeline. You pick the version per Node type. V1 and V2 Nodes connect through `{{ ref() }}` like any other Nodes.

### Do You Lose Lineage When Using V2 Nodes?

No. Coalesce parses `{{ ref() }}` to maintain DAG lineage, including column-level lineage for explicitly named columns. `SELECT *` records Node-level lineage only.

### Can You Convert an Existing Mapping-Grid Node to V2?

Not automatically. Create a new Node on a V2 Node type and rewrite the transformation in SQL. Follow [Upgrading from V1 to V2 Node Types][].

### What Happens on a Parse Error?

Coalesce shows an error with line and column information when it is available. Your SQL stays in the editor. Parse errors do not change previously compiled output or the deployed table on their own. Fix the error and save again.

### Do You Need to Use the Ref Helper?

Yes. `{{ ref('STORAGE_LOCATION', 'NODE_NAME') }}` is how Coalesce tracks dependencies and builds the DAG. Hard-coded table names do not create edges in the graph, and Job ordering does not treat them as declared dependencies.

### How Do Annotations Work?

In V2, you set per-Node options with SQL annotations and **Options** when the Node type definition declares them. The Node type YAML registers which annotations exist; **Create** and **Run** templates read `config.*` and column fields; Node SQL or **Options** supplies the values. Reserved `@id` and `@nodeType` are auto-populated. See [SQL Annotations Reference][] and [Getting Started with Node Type V2][].

### Is There a Query Size Limit?

There is no fixed character cap. Very large queries with hundreds of columns can take longer to parse.

### How Is V2 Different From Override Create SQL?

The V2 SQL editor replaces the mapping grid. You write a `SELECT` and Coalesce derives columns from it. Override Create SQL replaces the generated CREATE DDL and applies to views in that workflow. V2 is for authoring transformations. Override Create SQL is for customizing DDL output.

### Can You Use V2 With Custom Node Types?

Yes. Create a V2 Node type from the **Create Node Type** menu under **Build Settings > Node Types**, or duplicate an existing V2 type. If you use custom templates, add `{{ source.cteString }}` to the Run template when you need CTE text passed through.

### What SQL Constructs Are Supported?

CTEs, JOINs, window functions, aggregations, CASE expressions, UNION, subqueries, casting, and most standard Snowflake SQL you would run in a `SELECT` pipeline are supported.

### What About the inputMode SQL Setting?

The `inputMode: 'sql'` setting on V1 Node type definitions is deprecated and will be removed in a future release. Node types that still use it show a deprecation warning in the Coalesce App. Migrate those definitions to `version: 2`. See [Upgrading from V1 to V2 Node Types][].

### Can You Use Stored Procedures?

The main body of the file must be a `SELECT`. You can run procedure calls from a `@preSQL` annotation, for example `@preSQL("CALL my_schema.my_proc()")`. If the Node must deploy procedure DDL, handle that in custom templates.

## What's Next?

- [Node Type V2][] for overview, comparison to V1, and links across the section.
- [Getting Started with Node Type V2][] for a guided first Node.
- [The V2 Editor][] for the SQL workspace, Mapping tab, and actions.
- [SQL Annotations Reference][] for reserved names, quoting, and template wiring.
- [Upgrading from V1 to V2 Node Types][] if you are migrating from deprecated `inputMode: 'sql'`.

[Declare Annotations in the Node Type Definition]: /docs/build-your-pipeline/v2-node-types/annotations-reference#declare-annotations-in-the-node-type-definition
[Node Type V2]: /docs/build-your-pipeline/v2-node-types/
[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes
[The V2 Editor]: /docs/build-your-pipeline/v2-node-types/node-editor
[SQL Annotations Reference]: /docs/build-your-pipeline/v2-node-types/annotations-reference
[Upgrading from V1 to V2 Node Types]: /docs/build-your-pipeline/v2-node-types/upgrading-v1-v2
[Jinja Templates Beyond Ref]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq#jinja-templates-beyond-ref

---

## Upgrading from V1 to V2 Node Types

This guide walks you through moving Nodes from deprecated `inputMode: 'sql'` V1 Node types to Node Type V2.

## Overview

Other Node types in Coalesce could set `inputMode: 'sql'` in YAML by duplicating a built-in Node type and adding the setting manually. The Coalesce App shows a deprecation warning for any Node type that still uses this setting. The `inputMode: 'sql'` setting will be removed in a future release.

Node Type V2 is the supported replacement. It is not backwards compatible with `inputMode: 'sql'` Nodes. The two formats do not convert directly and do not share the same storage. V1 Nodes save as `.yml` files. V2 Nodes save as `.sql` files. The upgrade is a manual copy-and-paste process.

:::warning[Deprecated inputMode SQL]

`inputMode: 'sql'` is deprecated and will be removed in a future release. Node types that use it show a deprecation warning in the Coalesce App. Finish this migration before the setting is removed so Jobs and deploys stay predictable.

:::

## What You Are Moving

When you upgrade a Node from V1 `inputMode: 'sql'` to V2, you copy three things into the new V2 Node:

1. **The SQL query** - Your `SELECT` statement, including any CTEs, joins, and expressions.
2. **Node configuration** - Options that used to live in the **Config** panel, such as materialization type, insert strategy, `preSQL` and `postSQL`, become Node-level annotations above the `SELECT` in the `.sql` file.
3. **Column configuration** - Per-column options from the mapping grid, such as business keys and change tracking, become column-level annotations after each expression in the `SELECT` list.

There is no automated migration. You re-create each Node manually on a V2 Node Type.

## Step 1: Inventory Your V1 Nodes

Identify every Node type in your Workspace that still uses `inputMode: 'sql'`. They show a deprecation warning under **Build Settings > Node Types**. List the Nodes on each deprecated type so you know how much you need to migrate.

## Step 2: Create a Replacement V2 Node Type

1. Go to **Build Settings > Node Types** and open **Create Node Type**.
2. Choose **V2** so the new type has `version: 2`.
3. Give the type a clear name so it is easy to tell from the deprecated one during migration. If the old type was `SQL Stage`, you might use `SQL Stage V2`.

Copy the Create, Run, Join, and Macro templates from the deprecated Node type into the new one. Keep templates aligned with what runs in production so deployment behavior does not drift.

If your Run template loops over `sources` and emits a `SELECT`, add `{{ source.cteString }}` immediately before that `SELECT` inside the loop so CTEs from the SQL editor compile. For example:

```jinja
{% for source in sources %}
    {{ source.cteString }}
    SELECT
    {% for col in source.columns %}
        ...
    {% endfor %}
    {{ source.join }}
{% endfor %}
```

## Step 3: Recreate Each Node

For each Node on the deprecated type, add a new Node on your V2 Node type and move the content by hand.

### Copy the SQL Query

Open the V1 Node and copy the full `SELECT`, including any CTEs. Paste it into the V2 Node's SQL editor.

### Rewrite Node Configuration as Annotations

Options from the **Config** panel become Node-level annotations above the `SELECT`. Use this table to map common panel settings to annotations:

| V1 **Config** panel setting | V2 annotation |
| --- | --- |
| Materialization type: table | `@materializationType("table")` |
| Insert strategy: MERGE | `@insertStrategy("MERGE")` |
| Pre-SQL | `@preSQL("your SQL here")` |
| Post-SQL | `@postSQL("your SQL here")` |
| Tests enabled | `@testsEnabled` |

### Rewrite Column Configuration as Annotations

Per-column options from the mapping grid become annotations after each column expression. Use this table for common mappings:

| V1 column setting | V2 annotation |
| --- | --- |
| Business key | `@isBusinessKey` |
| Change tracking | `@isChangeTracking` |
| Not null | `@notNull` |
| Unique | `@isUnique` |
| Surrogate key | `@isSurrogateKey` |

A fully annotated `SELECT` can look like this:

```sql
@materializationType("table")
@insertStrategy("MERGE")

SELECT
    C_CUSTKEY AS customer_key @isBusinessKey @notNull,
    C_NAME AS customer_name @isChangeTracking @notNull,
    C_ADDRESS AS customer_address @isChangeTracking,
    C_NATIONKEY AS nation_key,
    C_ACCTBAL AS account_balance,
    C_MKTSEGMENT AS market_segment
FROM {{ ref('STAGING', 'STG_CUSTOMER') }}
```

See the [SQL Annotations Reference][] for the full annotation list and template wiring.

## Step 4: Validate the New Node

Before you migrate every Node, prove one path end to end:

1. Save the new V2 Node and confirm the **Columns** tab lists the expected columns and types.
2. Check the DAG and confirm upstream links from your `{{ ref() }}` calls look correct.
3. Run **Validate Select** to catch parse errors.
4. Run **Create** and **Run** and confirm warehouse results match what the V1 Node produced.

Fix template or annotation gaps before you continue in bulk.

## Step 5: Migrate Remaining Nodes

Repeat **Step 3: Recreate Each Node** and **Step 4: Validate the New Node** for every Node still on the deprecated type. You can migrate one subgraph at a time to reduce risk. V1 and V2 Nodes can live in the same pipeline and connect through `{{ ref() }}`.

## Step 6: Retire the Deprecated Node Type

After every Node is migrated and branches are merged, remove the deprecated Node type under **Build Settings > Node Types**. That clears the deprecation warning and drops the old `inputMode: 'sql'` definition from your Workspace.

## Preserving Lineage and Job Order

Use these practices so lineage and Job order stay correct after migration:

- **Refs for Coalesce Nodes** - Use `{{ ref('LOCATION', 'NODE_NAME') }}` for every upstream Coalesce Node you query. Hard-coded table names do not create graph edges and Job ordering does not treat them as declared dependencies.
- **Top-level CTEs** - Prefer CTEs over deeply nested subqueries. `{{ ref() }}` inside deeply nested subqueries does not always register as a DAG dependency even when the warehouse runs the SQL. Restructure so each `ref()` sits where the parser can see it, usually in a top-level CTE.

## What's Next?

- [SQL Annotations Reference][] for every reserved name and quoting rule.
- [The V2 Editor][] for the SQL workspace, Mapping tab, and actions.
- [Troubleshooting and FAQ][] for parse, column, and ref issues after migration.
- [Getting Started with Node Type V2][] if you want a guided first Node before you scale the work.
- [Node Type V2][] for overview, platform support, and links across the section.

[SQL Annotations Reference]: /docs/build-your-pipeline/v2-node-types/annotations-reference
[The V2 Editor]: /docs/build-your-pipeline/v2-node-types/node-editor
[Troubleshooting and FAQ]: /docs/build-your-pipeline/v2-node-types/troubleshooting-faq
[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes
[Node Type V2]: /docs/build-your-pipeline/v2-node-types/

---

## AI Administration

Control how AI behaves in the Catalog. Enable or disable the AI assistant, add company context, and connect it to your communication tools.

## Activate AI

<details>

<summary>You want to enable or disable the AI assistant? Go to this page and click "Activate/Deactivate"</summary>

<figure><figcaption></figcaption></figure>

</details>

## Drive AI: Describe Your Company

Improve the quality of AI-generated descriptions by adding your company description on the settings page.

This will influence the description generated by AI for Knowledge pages.

Edit the description on the settings page.

<figure><figcaption></figcaption></figure>

## Make It Accessible in Your Communication Tools

<ButtonLink to="/docs/catalog/integrations/communication/microsoft-teams">
  Microsoft Teams
</ButtonLink>

<ButtonLink to="/docs/catalog/integrations/communication/slack">
  Slack
</ButtonLink>

---

## AI Assistant Monitoring

This dashboard allows admins and governance teams to monitor how the AI Assistant is used, how it performs, and identify documentation opportunities. Learn more in the [AI Assistant introduction][].

## What Is This Dashboard About

Learn how to activate the assistant and what it does in the [AI Assistant introduction][]. This dashboard shows usage metrics, conversation history, and fields that need documentation.

## Monitor Usage

### Usage Evolution

You can view the number of interactions your users had with the assistant over the last 60 days.

<figure><figcaption></figcaption></figure>

### Conversation Overview

See all conversations. Note that a conversation can have several questions and answers. You can click the **See History** button to load the questions and answers of the conversation.

This is where you can dig into the content of each question and answer.

<figure><figcaption></figcaption></figure>

## Identify Missing Content

How to improve the answers of the assistant?

On the **Missing content** tab you will find the most used fields in your BI tool. You will be able to see if these popular fields are documented.

:::info[Document Top Fields]

We strongly recommend documenting your top 20 fields with detailed knowledge pages.

:::

<figure><figcaption></figcaption></figure>

## Export Raw Data

On any of these dashboard elements, you can download the underlying raw data. Simply click the `...` on the top right corner of the element.

Then click **Download results**.

[AI Assistant introduction]: https://docs.coalesce.io/docs/catalog/ai-assistant/introduction

---

## Data Documentation Coverage

The [Data Documentation Coverage dashboard][] is designed to empower data governance teams with valuable insights into their data landscape.

## What Does It Mean

- See which assets are missing an owner
- Find out your global documentation coverage
- Drill down by databases, schemas, tables, and users
- See which data sets are the most or least documented
- Identify which important assets are missing documentation
- Identify where you need to take action and improve the governance and knowledge of your assets

## Where To Find It

As an admin, go to your **Governance** page and you will see an **Analytics** tab. Click [Data Documentation Coverage dashboard][].

Navigate the dashboard content using the tabs.

<figure><figcaption></figcaption></figure>

## Overview

Get an overview of your data sets and who is actively documenting.

<figure><figcaption>Global numbers about your coverage</figcaption></figure>

<figure><figcaption>See your documentation effort for the last 90 days</figcaption></figure>

## Ownership and Documentation Coverage

Once filtered, navigate to the View per warehouses, databases, schemas, or tables so you can see those percentages at the granularity that makes sense for your use case.

- Ownership: A table is owned when it has an assigned owner (user or teams). Having a technical owner (imported from the warehouse) is not considered as having an owner
- Table description: A table is considered as described once it has a non-empty README section or a source description
- Columns description: A column is considered as described once the description field is not empty anymore

:::warning[Description Sources]

For both tables and columns, a "non-empty" description means either a description in Catalog or an external description that we imported from your warehouse.

:::

You can also leverage the **View per Owners** tab to check who should spend more time documenting in Catalog.

<figure><figcaption>Visualization of the documentation coverage by teams or users in the scope they own</figcaption></figure>

:::info[Share Feedback]

Share your thoughts about this feature so we can improve it. Slack is the best option, but [email][] works as well.

:::

## Export Raw Data

On any of these dashboard elements, you can download the underlying raw data. Simply click the `...` on the top right corner of the element.

Then click **Download results**.

<figure><figcaption></figcaption></figure>

## Take Actions

Once you have identified the missing owners or descriptions, here is an example of the next steps you could take:

- Search for an item from the search bar at the top of the page or from the [advanced search page][], so you can assign an owner, comment on the page to notify the admin, review the documentation, and so on.
- Leverage the [Metadata Editor][] page to assign ownership in bulk. See the [Metadata Editor documentation][] for details.

[Data Documentation Coverage dashboard]: https://app.castordoc.com/governance/analytics
[advanced search page]: https://app.castordoc.com/results?dashboard=true&table=true&term=true&vizModel=true
[Metadata Editor]: https://docs.coalesce.io/docs/catalog/document-your-data/metadata-editor
[Metadata Editor documentation]: https://docs.coalesce.io/docs/catalog/document-your-data/metadata-editor
[email]: mailto:support@coalesce.io

---

## Analytics

Catalog offers admins and governance teams several monitoring dashboards.

<ButtonLink to="./analytics/data-documentation-coverage">
  Data Documentation Coverage
</ButtonLink>

It will allow you to identify how well your database, projects, schemas, and data sets are documented. You will be able to see which teams and users have documented their data well.

<ButtonLink to="https://app.castordoc.com/governance/analytics/dashboards">
  Dashboard Documentation Coverage
</ButtonLink>

It will allow you to identify how well your dashboards are documented, by source or folder, applying any useful filter. You will be able to see which teams and users have documented their dashboards well.

<ButtonLink to="https://app.castordoc.com/governance/analytics/knowledge">
  Knowledge Documentation Coverage
</ButtonLink>

It will allow you to identify how well your knowledge pages are documented, applying any useful filter. You will be able to see which teams and users have created their knowledge pages.

<ButtonLink to="./analytics/users-activities">
  Users Activities
</ButtonLink>

Get insights about how widely Catalog is used in your company. Which users or teams log in, contribute, and how frequently?

<ButtonLink to="./analytics/ai-assistant-monitoring">
  AI Assistant Monitoring
</ButtonLink>

This dashboard allows admins and governance teams to monitor how the assistant is used, how it performs, and identify documentation opportunities.

<ButtonLink to="https://app.castordoc.com/governance/analytics/newly-added">
  Newly Added to Your Stack
</ButtonLink>

Find out about new popular dashboards, tables, and SQL users on your data stack.

---

## Users Activities

Get insights about how widely the Catalog is used in your company and pilot your rollout.

You can find the [user activity dashboard][] in the Catalog under the [Governance section][].

## What This Dashboard Is About

Get insights about how widely the Catalog is used in your company. Which users or teams log in, contribute, and how frequently?

The Catalog also helps you pilot your rollout by suggesting accounts you should invite based on their activity in your data stack.

## Pilot Your Users Rollout

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

## Monitor Usage and Contributions

### By Users

Take control of the Catalog's adoption in your company and lead the documentation effort.

- Monitor how many users actually use the Catalog every month

    <figure><figcaption></figcaption></figure>

- See who does and doesn't participate in the documentation effort

    <figure><figcaption></figcaption></figure>

- Identify your most recent active users

    <figure><figcaption></figcaption></figure>

### By Teams

Monitor activities scoped by teams.

- Monthly active users by teams

    <figure><figcaption></figcaption></figure>

- Monthly descriptions edited per team

    <figure><figcaption></figcaption></figure>

[user activity dashboard]: https://app.castordoc.com/governance/analytics/user-activity
[Governance section]: https://app.castordoc.com/governance/analytics

---

## Assets Access Control

On any database, schema, or dashboard folder, admins can manage who can access it in Catalog using the **Manage access** interface:

<figure></figure>

## Tables Access Control

Use these steps when you need to limit which teams can see tables for a database or schema in Catalog:

1. Navigate to your database or schema page through the left menu

    <figure></figure>
2. Click the top right corner button **Manage access**

    <figure><figcaption></figcaption></figure>

3. Then select the teams that should have access to the data set

   

## Dashboards Access Control

Follow this flow to restrict which teams can open a dashboard folder and its contents in Catalog:

1. Navigate to your **dashboard folder** page through the left menu
2. Click the top right corner button **Manage access**
3. Then select the teams that should have access to this folder and its subfolders

<figure></figure>

## How Does It Work?

Set the teams that should have access to the tables or dashboards in this data set or folder. Anyone who should see the data set and its tables needs to belong to at least one of those teams.

For example, any member of the HR or Sales team can access this data set's tables:

<figure></figure>

:::info[Admins and lineage visibility]

Admins always see everything in Catalog, regardless of **Manage access** settings. Restricted assets can still appear in lineage graphs, but with limited access elsewhere in Catalog.

:::

By default, any new data set or folder is accessible to anyone in the company. When a data set or folder access is given to only a few teams, any new asset in this perimeter inherits the access parameters.

Catalog **Manage access** applies to Catalog navigation, search, and asset pages. It's team-based in Catalog and doesn't replace warehouse object privileges such as Snowflake grants.

## Choose Where to Restrict Access in Catalog

Use this section when you are deciding whether to set **Manage access** on a database, on individual schemas, on dashboard folders, or on a combination. Start from the perimeter your teams should share, then use [Hierarchical data sets interaction][] for how parent and child settings combine.

```mermaid
flowchart TD
  start[Who should see this content?]
  start --> scope{What is the shared boundary?}
  scope -->|One set of teams for the whole database| db[Restrict on the database]
  scope -->|Different teams per schema under one database| mix[Set database baseline, then schema overrides]
  scope -->|Only some schemas are sensitive| sch[Leave database open or restrict only those schemas]
  scope -->|BI dashboards and folders| fld[Restrict on the dashboard folder]
  db --> check[People on allowed teams]
  mix --> check
  sch --> check
  fld --> check
```

### One Team or a Fixed Set of Teams Should Own an Entire Database

**Use:** set **Manage access** on the database so only those teams can see any schema or table under it until you intentionally override a child data set.

New tables and schemas under that database inherit the same teams. If you need an exception for one schema, open that schema and use **Manage access** there. Remember that a database-level list of allowed teams still limits which schemas are visible overall, as described in [Hierarchical data sets interaction][] and [Troubleshooting visibility for asset access][].

### Only Certain Schemas Should Be Limited

**Use:** leave the database unrestricted for company-wide discovery, or keep a wide team list on the database, then restrict individual schemas to the teams that should see those schemas.

This path fits when most of the database is public in Catalog but a few schemas need a tighter team list.

### Different Teams Need Different Schemas Under the Same Database

**Use:** combine a database-level configuration with schema-level **Manage access** so each schema lists the right teams. A child schema can add teams compared to the database, but it can't make sibling schemas visible to a team the database blocks. See [Hierarchical data sets interaction][] and [Troubleshooting visibility for asset access][].

### Dashboards and Subfolders Need Their Own Audience

**Use:** set **Manage access** on the dashboard folder that should gate a branch of your BI tree. Subfolders inherit unless you override them at the child folder, the same pattern as database and schema.

### Power BI Workspaces and Catalog Visibility

Power BI workspaces appear in Catalog as **dashboard folders** under **Dashboards** in the left navigation. Workspace names from Power BI often show up as segments in each folder path. Catalog does not expose a separate Power BI workspace object for access control. Restrict visibility on the dashboard folder that represents the workspace.

By default, every new dashboard folder is visible to anyone in your company until you restrict it with **Manage access**. This follows the company-wide default in [How Does It Work?][]. That includes folders Catalog creates when it ingests a new Power BI workspace. Until you restrict those folders, anyone in your company who uses Catalog can see unrestricted Power BI workspace folders, not only the workspaces you intend for each team.

When CI/CD or admins create workspaces in Power BI, Catalog creates or updates the matching folder on the next successful extraction. That folder starts with company-wide visibility unless a restricted parent folder applies or you set **Manage access** on it.

Use this pattern for ongoing Power BI governance:

- After initial ingestion, review top-level dashboard folders and set **Manage access** on production workspace folders so only the right teams can browse and search them.
- When new workspaces appear after deployment, restrict their folders on a schedule that matches how often you run extraction.
- Rely on subfolder inheritance when a branch of your Power BI tree should share one team list.

**Manage access** limits who can discover content in Catalog. It does not change who can open reports in the Power BI service. To limit which workspaces Catalog ingests, use path allow lists, block lists, or capacity boundaries on the Power BI integration. See [Scope Power BI ingestion][].

## Hierarchical Data Sets Interaction

Admins can define access rights at different levels in their stack, for example at database and schema level.

**Child overrides parent:** A child data set or subfolder access configuration overrides its parent's.

:::warning[Schema teams and database perimeter]

Adding a team on a schema does **not** grant that team access to other schemas in the same database when the database **Manage access** list excludes them. The database perimeter still applies to the whole database tree. Teams you add on the child only affect assets under that child.

:::

The behavior is the following:

- By default a child data set, such as a schema, inherits the parent configuration. For example, if the Sales team is the only one that can see the database assets, the child schema inherits it:

    <figure></figure>
- You can then choose to modify this configuration, which overrides the database access rights. Here, all the members of the frontend team and the sales team can access the assets in the schema. They still can't browse assets in other schemas because the database-level restriction applies to the rest of the tree.

    <figure></figure>
- You can also completely remove the inherited team from the schema access rights, which hides its assets from that team. For example:
  - The Frontend team will see the schema assets but nothing else from this database.
  - The Sales team, the only team that can see assets in this database, won't see the assets in this schema but will see all the other database assets.

      <figure></figure>

## Troubleshooting

Work through these checks in order when someone reports missing tables, schemas, or dashboard folders.

### Confirm Admin Status and Team Membership

1. **Catalog admin:** Admins see every asset regardless of **Manage access**. Reproduce the issue using a non-admin account that matches the affected person's teams so **Manage access** rules apply.
2. **Team membership:** The person must belong to at least one team on the asset's effective allow list. Verify teams in [Teams in Catalog][] and provisioning in [User and Team Provisioning][].

### Walk the Parent Chain for Tables and Schemas

For each missing asset, open **Manage access** on the asset, then on its parent database when the asset is a schema or table. The effective rule is applied along the hierarchy described in [Hierarchical data sets interaction][].

- **Symptom:** Someone is on a team listed on the schema but still can't see the schema or its tables.
- **Cause:** The database **Manage access** list doesn't include them or any of their teams, so the database perimeter blocks the whole branch.
- **Fix:** Add them to a team allowed on the database, widen the database allow list if policy allows, or restructure so the schema lives under a database with the right perimeter.

- **Symptom:** Someone lost access after new tables or folders appeared.
- **Cause:** New assets inherit restrictions from their parent data set or folder.
- **Fix:** Adjust **Manage access** on the parent or child so the intended teams are included where you want inheritance to apply.

- **Symptom:** Someone is on the database teams but can't see one schema.
- **Cause:** The schema **Manage access** list removed or omitted that person's team while other schemas still list it.
- **Fix:** Add the team back on the schema if they should see it, or confirm the omission is intentional.

### Lineage Shows an Asset but Browse or Search Does Not

Restricted assets can still appear in lineage graphs with limited access elsewhere, as described in the **Admins and lineage visibility** note under [How Does It Work?][]. If the graph references an object they can't open from search or the left navigation, treat that as a **Manage access** outcome, not a missing sync.

### Data Quality or Other Surfaces Look Empty

The [Data Quality Dashboard][] follows the same asset visibility rules as tables and schemas. If a schema or table is hidden by **Manage access**, its quality results won't surface for them. After you confirm integrations, see [Data quality dashboard troubleshooting][] for tests that don't appear.

### New User Sees All Power BI Workspaces

- **Symptom:** A newly provisioned user can browse or search every Power BI workspace folder in Catalog.
- **Cause:** Dashboard folders default to company-wide visibility until an admin restricts them.
- **Fix:** Set **Manage access** on each workspace folder that should not be company-wide. Confirm behavior with a non-admin account in the same teams as the user. See [Power BI workspaces and Catalog visibility][].

### New Power BI Workspace Appears With Broad Access

- **Symptom:** After CI/CD or a new workspace in Power BI, everyone in Catalog can see the new folder.
- **Cause:** Catalog ingested the workspace as a new folder with the default open visibility.
- **Fix:** Open the folder and set **Manage access** for the intended teams. If the workspace should not be in Catalog at all, work with your Catalog point of contact or [Coalesce Support][] on [Scope Power BI ingestion][] instead.

## What's Next?

- [Teams in Catalog][]
- [User roles in Catalog][]
- [User and Team Provisioning][]
- [Lineage troubleshooting][]
- [Data Quality Dashboard][]
- [Troubleshooting visibility for asset access][]
- [Power BI setup][]

[How Does It Work?]: /docs/catalog/administration/assets-access-control#how-does-it-work
[Hierarchical data sets interaction]: /docs/catalog/administration/assets-access-control#hierarchical-data-sets-interaction
[Troubleshooting visibility for asset access]: /docs/catalog/administration/assets-access-control#troubleshooting
[Data quality dashboard troubleshooting]: /docs/catalog/administration/data-quality-dashboard#troubleshooting
[Teams in Catalog]: /docs/catalog/administration/teams-in-catalog
[User roles in Catalog]: /docs/catalog/administration/user-roles-in-catalog
[User and Team Provisioning]: /docs/catalog/administration/user-and-team-provisioning
[Lineage troubleshooting]: /docs/catalog/navigate/lineage/lineage-troubleshooting
[Data Quality Dashboard]: /docs/catalog/administration/data-quality-dashboard
[Coalesce Support]: mailto:support@coalesce.io
[Power BI workspaces and Catalog visibility]: /docs/catalog/administration/assets-access-control#power-bi-workspaces-and-catalog-visibility
[Scope Power BI ingestion]: /docs/catalog/integrations/data-viz/power-bi/powerbi#scope-power-bi-ingestion
[Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi

---

## Azure AD Authentication Using SAML

Configure Azure AD authentication for Catalog using SAML. In this guide you'll create an Azure AD app, configure claims, and add the certificate to Catalog.

## Before You Begin

You'll need to be an Entra admin.

## Create the Entra ID App for Catalog

You'll register an enterprise application that acts as the SAML identity provider for Catalog.

1. Go to the [Microsoft Entra admin portal][].
2. Go to **Enterprise applications** > **All applications**.

    

3. Click **New Application**.
4. Click **Create your own application**.

    

5. Choose **Integrate any other application you don't find in the gallery**.
6. Name your app.

    

7. On the Overview page, select **2. Set up single sign on**.

    

8. Then select **SAML**.
9. On the **Set up Single Sign-On with SAML** page, open **Basic SAML Configuration**. Set **Identifier** to `production-castorSAML`. This value is the SAML entity ID.
10. Set **Reply URL** based on your account region:

    - For accounts using `app.castordoc.com`, use `https://api.castordoc.com/auth/saml/callback`.
    - For accounts using `app.us.castordoc.com`, use `https://api.us.castordoc.com/auth/saml/callback`.

    

11. Open **Attributes & Claims** and map the following optional claims:

    - `user.givenname` maps to `firstName`
    - `user.surname` maps to `lastName`
    - `user.mail` maps to `email`
    - You can delete `user.userprincipalname`.
    - Make sure to delete the Namespace.

    

    :::warning[Claim configuration]

    - Keep the Namespace empty for each claim.
    - Claim names are case sensitive.

    :::

12. Download the **Certificate (Base64)**.
13. Make note of the **Login URL**.

## Allow Users To Connect To The Catalog App

Make sure to add the users and groups in Microsoft Entra admin center who need to connect to Catalog.

:::info[Multiple Authentication Options]

Catalog can keep both SAML and email with password strategies active.

:::

## Add URL and Certificate to Catalog

Paste the signing certificate and login URL from Microsoft Entra into Catalog so SAML sign-in can complete.

1. In Catalog, go to [**Settings > Authentication**][].
2. Copy and paste the certificate and URL, making sure to format them correctly.

```json
{
  "entrypoint": "https://...",
  "certificate": "..."
}
```

---

[**Settings > Authentication**]: https://app.castordoc.com/settings/authentication
[Microsoft Entra admin portal]: https://aad.portal.azure.com/

---

## Google SSO with SAML

Connect Catalog to Google Workspace using Single sign-on (SSO) with SAML. This guide walks you through registering a SAML app in Google Workspace and configuring it in Catalog.

## Prerequisites

Before you begin, ensure you have the right access. You must be an administrator or have security privileges on your company’s Google Workspace.

## Set Up Your Google Workspace

Configure the SAML app in your Google Workspace admin console.

1. In your Google Workspace admin console, go to **Apps > Web and mobile apps** and click **Add custom SAML app**.

    Web and mobile apps page with the Add app menu open and “Add custom SAML app” selected." className="mdImages" width="60%"/>

2. In **App details**, add the app name.

   

3. In **Google Identity Provider details**, you'll see **Option 2: Copy the SSO URL, entity ID, and certificate**. Save the SSO URL and certificate for later.

   

4. In **Service provider details**, set the ACS URL to:
   1. `https://api.castordoc.com/auth/saml/callback` for accounts using `app.castordoc.com`
   2. `https://api.us.castordoc.com/auth/saml/callback` for accounts using `app.us.castordoc.com`
5. Set the **`entity_id`** to `production-castorSAML`.

   

6. In **Attribute mapping**, add these attributes in the exact format below so Catalog can read them.
   1. First name = `firstName`
   2. Last name = `lastName`
   3. Primary email = `email`

   

## Add to Catalog

Enter the SSO credentials you saved in Catalog.

1. In Catalog, go to **Settings > Authentication**.
2. Enter the SSO URL and Certificate.

---

## Authentication

<DocCardList />

---

## Set Up Okta SSO for Catalog

In this guide, you’ll learn how to set up Okta SSO in Catalog.

:::info[Okta Administrator]

You must be an Okta Administrator to complete this process.

:::

## Create Okta App Integration

To use Okta as your Single Sign-On provider, you'll want to create a new **App Integration** in Okta.

1. Open the admin panel for your Okta organization
2. Click on **Applications**.
3. Click on **Create App Integration**.

      

4. For **Create a new app integration**:
   1. **OIDC - OpenID Connect** as the **Sign-in method**.
   2. **Web Application** as the **Application Type** and then create the new app integration.

    

5. On the settings page for this newly created integration, enter the following:
    1. **App Integration Name** - You can use Catalog or any name.
    2. **Grant type**: `AuthorizationCode`
    3. **Sign-in redirect URIs**:
       1. `https://api.castordoc.com/auth/okta/callback` for accounts using `app.castordoc.com`
       2. `https://api.us.castordoc.com/auth/okta/callback` for accounts using `app.us.castordoc.com`
    4. **Sign-out redirect URI**:
       1. `https://app.castordoc.com` for accounts in the EU server
       2. `https://app.us.castordoc.com` for accounts in the US server
    5. **Controlled Access** - Select the setting that is appropriate for your organization.

    <DocCarousel slidesPerView={1}>
      <figure>
      
      <figcaption>Enter the app integration name and grant type</figcaption>
      </figure>
      <figure>
      
      <figcaption>Enter the sign-in and sign-out redirect URIs</figcaption>
      </figure>
      <figure>
      
      <figcaption>Choose the access type for your organization</figcaption>
      </figure>
    </DocCarousel>

6. Click **Save**. You'll now be at a window with all your **App Integration** settings. Keep this browser tab open as you'll need to enter some information from it into Catalog.

## Get Your Okta Application Information

You'll gather the information needed for Catalog.

1. Get your **Okta domain** by clicking on your username and then copying the domain name.

   

2. In the app settings, under **General**, copy the Client ID and Client Secret.

   

## Add Your Connection Settings to Catalog

You'll use your Okta Domain, Okta Client Id, and Okta Client Secret.

1. Open [Catalog > Settings > Authentication][].
2. Click **Setup** under Okta.
3. Enter your Okta Domain, Okta Client Id, and Okta Client Secret then **Save Configuration**.

[Catalog > Settings > Authentication]: https://app.castordoc.com/settings/authentication

---

## Data Quality Dashboard

The Catalog Data Quality Dashboard gives you a centralized view of data quality test results across your data assets. You can see which tables pass or fail tests, how much of your estate has coverage, and which tools are running those tests.

:::info[Catalog Does Not Run Tests]

Catalog displays results from your existing data quality tools. It does not run tests itself. Configure integrations such as [dbt][], [Soda][], other [Quality integrations][], or the [Quality API][] to push test results into Catalog.

:::

## Accessing the Dashboard

Open the [Data Quality Dashboard][] at `https://app.castordoc.com/data-quality`.

## Tabs and Filters

The dashboard has two tabs: **Overview** and **Result Details**. Use the filters at the top to narrow the data:

- **Warehouse:** Filter by data warehouse
- **Database:** Filter by database
- **Schema:** Filter by schema
- **Table Tag:** Filter by table tag
- **Owner:** Filter by table owner
- **Result Status:** Filter by outcome using pass, fail, or warning. Those map to the same states you see in the tables: **pass** is **SUCCESS**, **fail** is **ALERT**, and **warning** is **WARNING**.

## Overview Tab

The Overview tab shows a high-level summary of data quality across your tables.

<figure><figcaption>Overview tab with summary metrics, coverage, and table-level test results</figcaption></figure>

### Tables Overall Status

This section summarizes the health of your tables:

- **Tables with all tests passed:** Percentage of tables where every test is passing. For example, 60.7% means most of your tables are in good shape.
- **Tables by Status:** A breakdown of tables by their current status:
  - **ALERT:** Tables with one or more failing tests
  - **WARNING:** Tables with tests in a warning state
  - **SUCCESS:** Tables where all tests pass
- **UTC Time - Latest tests:** Timestamp of the most recent test run. Use this to see how fresh the data is.

### Tables Coverage

The Tables Coverage chart shows what percentage of your tables have quality tests:

- **Has tests:** Tables that have at least one quality test
- **No test:** Tables with no quality tests

Use this to identify gaps in coverage and prioritize where to add tests.

### Number of Tests by Technology

This chart shows how tests are distributed across the tools that run them. You will often see labels such as:

- **API:** Tests pushed through the Catalog Quality API
- **Coalesce Quality:** Tests from Coalesce Transform (built-in data quality)
- **dbt:** Tests from dbt
- **Soda:** Tests from Soda
- **Monte Carlo:** Tests from Monte Carlo
- **Sifflet:** Tests from Sifflet

### Tables Details

The Tables Details table lists individual tables and their test performance. Columns include:

- **Database . Schema:** Location of the table (for example, `NY_TAXI . GOLD`)
- **Table:** Table name
- **Popularity:** Usage score based on query activity
- **Tests Failing:** Count of failed tests (highlighted in red)
- **Tests Warning:** Count of tests in warning state (highlighted in yellow)
- **Tests Passing:** Count of passed tests (highlighted in green)

Use this table to identify tables that need attention and drill into specific assets.

## Result Details Tab

The Result Details tab shows a granular view of individual test results. Use it to see which specific tests failed, when they ran, and what the outcome was. This helps you investigate issues and track resolution over time.

<figure><figcaption>Result Details tab with individual test results</figcaption></figure>

### Test Result Details Table

The **Test Result Details** table lists each test run with the following columns:

- **Database . Schema . Table:** Full path of the data asset being tested (for example, `AUSTIN_TEST_DB . PROD . EDW_PROD . WRK_CUSTOMER_LOYALTY`)
- **Scope:** Whether the test applies to a specific column (for example, `Column: E_MAIL`) or the entire table (`Table`)
- **Name:** The name of the data quality rule or test (for example, `E_MAIL: Unique`, `quantity > 0`, `Includes Today's Data`)
- **Test run time:** When the test was last executed
- **Status:** Outcome of the test: **ALERT** (failed), **WARNING**, or **SUCCESS** (passed)

Use the cloud icon or the three-dot menu at the top right of the table to export or download the results.

## Data Quality at the Asset Level

You can also see data quality when browsing individual tables. Click the table name to view the [Data Quality][].

- Overall quality status for that table
- Individual test results (passed, failed, warning)
- Test execution timestamps
- Historical test performance

The **Overview** and **Result Details** tabs summarize estate-wide coverage and recent runs. They are not week-over-week trend charts. For longer historical context on one table, click on the table and go to the **Data Quality** tab.

## Setup and Configuration

### Enabling Data Quality Integration

To surface test results in the dashboard, connect the tool that runs your tests. For Coalesce-built pipelines, work through the following in order.

1. **Coalesce Transform and Coalesce Quality**: Configure a Warehouse integration in Catalog first. Without it, the first Coalesce Transform ingestion cannot complete. Then create a [Coalesce Quality][] source.
2. **External tools:** Use the [Quality API][] to push test results to Catalog.
3. **dbt, Soda, or other partners:** Configure the matching integration under [Quality integrations][].

### Visibility

Data quality visibility follows asset visibility. If a table or schema is hidden in Catalog, its quality tests will not appear in the dashboard. Quality information uses the same permission model as data assets.

## Troubleshooting

### Quality Tests Not Appearing

- Verify the integration is configured correctly (dbt, Soda, API, Coalesce Transform plus Coalesce Quality source, or another quality tool).
- Check that the source tables and schemas are visible in Catalog.
- Ensure API credentials have the right permissions if you use the Quality API.

### Missing Test Results

- Confirm tests are running in your quality tool.
- Check API integration logs for errors.
- Verify test metadata is being sent correctly to Catalog.

## What's Next?

- [Integrate dbt tests][dbt] or [Soda][] with Catalog
- [Push test results with the Quality API][Quality API]
- [Document tables][] and view quality in the Data Quality tab

---

[Data Quality Dashboard]: https://app.castordoc.com/data-quality
[dbt]: https://docs.coalesce.io/docs/catalog/integrations/quality/dbt-tests
[Soda]: https://docs.coalesce.io/docs/catalog/integrations/quality/soda
[Quality API]: https://apidocs.castordoc.com/#e4604160-95c5-4f6d-88ce-7908f5e13686
[Quality integrations]: https://docs.coalesce.io/docs/catalog/integrations/quality
[Document tables]: https://docs.coalesce.io/docs/catalog/assets/tables
[Data Quality]: https://docs.coalesce.io/docs/catalog/integrations/quality
[Coalesce Quality]: https://docs.coalesce.io/docs/catalog/integrations/transformation/coalesce/set-up

---

## Integration Settings

This page gives administrators visibility and control over integration features in Catalog. You can edit credentials, request sync back, configure blocked accounts, and more.

## Credentials and Authentication

Many integrations store API keys, personal access tokens, or OAuth-related values such as client id, client secret, and refresh or authorization codes. When authentication fails after a vendor-side rotation or policy change, fix the OAuth app or keys in your vendor admin console first, then open the integration in the Catalog app and use **Edit credentials** so the next sync uses the updated values.

For symptom-based checks that apply across integrations (for example redirect URI mismatches, scope errors, or stale tokens), see [Integration OAuth troubleshooting][]. That guide stays separate from Snowflake OAuth in the Coalesce App (Transform) and from Catalog Public API token authentication.

## Integration Settings

This page is designed to provide administrators with enhanced visibility and control over the various features available in Catalog for each type of integration. Non-exhaustive list of options you will find there based on the technology:

- Edit credentials
- Request sync back
- Blocked accounts configuration
- Retrieve dbt owners
- Request S3 assets extraction

<figure><figcaption></figcaption></figure>

## Database and Schema Visibility

Where to find it: Go to your [integration page][] and click on the source you want to edit, then navigate to the **Data Visibility** tab.

On this interface, for a given source, you will see all databases and schemas Catalog has access to. You can choose which ones you want to display to your users in the app by checking or unchecking them.

Allow a few moments for the changes to apply, then refresh your page to see them.

<figure><figcaption></figcaption></figure>

---

[Integration OAuth troubleshooting]: /docs/catalog/integrations/oauth-troubleshooting
[integration page]: https://app.castordoc.com/settings/integrations

---

## Tag Settings

Edit the settings on this [page][] to configure tag governance in Catalog. Tag governance includes two distinct permission settings: who can create new tags and who can apply existing tags to assets.

<figure><figcaption></figcaption></figure>

## Tag Governance Settings

Tag governance in Catalog includes two distinct permission settings:

1. **Tag Creation**: Controls who can create new tags.
2. **Tag Assignment**: Controls who can apply existing tags to assets.

These settings help prevent:

- Uncontrolled tag sprawl
- Unauthorized tagging of assets

## 1. Tag Creation

### Tag Creation Purpose

Controls who can create new tag objects (for example, "Sensitive", "PII", and so on).

### Tag Creation Access Options

- **Admins and Stewards only** (recommended for stricter governance)
- **Contributors, Stewards, and Admins** (more open tagging environment)

### How Tag Creation Works

When enabled:

- Users with appropriate permission can go to **Governance > Tag Manager**
- Create new tags by providing a name and selecting a color

**Example:**

- Navigate to Tag Manager
- Click **Create New Tag**
- Name it (for example, `Sensitive-Test`) and choose a color
- Save

### Tag Creation Best Practice

Restrict tag creation to Admins and Stewards to avoid tag clutter and maintain a clean tag taxonomy.

## 2. Tag Assignment

### Tag Assignment Purpose

Controls who can assign existing tags to assets (for example, tables, dashboards).

### Tag Assignment Access Options

- **Restricted**: Only asset owners (must also be contributors) can assign tags
- **Open**: Any contributor can assign tags to any asset

### How Tag Assignment Works

When enabled:

- Users can assign tags if they meet the configured permission
- Asset ownership is enforced only in the restricted mode

### Tag Assignment Best Practice

Use the restricted setting if:

- You want to prevent contributors from tagging assets they do not own
- You need tighter control over who labels data with governance tags

## Governance Strategy Recommendations

- **Avoid Tag Sprawl**: Restrict tag creation to Admins or Stewards.
- **Maintain Tag Integrity**: Restrict tag assignment if you are concerned about misuse or inconsistency.
- **Balance Access and Engagement**: Avoid over-restricting contributors. Only enable limitations if actual governance issues are occurring.

:::warning[Contribution Incentives]

Contribution to data documentation and tagging is hard to incentivize. Only restrict permissions when there is a clear need.

:::

### Summary

| Setting        | Purpose                       | Recommended For                                |
| -------------- | ----------------------------- | ---------------------------------------------- |
| Tag Creation   | Who can create new tags       | Limit to Admins or Stewards to avoid sprawl    |
| Tag Assignment | Who can assign tags to assets | Restrict if asset integrity is at risk         |

Use these controls to strike the right balance between collaboration and governance in your data environment.

[page]: https://demo.castordoc.com/settings/account-configuration

---

## Teams in Catalog

Admins can create and edit teams within Catalog. Manage team structures to enhance collaboration and organization.

You can create teams from the [teams page][] in Catalog, then edit an existing team on its page.

Teams fields are as follows:

- **Team Name** (required): How it will appear everywhere in Catalog
- **Team members list of emails**: List of all user emails belonging to the team
- **Slack Channel** (for example, #business_development): Where to reach the team on Slack
- **Slack Group** (for example, @hr_team): The team alias on Slack
- **Team email**: The email for reaching out to the whole team

<YouTube id="yiKgvthbr_s" />

## Team Page

Once you have created the teams, they each have a dedicated page where you can find:

1. All the team information and members
2. The dashboards, knowledge pages, and tables related to the team and its members

The team relation to an asset is shown in the **Team ties** column:

- Owned: The team itself is flagged as owner of this asset

The team member relation to an asset is shown in the **Members ties** column:

- Owned: A member of the team is flagged as owner of this asset
- Favorite: One of the team members has added the asset to favorites
- Editor: The team itself or a team member is responsible for the creation or edition of the asset
- Most Used: A team member is a frequent user of the asset

[teams page]: https://app.castordoc.com/people/teams

---

## User and Team Provisioning

Request a Catalog account, sync users and teams with SCIM, and assign roles based on team membership.

New Catalog accounts for new customers are created by the Coalesce team and are not self-service. Contact [support@coalesce.io](mailto:support@coalesce.io) or your Coalesce representative to request a new Catalog account. Include the customer name, key user emails, integrations needed (for example, Snowflake, Sigma, Coalesce Transform), and timeline.

If you are using an identity provider you might want to automatically synchronize your users and teams directly into Catalog. To do so, we expose an SCIM API allowing identity providers to handle users’ statuses and their team assignments.

This SCIM API allows you to add, update, and deactivate synchronized users and teams.

Here are walkthroughs to set this up for Microsoft Entra ID (formerly Azure AD) and Okta.

If sync fails, users never appear, or attributes look wrong after you configure the app, use [SCIM Provisioning Troubleshooting][] for shared checks across providers.

## Role Assignment Based on Team Membership

Once the setup is configured, an automated script in **Catalog** will assign roles to users based on their team membership. To enable this functionality, you'll need to provide a **role mapping** that links teams to specific roles.

**Handling Multiple Team Memberships:**

If a user belongs to multiple teams, the system will automatically assign the role associated with the team that has the highest level of access.

**Role Mapping Example:**

You’ll provide a mapping that specifies which roles correspond to each team.

```json
{
  "VIEWER": ["dev"],
  "CONTRIBUTOR": ["marketing", "sales"],
  "ADMIN": ["founders"]
}
```

## When To Contact Support

Org admins can add users, change roles, and manage access directly in the Coalesce App. Some account and access requests are handled by the Coalesce team and are not self-service.

**Self-service (org admins):**

- Add users to your organization.
- Edit user names and roles (Admin vs User).
- Disable, activate, or delete users.
- Assign RBAC roles at the organization, project, and environment levels.

**Support-assisted (contact Coalesce):**

- New Catalog accounts for new customers.
- New customer organization setup.
- POC account setup and adding specific users.

When requesting new accounts or access, include the following in your support request:

- Customer or organization name.
- Key user email addresses.
- Integrations needed (for example, Snowflake, Sigma, Coalesce Transform).
- Timeline and any organizational context (for example, merger, hierarchy change).

---

[SCIM Provisioning Troubleshooting]: /docs/catalog/administration/user-and-team-provisioning/scim-provisioning-troubleshooting

---

## SCIM Provisioning Troubleshooting

Use this guide when Catalog users or teams are missing, out of date, or fail to sync from your identity provider after you have started SCIM setup. It complements the provider-specific walkthroughs with shared checks and common failure patterns.

:::info[Catalog SCIM Scope]

SCIM in this section provisions Catalog accounts and team membership. It does not replace just-in-time SSO for the Coalesce Transform app. If your question is about Transform login or SSO buttons, start with [Troubleshooting Common SSO Errors][].

:::

## Before You Start

Confirm the following so you are not debugging the wrong surface:

- You completed the integration and token step with Catalog ops using [support@coalesce.io][] or your Coalesce representative, as described in [SCIM Setup for Okta][] or [SCIM Setup for Microsoft Entra ID][].
- You know whether your Catalog tenant uses `app.castordoc.com` or `app.us.castordoc.com` so you can validate the regional base URL.
- You can open provisioning settings and logs for the Catalog app in Okta or the Entra Enterprise application.

## How Issues Show Up

You might notice one or more of the following:

- New hires in your identity provider never appear in Catalog, even when assignments look correct.
- Updates to names or team membership in the provider do not show in Catalog after a reasonable wait.
- Deactivated users in the provider still have Catalog access because provisioning did not deactivate them in Catalog.
- Provisioning jobs fail in the provider with connection, authorization, or attribute errors in the log.
- Catalog users are created but names look empty or wrong even though email is present.

These symptoms usually trace to integration not enabled, wrong URL or token, assignment scope, attribute mapping, or sync schedule.

## Verify the Integration and Connection

1. Open your Catalog SCIM application in Okta or Entra ID.
2. Go to the provisioning or API integration screen your provider uses for outbound SCIM.
3. Confirm provisioning is **enabled** or **on**, not paused.
4. Run **Test Connection** or the equivalent health check and fix reported errors before chasing data issues.

If the connection test fails, fix URL and token first. Attribute mapping will not matter until authentication to the SCIM endpoint succeeds.

## Check Base URL and Token

The SCIM base URL must match your Catalog region. Use exactly one of:

| Catalog app host | SCIM base URL |
| --- | --- |
| `app.castordoc.com` | `https://api.castordoc.com/auth/scim` |
| `app.us.castordoc.com` | `https://api.us.castordoc.com/auth/scim` |

Token issues often look like auth failures or intermittent 401 responses in provider logs. If a token was exposed or rotated outside Coalesce, ask Coalesce to issue a new token. Do not paste old tokens in tickets. Use [support@coalesce.io][].

## Verify Attribute Mapping

Incorrect mapping usually produces users with email but missing first and last name, or failed creates when required fields are empty.

- **Okta:** Follow [SCIM Setup for Okta][] for **To App** actions and profile mapping. Ensure `givenName` and `familyName` in Okta are populated for affected people.
- **Microsoft Entra ID:** Follow [SCIM Setup for Microsoft Entra ID][] so users keep `userName`, `active`, `name.givenName`, and `name.familyName`, and `externalId` maps to `objectId` as documented. Remove extra attributes the walkthrough says to delete.

:::tip[Provider-Specific Steps]

Keep the setup doc open while you compare mappings. Small differences, such as an extra attribute or a wrong `externalId` source, are a frequent root cause.

:::

## Confirm Assignments and Group Scope

SCIM only processes identities your provider sends to this application.

- **Okta:** Under the Catalog SCIM app, check **Assignments** so people or groups that should get Catalog are actually assigned.
- **Microsoft Entra ID:** Under the Enterprise application, check **Users and groups** so the right users and groups are assigned. Remember that group-based assignment drives who is in scope for provisioning when you use groups.

If someone is missing from assignments, they will not reach Catalog through SCIM.

## Sync Timing and Logs

Entra ID provisioning commonly runs on an interval, and Okta may take time between runs depending on configuration and backlog.

1. After fixing configuration, allow at least one full sync cycle.
2. If your provider supports it, **restart** or **re-save** provisioning to queue a fresh run.
3. Read **provisioning logs** or **provisioning summary** in the provider and note error codes or failing step names. Those strings help Coalesce support match the failure to API behavior.

## Manual Catalog Users Versus SCIM

You can have manually created Catalog access while you roll out SCIM. Once SCIM is healthy, prefer a single source of truth in your identity provider so you do not keep duplicate or stale records. Align email and lifecycle with your IT standards so just-in-time SSO or SCIM does not fork identities for the same person.

## When To Contact Support

Contact [support@coalesce.io][] when:

- Connection tests keep failing after URL and token checks.
- Token reset or rotation is required.
- Provider logs show repeated errors you cannot map to configuration steps in the setup guides.
- You need help interpreting SCIM responses alongside your provider logs.

Include your Catalog host region, identity provider, approximate time of failures, and redacted screenshots or log excerpts where your policy allows.

## What's Next?

- [SCIM Setup for Microsoft Entra ID][]
- [SCIM Setup for Okta][]
- [User and Team Provisioning][]
- [Troubleshooting Common SSO Errors][]

---

[support@coalesce.io]: mailto:support@coalesce.io
[Troubleshooting Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors
[SCIM Setup for Okta]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-okta
[SCIM Setup for Microsoft Entra ID]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-microsoft-entra-id
[User and Team Provisioning]: /docs/catalog/administration/user-and-team-provisioning

---

## SCIM Setup for Microsoft Entra ID

This walkthrough shows how to set up a SCIM app on Microsoft Entra ID to automatically provision users and teams into Catalog.

:::info[Contact Catalog Ops]

This setup requires you to contact Catalog ops ([support@coalesce.io](mailto:support@coalesce.io) or Slack) to generate and share the required SCIM token you will use to connect your SCIM app to Catalog's SCIM API.

:::

## 1. Creating an Enterprise App

- You will need a dedicated Catalog Enterprise app in the Azure Portal. It will be used to assign users and teams to provision into the app.
- If you already have a Catalog app for SAML login, you can go directly to the second part of this documentation.

1. First, go to the [Enterprise applications list][] in the Azure Portal.

    <figure><figcaption></figcaption></figure>

2. Click on `+ New application`

    <figure><figcaption></figcaption></figure>

3. Then `Create your own application`

    <figure><figcaption></figcaption></figure>

4. Give it a name and tick `Integrate any other application...`

    <figure><figcaption></figcaption></figure>

## 2. Setting Up Provisioning

1. Once in the application, go to the `Provisioning` menu entry.

    <figure><figcaption></figcaption></figure>

2. If not already set up, click on `Get started`

    <figure><figcaption></figcaption></figure>

3. There you will set up provisioning information:
   - Choose `Automatic` provisioning mode
   - Input Tenant URL as:
     - `https://api.castordoc.com/auth/scim` for accounts using `app.castordoc.com`
     - `https://api.us.castordoc.com/auth/scim` for accounts using `app.us.castordoc.com`
   - Add the secret token the Catalog ops provided you (or reach out: [support@coalesce.io](mailto:support@coalesce.io) or via Slack)
   - Test the connection, then save

    <figure><figcaption></figcaption></figure>

## 3. Configuring Mappings

- In this part you will craft the mapping between your user and team information in Microsoft Entra ID and their Catalog accounts.

<figure><figcaption></figcaption></figure>

- **Groups**
  - Keep only those 3 fields and no others. Theoretically this should be the mapping by default, nothing to change here.

<figure><figcaption></figcaption></figure>

- **Users**
  - We only use a limited amount of Microsoft Entra ID fields in Catalog, so you need to refine the attributes list to keep only the ones feeding our SCIM API.
  - Attributes (4) to keep as default:
    - `userName`
    - `active`
    - `name.givenName`
    - `name.familyName`
  - Attribute (1) to edit:
    - Edit the `externalId` attribute so it matches `objectId` instead of `mailNickname`

<figure><figcaption></figcaption></figure>

- Delete all other attributes.
- End result should look like this:

<figure><figcaption></figcaption></figure>

## 4. Trigger Provisioning

- Once the mapping updates are done, you can start assigning users and groups that will be provisioned to Catalog

<figure><figcaption></figcaption></figure>

- Review groups that can access the Catalog app. All users in these groups will be provisioned with a Catalog account.
- When everything is set up, you can start triggering the provisioning from the Overview section. It will start importing users and teams into Catalog, and every new user and team update will be forwarded to Catalog in the next 40 minutes.

<figure><figcaption></figcaption></figure>

## Troubleshooting

- If your Catalog user appears without first and last name, ensure their `givenName` and `familyName` are filled in their Microsoft Entra ID profile.
- If you had an issue or leakage of your token, reach out to Catalog ops ([support@coalesce.io](mailto:support@coalesce.io)) to reset the token.
- For broader checks on connection tests, assignments, sync timing, and logs, see [SCIM Provisioning Troubleshooting][].

[SCIM Provisioning Troubleshooting]: /docs/catalog/administration/user-and-team-provisioning/scim-provisioning-troubleshooting
[Enterprise applications list]: https://portal.azure.com/#view/Microsoft_AAD_IAM/StartboardApplicationsMenuBlade/~/AppAppsPreview

---

## SCIM Setup for Okta

This walkthrough shows how to set up a SCIM app on Okta to automatically provision users and teams into Catalog.

:::info[Contact Catalog Ops]

This setup requires you to contact Catalog ops ([support@coalesce.io](mailto:support@coalesce.io) or Slack) to generate and share the required SCIM token you will use to connect your SCIM app to Catalog's SCIM API.

:::

## 1. Creating a SAML App

1. Go to your Okta applications and click on **Browse App Catalog**.

    <figure><figcaption></figcaption></figure>

1. Then search for SCIM and choose **SCIM 2.0 Test App (Header Auth)**

    <figure><figcaption></figcaption></figure>

1. And click the **Add Integration** button.

    <figure><figcaption></figcaption></figure>

1. Give the app a proper name and click **Next**.

    <figure><figcaption></figcaption></figure>

1. As we will only use this app for provisioning, you do not need to specify proper SAML information. Go to the end of the page and click **Done**

  <figure><figcaption></figcaption></figure>

## 2. Setting Up Provisioning

1. Once in the application, go to the **Provisioning** tab and click the **Configure API Integration** button

    <figure><figcaption></figcaption></figure>

2. There you will set up provisioning information.
   1. Input Base URL as:
      1. `https://api.castordoc.com/auth/scim` for accounts using **app.castordoc.com**
      2. `https://api.us.castordoc.com/auth/scim` for accounts using **app.us.castordoc.com**
   2. Add the API token the Catalog ops provided you (or reach out: [support@coalesce.io](mailto:support@coalesce.io) or through Slack)
   3. Test the connection, then save.

<figure><figcaption></figcaption></figure>

## 3. Configuring Mappings

In this part you will craft the mapping between your user and team information in Okta and their Catalog accounts. For that you need to update the two mappings.

1. In the **Provisioning** tab, go to the **To App** section:
1. Click on **Edit** and enable **Create Users**, **Update User**, and **Attributes and Deactivate Users**, then **Save**

    <figure><figcaption></figcaption></figure>

1. Click on **Go to Profile Editor** under the **Attribute Mappings** to select the desired fields to send to Catalog

    <figure><figcaption></figcaption></figure>

1. In the **Profile Editor** click on **Mappings**

    <figure><figcaption></figcaption></figure>

1. On the **CastorDoc SCIM to Okta User** tab:
   1. Unmap all fields except `appuser.givenName` and `appuser.familyName` (Okta attribute names).
   2. Update the `appuser.email` to `email` mapping to `appuser.userName` to `email`.

    <figure><figcaption></figcaption></figure>

1. On the **Okta User to CastorDoc SCIM** tab:
   1. Remove all field mappings except **user.firstName** and **user.lastName**

    <figure><figcaption></figcaption></figure>

## 4. Trigger Provisioning

Once the mapping updates are done, you can start assigning users and groups that will be provisioned to Catalog

<figure><figcaption></figcaption></figure>

## Troubleshooting

- If your Catalog user appears without first and last name, ensure their `givenName` and `familyName` are filled in their Okta profile.
- If you had an issue or leakage of your token, reach out to Catalog ops ([support@coalesce.io](mailto:support@coalesce.io)) to reset the token.
- For broader checks on connection tests, assignments, sync timing, and logs, see [SCIM Provisioning Troubleshooting][].

[SCIM Provisioning Troubleshooting]: /docs/catalog/administration/user-and-team-provisioning/scim-provisioning-troubleshooting

---

## User Roles in Catalog

Understand the different roles in the Catalog and what actions each role can perform.

## Roles in Catalog

<BetterMDTables
  headers={['Action', 'Viewer', 'Contributor', 'Steward', 'Admin']}
  rows={[
    ['Search assets', '✅', '✅', '✅', '✅'],
    ['View assets', '✅', '✅', '✅', '✅'],
    ['Add comments', '✅', '✅', '✅', '✅'],
    ['Request descriptions', '✅', '✅', '✅', '✅'],
    ['Edit descriptions', '❌', '✅', '✅', '✅'],
    ['Create knowledge pages', '❌', '✅', '✅', '✅'],
    ['Edit tags', '❌', '✅', '✅', '✅'],
    ['Certify owned assets', '❌', '✅', '✅', '✅'],
    ['Publish Data Product to Marketplace', '❌', '✅', '✅', '✅'],
    ['Make yourself owner', '❌', '✅', '✅', '✅'],
    ['Certify any asset', '❌', '❌', '✅', '✅'],
    ['Add anyone owner', '❌', '❌', '✅', '✅'],
    ['Metadata analytics', '❌', '❌', '✅', '✅'],
    ['Manage tags', '❌', '❌', '✅', '✅'],
    ['Manage templates', '❌', '❌', '✅', '✅'],
    ['Download reports', '❌', '❌', '✅', '✅'],
    ['Bulk edit metadata', '❌', '❌', '✅', '✅'],
    ['Manage users & teams', '❌', '❌', '❌', '✅'],
    ['Manage content access', '❌', '❌', '❌', '✅'],
    ['Add/edit data sources', '❌', '❌', '❌', '✅'],
    ['Manage account settings', '❌', '❌', '❌', '✅']
  ]}
/>

---

## AI Assistant Context

Here is the information we use for AI Search as of July 7, 2025.

**AI Search uses information from the `assets` most relevant to the user's `question`, and then generates an `answer` based on that.**

- Terms (Knowledge pages)
  - id
  - name
  - description
  - owner
  - tags
  - slug
  - type = `TERM`
  - score (relevant score)
- Tables
  - id
  - name
  - description
  - owner
  - tags
  - slug
  - type = `TABLE`
  - tableType: `EXTERNAL`/ `TABLE`/ `TOPIC`/ `VIEW`
  - score (relevant score)
- Dashboards
  - id
  - name
  - description
  - owner
  - path
  - slug
  - dashboardType: `DASHBOARD`/ `TILE`/ `VIZ_MODEL`
  - source (sourceName)
  - tags
  - technology
  - type = `DASHBOARD`
  - score (relevant score)
- Columns
  - id
  - name
  - description
  - dataType
  - isPii
  - isPrimaryKey
  - tableId
  - tableName
  - type = `COLUMN`
  - score (relevant score)

For Catalog assets (tables, dashboards, or columns), we also take into account assets pinned to knowledge pages that are related to the user's question

***

- _payload example_
  - terms

```json
      {
        "terms": [
          {
            "description": "DEFINITION\n\nMonthly Recurring Revenue (MRR) is a key financial metric that represents the\npredictable and recurring revenue generated by a business on a monthly basis. It\nis particularly relevant for subscription-based businesses as it provides a\nclear view of revenue trends and growth potential. Understanding MRR helps\nbusiness analysts and decision-makers evaluate the company's financial health,\nforecast future revenue, and identify opportunities for growth or improvement.\n\n\nKEY METRICS & ENTITIES\n\n * Total MRR\n * New MRR\n * Expansion MRR\n * Churned MRR\n * Net MRR Growth Rate\n * Customer Segments\n * Subscription Plans\n\n\nKEY STAKEHOLDERS\n\n * Finance Team\n * Sales Team\n * Marketing Team\n * Product Team\n * Executive Leadership\n\n\nRELATED DOMAINS\n\n * Customer Retention\n * Revenue Forecasting\n * Churn Analysis\n * Customer Lifetime Value (CLV)\n * Subscription Management",
            "id": "1e0b1aea-961f-4fa4-aa3f-5ab66a8ccc86",
            "internalLink": "https://app.castordoc.com/terms/-ddff7a34",
            "name": "MRR",
            "slug": "-ddff7a34",
            "score": 0.42455316,
            "tags": [],
            "type": "TERM"
          },
          {
            "description": "🐻 WHO WE ARE 🐻\n\n    At Timber, we believe that robotics is the future of innovation. Our products aim\n    to revolutionize everyday tasks through advanced automation. Timber robots are\n    designed to be intelligent, efficient, and collaborative, offering solutions\n    that simplify complex problems while enhancing productivity. By doing so, we\n    empower businesses and individuals to achieve more with cutting-edge robotics.\n\n    Since our founding, Timber has been at the forefront of innovation, driving\n    progress in sectors from electric vehicles to energy solutions. Now, with the\n    creation of Timber Robotics, we are taking the next step in automation. Our\n    robots are already making an impact across various industries, from\n    manufacturing to healthcare.\n\n    Our team, 40+ people strong, consists of experts and soon-to-be experts in\n    robotics, engineering, and AI, united by a bold vision to transform industries\n    through robotics. To support our mission, Timber has raised significant\n    investments, gaining support from world-class partners and investors.\n\n\n    👨‍💻 ABOUT THE ROLE 👩‍💻\n\n    We are looking for a Senior or Principal Product Manager to join our growing\n    Timber Robotics team.\n\n    The ideal candidate will be skilled at uncovering user needs, identifying pain\n    points, and shaping the future of robotics technology. You will join our dynamic\n    product team alongside John, Jane, Alex, and Emily.\n\n    Our vision is to Revolutionize Automation and Democratize Robotics Access\n\n    We focus on three core tracks:\n\n    🎯 Discovery: We enable users to understand and leverage robotic systems\n    effectively.\n\n    🏗️ Build Solutions: We streamline the creation and customization of robotics\n    applications, making it easier to deploy in various environments.\n\n    🏎️ Drive Innovation: We empower leaders to optimize their operations with smart\n    robotics solutions that enhance efficiency and safety.\n\n\n    🏄 ABOUT THE FIT? 🏇\n\n    Who are we looking for:\n\n    * Experience: You have a strong background in product management, with\n      experience discovering new market opportunities, building roadmaps, and\n      delivering innovative features.\n    * Problem solver: You excel at identifying challenges and digging deep to\n      understand the root causes. You obsess over finding the most effective\n      solutions for users.\n    * Tech curiosity: You are passionate about technology and eager to dive into\n      the world of robotics and AI. You will have the opportunity to work with\n      cutting-edge technology, including AI and machine learning.\n    * Impact maker: You are dedicated to ensuring that the product delivers value\n      to end-users, pushing projects forward to see real-world impact.\n    * Communication & collaboration skills: You are an excellent communicator, able\n      to clearly articulate customer challenges and collaborate across teams.\n    * Fluent in English, with strong communication skills.\n\n    We offer a dynamic work environment where you can focus on making an impact,\n    supported by:\n\n    * Competitive compensation packages, including equity\n    * Flexible and remote-friendly work environment, with offices in California\n    * Any tools and support you need to succeed\n\n\n    👋 HOW TO APPLY\n\n    Reach out on LinkedIn to John Doe.\n\n    If this offer interests you but you're unsure if you meet 100% of the\n    requirements, we still encourage you to apply. We welcome diverse perspectives\n    and are open to exploring what you can bring to the team.",
            "id": "44f6c50c-a7b2-4da9-bcdf-c231fb7828cc",
            "internalLink": "https://app.castordoc.com/terms/job-post-4e159f35",
            "name": "Job post",
            "slug": "job-post-4e159f35",
            "score": 0.42678326,
            "tags": [],
            "type": "TERM"
          },
          {
            "description": "Policy Owner: Arnaud de Turckheim Effective Date: 07-Dec-2021 1. Purpose To ensure that information is classified, protected, retained and securely disposed of in accordance with its importance to the organization. 2. Scope All Acme data, information and information systems. 3. Policy Acme classifies data and information systems in accordance with legal requirements, sensitivity, and business criticality in order to ensure that information is given the appropriate level of protection. Data owners are responsible for identifying any additional requirements for specific data or exceptions to standard handling requirements. Information systems and applications shall be classified according to the highest classification 4. Data Classification To help Acme and its employees easily understand requirements associated with different kinds of information, the company has created three classes of data. Confidential Highly sensitive data requiring the highest levels of protection; access is restricted to specific employees or departments, and these records can only be passed to others with approval from the data owner, or a company executive. Example include: ● Customer Data ● Personally identifiable information (PII) ● Company financial and banking data ● Salary, compensation and payroll information ● Strategic plans ● Incident reports ● Risk assessment reports ● Technical vulnerability reports ● Authentication credentials ● Secrets and private keys ● Source code ● Litigation data Restricted Acme proprietary information requiring thorough protection; access is restricted to employees with a “need-to-know” based on business requirements. This data can only be distributed outside the company with approval. This is default for all company information unless stated otherwise. Examples include: ● Internal policies ● Legal documents ● Meeting minutes and internal presentations ● Contracts ● Internal reports ● Slack messages ● Emails Public Documents intended for public consumption which can be freely distributed outside Acme. Examples include: ● Marketing materials ● Product descriptions ● Release notes ● External facing policies 5. Labeling Confidential data should be labeled “confidential” whenever paper copies are produced for distribution. 6. Data Handling Confidential Data Handling Confidential data is subject to the following protection and handling requirements: ● Access for non-preapproved-roles requires documented approval from the data owner ● Access is restricted to specific employees, roles and/or departments ● Confidential systems shall not allow unauthenticated or anonymous access ● Confidential Customer Data shall not be used or stored in non-production systems/environments ● Confidential data shall be encrypted in transit over public networks ● Mobile device hard drives containing confidential data, including laptops, shall be encrypted ● Mobile devices storing or accessing confidential data shall be protected by a log-on password or passcode and shall be configured to lock the screen after five (5) minutes of non-use ● Backups shall be encrypted ● Confidential data shall not be stored on personal phones or devices or removable media including USB drives, CD’s, or DVD’s ● Paper records shall be labeled “confidential” and securely stored and disposed ● Hard drives and mobile devices used to store confidential information must be securely wiped prior to disposal or physically destroyed ● Transfer of confidential data to people or entities outside the company shall only be done in accordance with a legal contract or arrangement, and the explicit written permission of management or the data owner Restricted Data Handling Restricted data is subject to the following protection and handling requirements: ● Access is restricted to users with a need-to-know based on business requirements ● Restricted systems shall not allow unauthenticated or anonymous access ● Transfer of restricted data to people or entities outside the company or authorized users shall require management approval and shall only be done in accordance with a legal contract or arrangement, or the permission of the data owner ● Paper records shall be securely stored and disposed ● Hard drives and mobile devices used to store restricted information must be securely wiped prior to disposal or physically destroyed Public Data Handling No special protection or handling controls are required for public data. Public data may be freely distributed. 7. Data Retention Acme shall retain data as long as the company has a need for its use, or to meet regulatory or contractual requirements. Once data is no longer needed, it shall be securely disposed of or archived. Data owners, in consultation with legal counsel, may determine retention periods for their data. Retention periods shall be documented in the Data Retention Matrix in Appendix B to this policy. 8. Data & Device Disposal Data classified as restricted or confidential shall be securely deleted when no longer needed. Acme shall assess the data and disposal practices of third-party vendors in accordance with the Third-Party Management Policy. Only third-parties who meet Catalog requirements for secure data disposal shall be used for store and process restricted or confidential data. Acme shall ensure that all restricted and confidential data is securely deleted from company devices prior to, or at the time of disposal. 9. Annual Data Review Management shall review data retention requirements during the annual review of this policy. Data shall be disposed of in accordance with this policy. 10. Legal Requirements Under certain circumstances, Acme may become subject to legal proceedings requiring retention of data associated with legal holds, lawsuits, or other matters as stipulated by Catalog legal counsel. Such records and information are exempt from any other requirements specified within this Data Management Policy and are to be retained in accordance with requirements identified by the Legal department. All such holds and special retention requirements are subject to annual review with Acme legal counsel to evaluate continuing requirements and scope. 11. Policy Compliance Acme will measure and verify compliance to this policy through various methods, including but not limited to, business tool reports, and both internal and external audits.",
            "id": "96ed7b99-8124-4d47-833a-b1014cc795cc",
            "internalLink": "https://app.castordoc.com/terms/-db30cba6",
            "name": "[Example] Data Management Policy",
            "slug": "-db30cba6",
            "score": 0.43141475,
            "tags": [],
            "type": "TERM"
          },
          {
            "description": "DEFINITION\n\nMonthly Active Users (MAU) is a key performance metric used to measure the\nnumber of unique users who engage with a product, service, or platform within a\ngiven month. It provides insights into user activity, engagement trends, and\noverall product adoption. For business analysts, MAU is crucial for evaluating\nthe success of customer acquisition strategies, retention efforts, and the\noverall health of a business's user base.\n\n\nIMPORTANCE\n\nTracking MAU helps businesses understand how effectively they are retaining\nusers and driving consistent engagement. It is often used alongside other\nmetrics like Daily Active Users (DAU) or retention rates to assess user behavior\nand identify growth opportunities. By analyzing MAU trends, businesses can make\ndata-driven decisions to improve user experience, optimize marketing campaigns,\nand prioritize product development efforts.\n\n\nCALCULATION\n\nMAU is calculated by counting the number of unique users who interact with a\nproduct or service at least once during a calendar month. This interaction can\ninclude logging in, making a purchase, or performing any other meaningful action\nthat signifies engagement. It is important to ensure that duplicate or inactive\nusers are excluded to maintain accuracy in the metric.\n\n\nUSE CASES\n\nMAU is widely used across industries to monitor user engagement and growth. For\nsubscription-based businesses, it helps evaluate customer retention and churn.\nIn digital platforms, it provides insights into user activity and content\nconsumption. Additionally, MAU is often used by investors and stakeholders to\nassess the scalability and market potential of a business.",
            "id": "d3907815-201b-4fac-9962-8c74aaa49151",
            "internalLink": "https://app.castordoc.com/terms/-2ebeb707",
            "name": "MAU",
            "slug": "-2ebeb707",
            "score": 0.42136905,
            "tags": [],
            "type": "TERM"
          },
          {
            "description": "DEFINITION\n\nARR, or Annual Recurring Revenue, is a key financial metric used to measure the\npredictable and recurring revenue generated by a business over a year. It is\nparticularly relevant for subscription-based or SaaS (Software as a Service)\nbusinesses as it provides insights into the company's revenue stability and\ngrowth potential. Understanding ARR helps business users evaluate performance,\nforecast future revenue, and make informed strategic decisions.\n\n\nKEY METRICS & ENTITIES\n\n * Total ARR\n * New ARR (revenue from new customers)\n * Expansion ARR (revenue from upselling or cross-selling to existing customers)\n * Churned ARR (revenue lost due to customer cancellations)\n * Net ARR Growth Rate\n * Customer Segments\n * Subscription Plans\n\n\nKEY STAKEHOLDERS\n\n * Finance Team\n * Sales Team\n * Marketing Team\n * Customer Success Team\n * Executive Leadership\n\n\nRELATED DOMAINS\n\n * Revenue\n * Customer Retention\n * Sales Performance\n * Subscription Management\n * Financial Forecasting",
            "id": "e3ec14f9-d5f5-4fc7-a942-ab06f2ff3c89",
            "internalLink": "https://app.castordoc.com/terms/-3e45f6c0",
            "name": "ARR",
            "slug": "-3e45f6c0",
            "score": 0.4504192,
            "tags": [],
            "type": "TERM"
          }
        ]
      }
```

- tables

```json
      {
        "tables": [
          {
            "description": "Each row represents a user in your organization.\nActive users is a measurement metric that is commonly used to measure the level of engagement a particular product or object, by quantifying the number of active interactions from visitors within a relevant range of time (daily, weekly and monthly). The metric has many uses in both commerce and academia, such as on social networking services, online games, or mobile apps. Although having extensive uses in digital behavioural learning, prediction and reporting, it also has impacts on the privacy and security, and ethical factors should be considered thoroughly. Like any metric, active users may have limitations and criticisms. Active Users is relatively new or neologistic in nature, that became important with the rise of the commercialised internet, with uses in communication and social-networking. It measures how many users visit or interact with the product or service over a given interval or period. This metric is commonly assessed per month as monthly active users (MAU), per week as weekly active users (WAU), per day as daily active users (DAU) or peak concurrent users (PCU).\nSELECT Db_Name(Database_id),\n    String_Agg(Login_Name + ' (' + Convert(VARCHAR(8), No_Sessions) + ')', ', ')\n    FROM\n      (\n      SELECT database_id, login_name, Count(*) AS No_Sessions\n        FROM sys.dm_exec_connections AS A\n          INNER JOIN sys.dm_exec_sessions AS B\n            ON A.session_id = B.session_id\n        GROUP BY login_name, database_id\n      ) AS f(Database_id, Login_Name, No_Sessions)\n    GROUP BY Database_id;",
            "id": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "internalLink": "https://app.castordoc.com/data/tables/d-table-user",
            "name": "user",
            "slug": "d-table-user",
            "score": 0.45968664,
            "tableType": "TABLE",
            "tags": [
              "finance",
              "sales",
              "campaign #3",
              "criticality:4-low",
              "source:salesforce",
              "metrics:mrr",
              "domain:marketing"
            ],
            "type": "TABLE"
          },
          {
            "description": "Represents an opportunity, which is a sale or pending deal.",
            "id": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "internalLink": "https://app.castordoc.com/data/tables/d-table-opportunity",
            "name": "opportunity",
            "slug": "d-table-opportunity",
            "score": 0.47584286,
            "tableType": "VIEW",
            "tags": [
              "finance",
              "source:salesforce"
            ],
            "type": "TABLE"
          },
          {
            "description": "",
            "id": "74de041a-0534-4885-88f7-e20268f63e52",
            "internalLink": "https://app.castordoc.com/data/tables/fct-profit-b6a32",
            "name": "fct_profit",
            "slug": "fct-profit-b6a32",
            "score": 0.4596931,
            "tableType": "TABLE",
            "tags": [
              "source:salesforce"
            ],
            "type": "TABLE"
          },
          {
            "description": "",
            "id": "a9c201aa-d8c2-4b12-aad5-dc4043ab7590",
            "internalLink": "https://app.castordoc.com/data/tables/fct-acquisition-events-3ebe5",
            "name": "fct_acquisition_events",
            "slug": "fct-acquisition-events-3ebe5",
            "score": 0.47976118,
            "tableType": "TABLE",
            "tags": [],
            "type": "TABLE"
          },
          {
            "description": "",
            "id": "fed5fbd6-2410-4f54-a0f4-4f4d9724da52",
            "internalLink": "https://app.castordoc.com/data/tables/fct-entities-mrr-reg-aac08",
            "name": "fct_entities_mrr_reg",
            "slug": "fct-entities-mrr-reg-aac08",
            "score": 0.45876038,
            "tableType": "TABLE",
            "tags": [],
            "type": "TABLE"
          }
        ]
      }
```

- dashboards

```json
      {
        "dashboards": [
          {
            "dashboardType": "DASHBOARD",
            "description": "Accounts and Users dashboards",
            "id": "0c75af9a-68e9-4c42-81c7-d83f9d8048c5",
            "internalLink": "https://app.castordoc.com/dashboards/package-cognos-timber-9d8048c5",
            "name": "Accounts and Users",
            "path": "/",
            "slug": "package-cognos-timber-9d8048c5",
            "score": 0.4647782,
            "source": "IBM Cognos",
            "tags": [],
            "technology": "COGNOS",
            "type": "DASHBOARD"
          },
          {
            "dashboardType": "VIZ_MODEL",
            "description": "Catalog Accounts and Users Module viz model",
            "id": "2d4e485f-f6b3-4906-96cf-00fb28a50e90",
            "internalLink": "https://app.castordoc.com/dashboards/package-cognos-timber-28a50e90",
            "name": "Catalog Accounts and Users Module",
            "path": "/__VIZ_MODEL__",
            "slug": "package-cognos-timber-28a50e90",
            "score": 0.46146128,
            "source": "IBM Cognos",
            "tags": [],
            "technology": "COGNOS",
            "type": "DASHBOARD"
          },
          {
            "dashboardType": "TILE",
            "description": "Battery Deliveries Dashboard",
            "id": "8be27aa5-9be5-4f47-9eca-34b09725a108",
            "internalLink": "https://app.castordoc.com/dashboards/8-dashboard-Battery-Deliveries",
            "name": "Battery Deliveries",
            "path": "/operations/deliveries",
            "slug": "8-dashboard-Battery-Deliveries",
            "score": 0.46035412,
            "source": "Looker",
            "tags": [
              "domain:marketing",
              "source:salesforce",
              "finance"
            ],
            "technology": "LOOKER",
            "type": "DASHBOARD"
          },
          {
            "dashboardType": "VIZ_MODEL",
            "description": "All dimensions and metrics about Items contained in Orders",
            "id": "cb5ef340-fc3f-4099-9bbe-31baeb5b4380",
            "internalLink": "https://app.castordoc.com/dashboards/item-pb-timber-df45eb86",
            "name": "Item",
            "path": "/__VIZ_MODEL__",
            "slug": "item-pb-timber-df45eb86",
            "score": 0.7246148,
            "source": "PowerBi",
            "tags": [
              "bi team"
            ],
            "technology": "POWERBI",
            "type": "DASHBOARD"
          },
          {
            "dashboardType": "VIZ_MODEL",
            "description": "All dimensions and metrics about Items contained in Deliveries",
            "id": "e265efe5-3c1c-43f2-a1c6-bf044cbdf196",
            "internalLink": "https://app.castordoc.com/dashboards/package-sig-timber-8b6b5dea",
            "name": "Package",
            "path": "/__VIZ_MODEL__",
            "slug": "package-sig-timber-8b6b5dea",
            "score": 0.73409593,
            "source": "Sigma",
            "tags": [
              "bi team"
            ],
            "technology": "SIGMA",
            "type": "DASHBOARD"
          }
        ]
      }
```

- columns

```json
      {
        "columns": [
          {
            "description": "ID of a related Campaign. This field is defined only for those organizations that have the campaign feature Campaigns enabled.",
            "id": "261a2e2d-531f-4525-bb0e-3c66c6349ff9",
            "internalLink": "https://app.castordoc.com/data/tables/d03f3278-0f9a-40b1-bbe6-f8e7c0d14046/columns/campaign_id",
            "name": "campaign_id",
            "score": 1.719985,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "d03f3278-0f9a-40b1-bbe6-f8e7c0d14046",
            "tableName": "user"
          },
          {
            "description": "The timestamp when the current user last accessed this record, a record related to this record, or a list view.",
            "id": "314906dc-5fd7-4837-876b-bfee39b6ae63",
            "internalLink": "https://app.castordoc.com/data/tables/42bf7abd-10d3-43e3-b594-7ccda7b1f02c/columns/last_referenced_date",
            "name": "last_referenced_date",
            "score": 2.6780999,
            "type": "COLUMN",
            "dataType": "Timestamp",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "42bf7abd-10d3-43e3-b594-7ccda7b1f02c",
            "tableName": "account"
          },
          {
            "description": "Path to be combined with the URL of a Salesforce instance (https://yourInstance.salesforce.com) to generate a URL to request the socialnetwork profile image associated with the account.",
            "id": "49bceb54-f3bc-42d1-8ba1-902364de7228",
            "internalLink": "https://app.castordoc.com/data/tables/42bf7abd-10d3-43e3-b594-7ccda7b1f02c/columns/photo_url",
            "name": "photo_url",
            "score": 2.3301709,
            "type": "COLUMN",
            "dataType": "String",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "42bf7abd-10d3-43e3-b594-7ccda7b1f02c",
            "tableName": "account"
          },
          {
            "description": "Required. This field is a restricted picklist field. A User time zone affects the offset used when displaying or entering times in the user interface.",
            "id": "4e9449da-27d2-477f-becc-96facbc2af74",
            "internalLink": "https://app.castordoc.com/data/tables/2e27d5e4-0662-47b1-a0af-91e04774dcdd/columns/time_zone_sid_key",
            "name": "time_zone_sid_key",
            "score": 2.4199986,
            "type": "COLUMN",
            "dataType": "String",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "tableName": "opportunity"
          },
          {
            "description": "Used with ShippingLongitude to specify the precise geolocation of a shipping address.",
            "id": "59baa6e8-6f03-4031-8df8-029732af1738",
            "internalLink": "https://app.castordoc.com/data/tables/42bf7abd-10d3-43e3-b594-7ccda7b1f02c/columns/shipping_latitude",
            "name": "shipping_latitude",
            "score": 2.4199986,
            "type": "COLUMN",
            "dataType": "Float",
            "isPii": true,
            "isPrimaryKey": false,
            "tableId": "42bf7abd-10d3-43e3-b594-7ccda7b1f02c",
            "tableName": "account"
          },
          {
            "description": "Indicates whether a user has a profile photo (true) ornot (false).",
            "id": "7e1c05d7-33d8-45d7-96d8-efdc93cd82d4",
            "internalLink": "https://app.castordoc.com/data/tables/2e27d5e4-0662-47b1-a0af-91e04774dcdd/columns/is_profile_photo_active",
            "name": "is_profile_photo_active",
            "score": 3.314382,
            "type": "COLUMN",
            "dataType": "Boolean",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "tableName": "opportunity"
          },
          {
            "description": "ID of a related Pricebook2 object. The Pricebook2Id field indicates which Pricebook2 applies to this opportunity. The Pricebook2Id field is defined only for those organizations that have products enabled as a feature. You can specify values for only one field (`Pricebook2Id` or `PricebookId`)—not both fields. For this reason, both fields are declarednillable.",
            "id": "86696957-fabd-4934-acd4-b0b6c599a6a5",
            "internalLink": "https://app.castordoc.com/data/tables/d03f3278-0f9a-40b1-bbe6-f8e7c0d14046/columns/Pricebook2Id",
            "name": "Pricebook2Id",
            "score": 1.5717354,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "d03f3278-0f9a-40b1-bbe6-f8e7c0d14046",
            "tableName": "user"
          },
          {
            "description": "A brief description of an organization’s line of business, based on its SIC code.",
            "id": "9a9cd28a-03f7-4fa5-a824-713d9353a13f",
            "internalLink": "https://app.castordoc.com/data/tables/27f924f6-b702-4f0a-b254-deb1c201929d/columns/sic_desc",
            "name": "sic_desc",
            "score": 2.058366,
            "type": "COLUMN",
            "dataType": "String",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "tableName": "user"
          },
          {
            "description": "A brief description of an organization’s line of business, based on its SIC code.",
            "id": "a2a073ae-7f9c-4587-8291-7ce50ada522a",
            "internalLink": "https://app.castordoc.com/data/tables/d03f3278-0f9a-40b1-bbe6-f8e7c0d14046/columns/sic_desc",
            "name": "sic_desc",
            "score": 2.058366,
            "type": "COLUMN",
            "dataType": "String",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "d03f3278-0f9a-40b1-bbe6-f8e7c0d14046",
            "tableName": "user"
          },
          {
            "description": "The timestamp when the current user last accessed this record, a record related to this record, or a list view.",
            "id": "a6b6f98b-1614-49e6-b767-3bdebed388b1",
            "internalLink": "https://app.castordoc.com/data/tables/27f924f6-b702-4f0a-b254-deb1c201929d/columns/last_referenced_date",
            "name": "last_referenced_date",
            "score": 2.6780999,
            "type": "COLUMN",
            "dataType": "Timestamp",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "tableName": "user"
          },
          {
            "description": "ID of a related Campaign. This field is defined only for those organizations that have the campaign feature Campaigns enabled.",
            "id": "aa081de1-5884-49d3-9f49-754064252323",
            "internalLink": "https://app.castordoc.com/data/tables/27f924f6-b702-4f0a-b254-deb1c201929d/columns/campaign_id",
            "name": "campaign_id",
            "score": 1.719985,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "tableName": "user"
          },
          {
            "description": "References the ID of a company in Data.com. If an account has a value in this field, it means that the account was imported from Data.com.",
            "id": "b2ced45e-7292-4a17-8a5c-62abd61a886d",
            "internalLink": "https://app.castordoc.com/data/tables/42bf7abd-10d3-43e3-b594-7ccda7b1f02c/columns/jigsaw_company_id",
            "name": "jigsaw_company_id",
            "score": 2.3742354,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "42bf7abd-10d3-43e3-b594-7ccda7b1f02c",
            "tableName": "account"
          },
          {
            "description": "ID of a related Pricebook2 object. The Pricebook2Id field indicates which Pricebook2 applies to this opportunity. The Pricebook2Id field is defined only for those organizations that have products enabled as a feature. You can specify values for only one field (Pricebook2Id or PricebookId)—not both fields. For this reason, both fields are declarednillable.",
            "id": "b4c50f3a-8e24-45e2-97e1-177ebfc2754b",
            "internalLink": "https://app.castordoc.com/data/tables/27f924f6-b702-4f0a-b254-deb1c201929d/columns/Pricebook2Id",
            "name": "Pricebook2Id",
            "score": 1.5717354,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "tableName": "user"
          },
          {
            "description": "ID of the Account associated with a Customer Portal user. *This field is null for Salesforce users.*",
            "id": "b6a17d0d-6513-4751-babc-f098cce607fa",
            "internalLink": "https://app.castordoc.com/data/tables/27f924f6-b702-4f0a-b254-deb1c201929d/columns/account_id",
            "name": "account_id",
            "score": 1.8740234,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "27f924f6-b702-4f0a-b254-deb1c201929d",
            "tableName": "user"
          },
          {
            "description": "Used with BillingLatitude to specify the precise geolocation of a billing address.",
            "id": "b9fbee07-197a-4bf9-9ae6-fd0d7f0d1f09",
            "internalLink": "https://app.castordoc.com/data/tables/42bf7abd-10d3-43e3-b594-7ccda7b1f02c/columns/billing_longitude",
            "name": "billing_longitude",
            "score": 2.2028239,
            "type": "COLUMN",
            "dataType": "Float",
            "isPii": true,
            "isPrimaryKey": false,
            "tableId": "42bf7abd-10d3-43e3-b594-7ccda7b1f02c",
            "tableName": "account"
          },
          {
            "description": "The timestamp when the current user last accessed this record, a record related to this record, or a list view.",
            "id": "bc5c15dc-f8de-49b2-b9d1-dadfe9e3f50c",
            "internalLink": "https://app.castordoc.com/data/tables/d03f3278-0f9a-40b1-bbe6-f8e7c0d14046/columns/last_referenced_date",
            "name": "last_referenced_date",
            "score": 2.6780999,
            "type": "COLUMN",
            "dataType": "Timestamp",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "d03f3278-0f9a-40b1-bbe6-f8e7c0d14046",
            "tableName": "user"
          },
          {
            "description": "Read-only field that indicates whether the opportunity has associated line items. A value of true means that Opportunity line items have been created for the opportunity.",
            "id": "bfec4c3f-29b5-47c1-b679-f1e45f1df68c",
            "internalLink": "https://app.castordoc.com/data/tables/d03f3278-0f9a-40b1-bbe6-f8e7c0d14046/columns/has_opportunity_line_item",
            "name": "has_opportunity_line_item",
            "score": 1.443192,
            "type": "COLUMN",
            "dataType": "Boolean",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "d03f3278-0f9a-40b1-bbe6-f8e7c0d14046",
            "tableName": "user"
          },
          {
            "description": "Indicates whether the user is enabled as a forecast manager (true) ornot (false).",
            "id": "e42ee192-f21c-4d8c-bdea-10cb843ec8b2",
            "internalLink": "https://app.castordoc.com/data/tables/2e27d5e4-0662-47b1-a0af-91e04774dcdd/columns/forecast_enabled",
            "name": "forecast_enabled",
            "score": 2.3301709,
            "type": "COLUMN",
            "dataType": "Boolean",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "tableName": "opportunity"
          },
          {
            "description": "The URL for a thumbnail of the user''s profile photo.",
            "id": "ef62a4de-c6b1-4c7e-8a32-cd81354f27bb",
            "internalLink": "https://app.castordoc.com/data/tables/2e27d5e4-0662-47b1-a0af-91e04774dcdd/columns/small_photo_url",
            "name": "small_photo_url",
            "score": 2.2829328,
            "type": "COLUMN",
            "dataType": "String",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "tableName": "opportunity"
          },
          {
            "description": "Id of the user who is a delegated approver for this user.",
            "id": "fced8967-584e-4cab-bfb6-81b3c640e961",
            "internalLink": "https://app.castordoc.com/data/tables/2e27d5e4-0662-47b1-a0af-91e04774dcdd/columns/delegated_approver_id",
            "name": "delegated_approver_id",
            "score": 2.2028239,
            "type": "COLUMN",
            "dataType": "Integer",
            "isPii": false,
            "isPrimaryKey": false,
            "tableId": "2e27d5e4-0662-47b1-a0af-91e04774dcdd",
            "tableName": "opportunity"
          }
        ]
      }
```

---

## AI Assistant

Ask questions about your knowledge base as you would ask a coworker. The AI Assistant uses Knowledge pages, descriptions, and data assets to answer questions and point you to relevant dashboards, tables, or documentation.

## What Does It Do

The AI Assistant answers your questions using filled Knowledge pages, asset descriptions, tags, and pinned content. It also redirects you to the best data assets, dashboards, tables, or Knowledge pages to get your answer.

Here are some examples of questions you can ask:

- What is ARR?
- What dashboards do we have about user activity?
- How do you calculate customer margin?
- What is the revenue evolution per quarter?
- Which tables are used for Google campaigns performance?

<details>

<summary>Questions it cannot answer yet</summary>

- Questions about lineage: give you all downstream dashboards of table X
- Questions about how to use the Catalog: how to tag a table
- Counting or aggregating assets: how many tables are there in schema Y

</details>

For a side-by-side comparison with Dashboard Q&A, see [Comparison Between Dashboard Q&A and AI Search][].

## How To Use It

Open AI Search from the Catalog, then type a question in natural language. The overview image below shows the main search interface.

<figure></figure>

### Where You Can Use It

You can use AI Search in the Coalesce App, Slack, or Microsoft Teams. Expand a section for steps and visuals for each surface.

<details>

<summary>AI Search in the web app</summary>

Use AI Search in the Coalesce App through the [web app][].

Click the image below to open it full screen.

</details>

<details>

<summary>AI Search in Slack</summary>

When you're in a channel, type `@CastorDoc` and ask your question. The Assistant will answer. Invite the assistant to the channel first if it is not already there.

Click the image below to open it full screen.

You need the CastorDoc Slack app installed. See [Slack][] for setup steps.

</details>

<details>

<summary>AI Search in Microsoft Teams</summary>

You can ask your question in two ways:

- In a group chat, mention `@Coalesce`.
- In a 1-on-1 chat with `Coalesce`, type your question in plain text. The screen recording below shows this flow.

The Assistant will reply directly to help you out.

You need the Coalesce Microsoft Teams app installed. Learn how to install it in [Microsoft Teams integration][].

</details>

### Iterate

You can refine answers the same way you would in ChatGPT. If the first answer is not what you need, tell the assistant what to change and ask again. The AI Assistant retrieves a set of relevant assets at the start of a conversation and reuses them for follow-up questions, so when you switch to a different topic, start a new discussion.

## How It Works

This search uses content from your Knowledge pages, dashboards, tables, and columns. Ask it any question, as you would ask someone next to you. It will answer directly and provide the assets used to answer you.

For the technical breakdown of which fields the AI Assistant pulls from each asset type, see the [AI Assistant Context][] reference.

### When New Metadata Appears in Answers

The AI Assistant rebuilds its asset index about once a day. Changes you make in the Catalog, including new or edited descriptions, Knowledge pages, tags, certifications, and deprecations, are typically reflected in answers the next day rather than instantly.

:::info[Plan Ahead for New Documentation]

If you update a Knowledge page or asset description today, expect the AI Assistant to use it in answers about a day later. The same applies to certification and deprecation changes.

:::

Metadata that flows in from connected integrations follows the same daily cadence for AI Search, separate from how often your warehouse data refreshes. For what is and isn't sent to the AI provider, see [Catalog AI Safety][].

## Troubleshooting

If an answer looks wrong, incomplete, or surprising, work through the steps below before assuming the assistant is broken. Most accuracy problems trace back to missing or out-of-date metadata, not the model itself.

### When Answers Look Wrong

Use this flow to investigate a specific answer:

1. **Open the cited assets.** Every answer shows the assets it used. Open them and confirm the description, owner, and tags match the question you asked.
2. **Check certification and deprecation.** Certified assets are boosted by the AI Assistant and deprecated assets are penalized. If a misleading asset keeps appearing, deprecate it. If the right one is missing, certify or document it. The full scoring impact is covered in the [AI FAQ][].
3. **Look at the Knowledge page that should answer the question.** Confirm the page exists, has a clear definition, and pins the relevant tables, columns, or dashboards.
4. **Start a new conversation when you change topics.** AI Search reuses the assets retrieved at the start of a thread. Switching subjects mid-thread can carry stale context into the new question.
5. **Rephrase with the right keywords.** Column searches are lexical, so the exact column name or a close keyword works better than a paraphrase. For asset searches, name the asset type, for example "dashboard," "table," or "Knowledge page."
6. **Confirm the question is in scope.** Lineage traversal, counts of all assets in a schema, and questions about how to use the Catalog in the Coalesce App are not supported yet.
7. **Review Missing Content.** The [AI Assistant Monitoring][] dashboard lists the most-used fields that still lack documentation. Filling those gaps tends to improve answers for the whole team.

If the answer still looks off after these checks, the underlying documentation is likely the gap. Apply the best practices in the [AI FAQ][] to enrich descriptions and pinned content, then re-ask after the next daily sync.

### Verification Checklist

Run through this checklist before sharing an AI Assistant answer with stakeholders or pasting it into another tool:

- [ ] You opened the assets the answer cited and they support the response.
- [ ] None of the cited assets are deprecated, or the deprecation is acknowledged in your share-out.
- [ ] The question matches a supported question type. Lineage, asset counts, and questions about how to use the Catalog in the Coalesce App are not supported yet.
- [ ] You're still in the same topic as the start of the conversation. If you switched topics, start a new discussion and re-ask.
- [ ] Any new descriptions or Knowledge pages you rely on were saved before the most recent daily sync.

## Privacy Concerns

For details, see the [Catalog AI Safety][] notice.

## What's Next?

- [AI FAQ][] for question patterns, scoring impact of certification and deprecation, and accuracy best practices.
- [AI Assistant Context][] for the exact metadata fields each asset type contributes to answers.
- [AI Assistant Monitoring][] for usage analytics and the Missing Content tab.
- [Catalog AI Safety][] for what metadata is shared with OpenAI and how.

---

[web app]: https://app.castordoc.com/ai/search
[Slack]: /docs/catalog/integrations/communication/slack
[Microsoft Teams integration]: /docs/catalog/integrations/communication/microsoft-teams
[Catalog AI Safety]: /docs/catalog/support/catalog-ai-safety
[AI Assistant Context]: /docs/catalog/ai-assistant/ai-assistant/ai-assistant-context
[AI Assistant Monitoring]: /docs/catalog/administration/analytics/ai-assistant-monitoring
[AI FAQ]: /docs/catalog/ai-assistant/ai-faq
[Comparison Between Dashboard Q&A and AI Search]: /docs/catalog/ai-assistant/comparison-between-dashboard-q-and-a-and-ai-search

---

## SQL Copilot

Use SQL Copilot to generate SQL queries from natural language. Find the right tables and get query suggestions in the web app or browser extension.

## What Does It Do

The SQL Copilot answers your data needs. It finds the right tables and writes SQL with them. It is embedded directly in your SQL cloud editors. Ask your questions and it will get you to the right SQL query. Use it in the [web app][] or in the [Catalog Chrome extension][].

Here are some examples of questions you can ask:

- Get the number of users per customer
- What is our revenue per country per month
- Give a list of the newly signed customers of last quarter
- How to join these two tables
- Fix this query (add query after)

## How To Use It

To write SQL, you need knowledge about tables and columns. You can pick the tables manually inside the copilot, or you can use the Table auto-select.

### Where You Can Use It

<details>

<summary>SQL Copilot in the Web App</summary>

Go to [app.castordoc.com/ai/copilot](https://app.castordoc.com/ai/copilot), describe the query you want to write. Iterate with follow-up requests if needed. The job is done.

</details>

<details>

<summary>SQL Copilot in the Browser Extension</summary>

First install the CastorDoc browser extension following the [Browser Extensions][] guide. Then open it when you are about to write SQL in your editor, and ask it your question.

Supported cloud editors by CastorDoc Browser Extension:

- Snowflake SQL Editor
- BigQuery SQL Editor
- Count SQL Editor
- Hex SQL Editor
- dbt Cloud IDE
- Metabase SQL Editor

_Note:_ We can easily add more editors. Let us know your preferences.

</details>

### Iterate

This is a key strength of the assistant. If you are not happy with the first answer, tell the assistant and iterate, as you would with ChatGPT. One thing to keep in mind: Table Auto-Select only works for the first message.

### Tables Auto-Select

<Loom id="d2c329aaa3ec4a13bcc6f90c384b2523" />

When starting a new query, describe what you are looking for and keep the Auto-select tables toggle on. Catalog AI will automatically find the best tables to write your SQL query.

You can then review these tables and edit them by clicking the Edit tables selected button. You will be able to remove tables automatically added and search for other ones you want to add.

## How It Works

### Leverage Your Query History To Find the Best Tables

Catalog checks across all previously run SQL queries, by you, by others, and even by your BI tool. It tries to find the best matching queries to answer your need. These queries will serve as examples for our LLM.

We also extract the tables used by these queries. These tables, along with all the metadata Catalog has (column names and types, descriptions, and so on), will be fed to our LLM and added to the context.

### About Joining Tables

Thanks to our systematic parsing of all your SQL queries, if anyone joined tables together, our assistant will know how to do it. This will feed our assistant as well.

### Build the Query

Finally, we provide our LLM with your question, the best past queries, the best tables to answer, and the relevant metadata. Now the LLM will answer you.

The assistant knows which SQL flavor you are using. It will adapt its answer to it.

## Improving Its Answers

This is where data stewards, analysts, and data engineers play a key role. The better the tables and columns are documented, the better the copilot will answer.

- When documenting columns, add frequent column values for enum types.
- Pin queries to tables in the Query tab in the app
- Certify tables to build trust

## Privacy Concerns

See the [Catalog AI safety][] notice for more information.

[web app]: https://app.castordoc.com/ai/copilot
[Catalog Chrome extension]: https://docs.coalesce.io/docs/catalog/navigate/browser-extensions#how-to-install-the-extension
[Browser Extensions]: https://docs.coalesce.io/docs/catalog/navigate/browser-extensions
[Catalog AI safety]: https://docs.coalesce.io/docs/catalog/support/catalog-ai-safety

---

## AI FAQ

Answers to common questions about the AI Assistant, Dashboard Q&A, and Describe with AI features.

## What Is the Difference Between the AI Assistant and the Dashboard Q&A?

- The **Dashboard Q&A** works within the context of the specific dashboard you are viewing. It uses its associated metadata (queries, calculations, lineage, pinned assets, and more) to generate answers.
- The **AI Assistant** (also referred to as **AI Search**) draws from a broader set of relevant assets across the Catalog, not limited to one dashboard. It uses metadata and content from the most relevant assets to respond to your question. The AI Assistant can:
  - Handle a conversation, taking into account multiple messages to adapt its answer
  - Search on different asset types: tables, columns, dashboards, knowledge pages, and their pinned assets
  - Use asset metadata to create an answer and provide its sources
  - Ask for clarification if there is a doubt on the action to take or if the question is out of its search scope

A summarized comparison between the two is available in the [Comparison Between Dashboard Q&A and AI Search][].

## AI Assistant: What Information Is Being Used?

Curious about what exactly powers the AI Assistant's answers? Or looking for ways to boost its performance by strengthening your documentation and enriching your asset metadata? Check out the full [AI Assistant Context][] page.

## Dashboard Q&A: What Information Is Being Used?

Wondering how Dashboard Q&A knows what to answer? It relies on your dashboard's metadata, such as titles, descriptions, fields, and linked data sets, to provide contextually relevant responses. You can find the full list of metadata used by the Dashboard Q&A in the [Dashboard Q&A Context][] page.

## Describe with AI: What Information Is Being Used?

The **Describe with AI** button generates an asset description based on its contextual information (SQL source query and all other available metadata). You can find the complete list of elements and metadata used by AI to suggest a description in [Describe with AI][].

## What Type of Questions Work Well Today?

**AI Assistant (AI Search)** is best for questions based on asset definitions and related pinned content. Examples:

- _"What tables can help you analyze `<concept>`?"_ Uses table descriptions and Knowledge page pins related to the concept.
- _"In which dashboards can you track `<metric>`?"_ Finds dashboards connected to relevant data assets.
- _"What is the definition of `<term>`?"_ Pulls from documented definitions and descriptions.
- _"What tables contain the column `<column_name>`?"_ Uses **lexical search** (not semantic), based on keyword matches.

**Tips:**

- Specify the asset type in your question when possible.
  - _"Do we have a knowledge page about `<concept>`?"_
  - _"Do we have a dashboard analyzing `<concept>`?"_
- Use terms that most probably appear in asset descriptions or titles.

**Dashboard Q&A** is best when you need insights within the scope of a specific dashboard, such as:

- _"What is this dashboard about?"_
- _"How is this metric calculated?"_
- _"What filters are applied here?"_
- _"Which assets power this dashboard?"_

## Best Practices To Improve AI Accuracy and Surface Key Assets

To ensure better results and increase the visibility of your most important data assets:

- **Write rich, clear descriptions.** Add context and FAQs directly into asset descriptions, especially for frequently asked questions by stakeholders. Prioritize asset descriptions for the ones that you would like to see being easily discoverable.
  - [See templates and examples][]
- **Certify or deprecate assets.** Certified assets are boosted by the AI Assistant, whereas deprecated assets are strongly penalized and have less chance to be part of the Assistant's answers. Certification boosts the asset's score by 1.5×, whereas deprecation cuts the asset's score in half, making it very unlikely to appear in results. By using certifications, you guide users by clearly marking trusted or outdated content.
- **Pin relevant assets** to Knowledge pages to make important relationships explicit and improve discoverability.
- **Use consistent naming and tagging** to ensure metadata is clean and searchable.

## Where Can You Use the AI Assistant?

- Through the Catalog app
- On Slack
- On Microsoft Teams
- On the Chrome extension (the AI Assistant used on the Chrome extension is also called the Dashboard Q&A, and its scope is narrowed to a specific dashboard)
- On Dust
- Through the public API

## What Is on the AI Assistant Roadmap?

Coalesce is actively expanding AI capabilities. Upcoming improvements include:

- Support for lineage-based queries

[Comparison Between Dashboard Q&A and AI Search]: https://docs.coalesce.io/docs/catalog/ai-assistant/comparison-between-dashboard-q-and-a-and-ai-search
[AI Assistant Context]: https://docs.coalesce.io/docs/catalog/ai-assistant/ai-assistant/ai-assistant-context
[Dashboard Q&A Context]: https://docs.coalesce.io/docs/catalog/ai-assistant/dashboard-q-and-a/dashboard-q-and-a-context
[Describe with AI]: https://docs.coalesce.io/docs/catalog/document-your-data/catalog-scribe/describe-with-ai
[See templates and examples]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/descriptions/templates-examples

---

## Comparison Between Dashboard Q&A and AI Search

Use this page when you need to decide between asking about one dashboard in the Catalog browser extension and searching across the Catalog with AI Search.

## Compare at a Glance

Use this table to choose the right capability for your question. For prerequisites, verification, and troubleshooting for Dashboard Q&A, see [Dashboard Q&A][]. For how to use AI Search, see [AI Assistant][].

| | Dashboard Q&A | AI Search |
| --- | --- | --- |
| Availability | Catalog browser extension in Chrome or Edge | Catalog App, Slack, Microsoft Teams, Dust, Public API |
| Purpose | Answer your question about the dashboard you have open | Answer your question and gather relevant assets |
| Input | Your question and the dashboard URL the extension matches | Your question |
| Output | An answer to your question | An answer and a list of relevant assets used to generate the answer |
| Context | Deeper metadata for one dashboard: descriptions, SQL, lineage-related fields, pinned assets, owners, and related context | Broader search across dashboards, tables, terms, columns, and other assets, with more generic depth per asset. For Catalog assets, also includes content pinned to related Knowledge pages |

## What's Next?

- [Dashboard Q&A][] for setup, usage, and troubleshooting
- [AI Assistant][] for AI Search in the Catalog App, Slack, Teams, and other surfaces

---

[AI Assistant]: /docs/catalog/ai-assistant/ai-assistant
[Dashboard Q&A]: /docs/catalog/ai-assistant/dashboard-q-and-a

---

## Dashboard Q&A Context

<!-- vale off -->

This page lists the information used for the Dashboard Q&A context as of July 7, 2025.

Dashboard Q&A takes into account information and extra metadata related to a **single** `dashboard`, allowing it to answer questions specifically about that dashboard.

This feature is only available through the Coalesce Catalog Chrome extension, which detects dashboards based on their URLs.

* Dashboard
  * additionalContext
  * descriptions
    * descriptionRaw
    * externalDescription
  * folderPath
  * isDeprecated
  * isVerified
  * name
  * source (sourceName)
  * tags (label)
  * technology
  * type
  * sqlSourceQueries: `DASHBOARD`/ `TILE`/ `VIZ_MODEL`
* Other metadata
  * editors
    * email
    * name
    * type = `SOURCE_USER`
  * frequentUsers
    * email
    * name
    * type = `SOURCE_USER`
  * pinnedAssets (assets that are pinned to this `dashboard`)
    * description
    * name
    * path
    * type: `COLUMN`/ `DASHBOARD`/ `DASHBOARD_FIELD`/ `TABLE`/ `TERM`
  * mentionedInAssets (assets that this `dashboard` is pinned to)
    * description
    * name
    * path
    * type: `COLUMN`/ `DASHBOARD`/ `DASHBOARD_FIELD`/ `TABLE`/ `TERM`
  * owners
    * userOwners
      * email
      * name
      * type = `USER` / `SOURCE_USER`
    * teamOwners
      * name
      * type = `TEAM`
  * parentAssets (parent tables/dashboards)
    * parentTables
      * description
      * name
      * path
      * type = `TABLE`
    * parentDashboards
      * description
      * name
      * path
      * sqlSourceQueries
      * type: `DASHBOARD`/ `TILE`/ `VIZ_MODEL`
  * parentFields (parent columns/fields)
    * columns
      * name
      * table
        * name
        * path
      * type = `COLUMN`
    * dashboardFields
      * name
      * model
        * name
        * path
      * type = `FIELD`

* _payload example_

    ```json
    {
      "dashboard": {
        "additionalContext": {
          "report_page_names": [
            "Yearly Overview",
            "Yearly BU Details ",
            "RMA Quarterly Monitoring",
            "Climate Assets Monitoring ",
            "Climate Assets Details",
            "Climate Asset Scope Variations",
            "QBR GHG emissions",
            "Tooltip nb of ClA published (new)",
            "Data Completion Tracking (new)",
            "TT completion tracking",
            "Details - Budget per BU",
            "----BLANK----",
            "----->",
            "WIP - Climate Asset Detailed Activities"
          ]
        },
        "descriptions": [
          "🎯 PURPOSE\n\nVeolia has committed to achieving carbon neutrality by 2050 and has launched a\nstrategic plan for 2024-2027 called GreenUp. The main objective is to reduce GHG\nemissions by 50% on scopes 1 and 2 by 2032. To achieve this, the company is\nmobilizing all its units and geographical areas.\n\nTo track its progress, Veolia is implementing a quarterly monitoring process for\nGHG emissions, focusing particularly on climate assets. This process, led by the\nFinance department, will begin in 2024 and will strengthen the current annual\nglobal GHG emissions reporting campaign.\n\n\n⚠️ GOOD TO KNOW\n\nThe data is broken down by scope (scope 1 / scope 2 / total emissions), which\nare globally accepted ways of describing greenhouse gas emissions\n(https://www.nationalgrid.com/stories/energy-explained/what-are-scope-1-2-3-carbon-emissions)\n\n\n📄 CONTENT\n\n\nMETRICS\n\nWhat are the key metrics included in this dashboard - provide links to their\ndefinitions (don't forget to pin all the relevant dashbaords to the metric\ndefinition)\n\n\nDIMENSIONS\n\n * Dim_BU : Managerial organization of Veolia\n * Dim_Entity : Key information about the sites: name, flag\n   “key_asset_for_climate”, entry & exit date of program, etc.. \n * Dim_Calendar : From Jan 1 2021 to end of current year\n\n\n🛢 SOURCE\n\nWhere is the data sourced from (Salesforce, Jira etc)\n\n\n📆 DATA UPDATE:\n\nHow often is the data updated\n\n\n",
          null
        ],
        "folderPath": "/IST-VE-GHG Monitoring Dashboard - PROD",
        "isDeprecated": false,
        "isVerified": true,
        "name": "GHG Monitoring Dashboard - PROD",
        "source": "PowerBI Corp",
        "tags": [
          "domain:finance",
          "domain:sustainability"
        ],
        "technology": "POWERBI",
        "type": "DASHBOARD",
        "sqlSourceQueries": []
      },
      "editors": [
        {
          "email": "data.visualisation@veolia.com",
          "name": "data visualisation",
          "type": "SOURCE_USER"
        }
      ],
      "frequentUsers": [
        {
          "email": "lucas.davain.ext@veolia.com",
          "name": "Lucas DAVAIN",
          "type": "SOURCE_USER"
        },
        {
          "email": "francois.porret.ext@veolia.com",
          "name": "François PORRET",
          "type": "SOURCE_USER"
        },
        {
          "email": "jana.patkova@veolia.com",
          "name": "Jana Pátková",
          "type": "SOURCE_USER"
        },
        {
          "email": "otmane.ait-bahajji@veolia.com",
          "name": "Otmane Ait-Bahajji",
          "type": "SOURCE_USER"
        }
      ],
      "pinnedAssets": [
        {
          "description": "DEFINITION\n\nThe \"number of equipment by Severity\" is a measure of the number of equipment\nthat have been ranked by severity. This indicator can be used to monitor the\noverall condition of an organization's equipment fleet and to identify those\nwhere maintenance or repair efforts may be required.\n\nTo use this indicator, you first need to establish a severity assessment for\nyour equipments. This assessment should be based on factors according to\nVeolia's standard methodology.\n\n\nREFERENCE\n\nA11\n\n\nCATEGORY\n\nSeverity & Condition Assessment\n\n\nCALCULATION METHOD & FORMULA\n\n\n\n\n\nDESCRIPTIVE EXAMPLE\n\n\n\n\n\n\n\nGOOD TO KNOW\n\n * The severity of equipment is an important factor to consider in maintenance\n   and asset management. Highly Severity equipment requires more frequent\n   maintenance and inspection to reduce the risk of failure, while low\n   criticality equipment can be maintained less frequently. Overall,\n   understanding the Severity of equipment is essential for effective\n   maintenance planning, risk management, and business continuity.\n * A11 : The number of equipment registered in CMMS represented by level of\n   Severity\n\n\n\n\n\n\n\nLAST REVIEW DATE\n\n\n",
          "name": "A11 : Total number of equipments by severity (criticality) level",
          "type": "TERM"
        },
        {
          "description": "DEFINITION\n\nGreenhouse gases (GHGs) are gases present in the Earth's atmosphere that have\nthe ability to trap heat and contribute to the greenhouse effect, leading to\nglobal warming and climate change. These gases effectively absorb and emit\ninfrared radiation, preventing a portion of the heat from escaping into space\nand causing the Earth's surface and lower atmosphere to warm.\n\n\nDESCRIPTIVE EXAMPLE*\n\nCarbon dioxide (CO2): This is the most prevalent greenhouse gas and is primarily\ngenerated through the burning of fossil fuels (such as coal, oil, and natural\ngas), deforestation, and other industrial processes.\n\nMethane (CH4): Methane is produced by natural processes, including the\ndecomposition of organic matter in landfills, wetlands, and agricultural\nactivities, as well as through the extraction and transportation of fossil\nfuels.\n\nNitrous oxide (N2O): Nitrous oxide is released through agricultural and\nindustrial activities, as well as the combustion of fossil fuels and solid\nwaste.\n\nFluorinated gases: These are synthetic gases used in various industrial\napplications, including refrigeration, air conditioning, and electronics\nmanufacturing. They have high global warming potentials and can persist in the\natmosphere for a long time.\n\n\nBUSINESS OBJECT'S DATA MODEL\n\nhttps://lucid.app/publicSegments/view/1e0c61ce-c1cd-4bdd-9fe9-433573858293/image.png\n\n\n\n\nBUSINESS OBJECT'S LIFECYCLE\n\nhttps://lucid.app/publicSegments/view/0009f0f7-a47d-4222-b66d-9b2ba8238a8e/image.jpeg\n\n\n\n\n\n\nhttps://lucid.app/publicSegments/view/30992d62-5711-452b-a342-8741d9f6e9f8/image.jpeg\n\n\n\n\n\n\n\nTARGET POINT OF TRUTH\n\n> Represent the system which we want to be the system source truth in the futur.\n\n\nREQUIREMENTS\n\n 1. Business :\n\ndéjà ça c'est le lien du GHG Protocol officiel :nhttps://ghgprotocol.org/sites/default/files/standards/ghg-protocol-revised.pdf scope\n3 en plus dans le Beges \n\n 2. Quality :\n\nCollection as it happens and GHG Emission calculated depending on data (most\nfreshness needed)\n\nCollected quarterly from BU with the CSV files\n\n 3. Compliance :\n\n * * * Décret BEGES\n     * GreenHouse Gas Protocol\n       \n\n\nBUSINESS OBJECTS ATTRIBUTES\n\n * [BOA] Bu Code - Mandatory\n * [BOA] GHG emission value - Mandatory\n * [BOA] GHG KPI Code - Mandatory\n * [BOA] Period End - Mandatory\n * [BOA] Periodicity - Mandatory\n * [BOA] Period Start - Mandatory\n * [BOA] Source System Name - Optional\n * [BOA] Source System Update Timestamp - Optional\n * [BOA] GHG KPI Reference (Group Id) - Optional\n * [BOA] Operational Reporting Entity - Id - Optional\n * [BOA] GHG HQ unit code - Optional\n * [BOA] GHG Value in international unit - Optional\n * [BOA] GHG international unit code - Optional\n * [BOA] GHG Value in local unit - Optional\n * [BOA] GHG Local unit code - Optional\n * [BOA] Kpi Is Validated - Optional\n * [BOA] Kpi Value Comment - Optional\n * [BOA] Study scenario name - Optional\n * [BOA] Study context - Optional\n * [BOA] GHG Scope - Optional\n\n\n",
          "name": "[BO] GreenHouse Gas",
          "type": "TERM"
        }
      ],
      "mentionedInAssets": [],
      "owners": [
        {
          "email": "otmane.ait-bahajji@veolia.com",
          "name": "Otmane Ait-Bahajji",
          "type": "SOURCE_USER"
        },
        {
          "name": "DB&T Core",
          "type": "TEAM"
        }
      ],
      "parentAssets": [
        {
          "description": "",
          "name": "01_Fact_Append_Global_GP_part",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_Business_Segment",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_business_segment",
          "type": "TABLE"
        },
        {
          "description": "Organizational structure that corresponds to a managerial breakdown allowing to monitor the performance.\n\n[BigQuery](https://console.cloud.google.com/bigquery?project=gbl-bsp-ve-kakpigroup&ws=!1m5!1m4!4m3!1sgbl-ist-ve-datalakefo!2s00063_domain210_finance_group!3smanagerial_organization)\n[Specification FO](https://docs.google.com/spreadsheets/d/1aKPM3gIO1xssz3QnFYqUwUL3EDZBl9LvTYZ7Pgrvuf0/edit#gid=0)",
          "name": "managerial_organization",
          "path": "gbl-ist-ve-datalakefo.00063_domain210_finance_group.managerial_organization",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_Calendar",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_Calendar",
          "path": "gbl-ist-ve-datamarts-dev.01316_ghg_monitoring_group.dim_calendar",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "01_Fact_Append_Global_GP_part_scopevariation",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part_scopevariation",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_GH_Key_Assets",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_gh_key_assets",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_MERGE_Operational_Reporting_Entity",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_Operational_Reporting_Entity",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_operational_reporting_entity",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_MERGE_Operational_Reporting_Entity",
          "path": "gbl-ist-ve-datamarts-dev.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "Dim_Managerial_Org",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_managerial_org",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "01_Fact_Append_Global_GP_completion_follow_up",
          "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up",
          "type": "TABLE"
        },
        {
          "description": "",
          "name": "GHG Monitoring Dashboard - PROD",
          "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD",
          "sqlSourceQueries": [],
          "type": "VIZ_MODEL"
        }
      ],
      "parentFields": [
        {
          "name": "managerial_organization_level_4_code",
          "table": {
            "name": "Dim_Managerial_Org",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_managerial_org"
          },
          "type": "COLUMN"
        },
        {
          "name": "country",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "Month",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "shared",
          "table": {
            "name": "01_Fact_Append_Global_GP_completion_follow_up",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up"
          },
          "type": "COLUMN"
        },
        {
          "name": "Activity Climate assets",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "business_level_3_code",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "asset_typology_climate",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "operational_reporting_entity_id",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "Zone",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "operational_reporting_entity_name",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "last_day",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "Date",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "Quarter",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "longitude",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_2_code",
          "table": {
            "name": "01_Fact_Append_Global_GP_completion_follow_up",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up"
          },
          "type": "COLUMN"
        },
        {
          "name": "Trigram Activity",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "key_asset_for_climate",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_4_name",
          "table": {
            "name": "Dim_Managerial_Org",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_managerial_org"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_4_code",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "bu_code",
          "table": {
            "name": "01_Fact_Append_Global_GP_part_scopevariation",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part_scopevariation"
          },
          "type": "COLUMN"
        },
        {
          "name": "operational_reporting_entity_gid",
          "table": {
            "name": "01_Fact_Append_Global_GP_part_scopevariation",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part_scopevariation"
          },
          "type": "COLUMN"
        },
        {
          "name": "period_end",
          "table": {
            "name": "01_Fact_Append_Global_GP_completion_follow_up",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up"
          },
          "type": "COLUMN"
        },
        {
          "name": "date_finale",
          "table": {
            "name": "01_Fact_Append_Global_GP_part_scopevariation",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part_scopevariation"
          },
          "type": "COLUMN"
        },
        {
          "name": "Month Number",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "operational_reporting_entity_gid",
          "table": {
            "name": "01_Fact_Append_Global_GP_completion_follow_up",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up"
          },
          "type": "COLUMN"
        },
        {
          "name": "latitude",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "Year",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_4_name",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_3_name",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "managerial_organization_level_3_name",
          "table": {
            "name": "Dim_Managerial_Org",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_managerial_org"
          },
          "type": "COLUMN"
        },
        {
          "name": "bu_code",
          "table": {
            "name": "01_Fact_Append_Global_GP_part",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_part"
          },
          "type": "COLUMN"
        },
        {
          "name": "Quarter_",
          "table": {
            "name": "Dim_Calendar",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_calendar"
          },
          "type": "COLUMN"
        },
        {
          "name": "validated",
          "table": {
            "name": "01_Fact_Append_Global_GP_completion_follow_up",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.01_fact_append_global_gp_completion_follow_up"
          },
          "type": "COLUMN"
        },
        {
          "name": "Climate assets",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "operational_reporting_entity_gid",
          "table": {
            "name": "Dim_MERGE_Operational_Reporting_Entity",
            "path": "gbl-ist-ve-datamarts.01316_ghg_monitoring_group.dim_merge_operational_reporting_entity"
          },
          "type": "COLUMN"
        },
        {
          "name": "Total Veolia",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "unit_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Formula",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "key_asset_for_climate",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_Metric_unit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "test_non_null selected measure",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Zone",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test3",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CO2eq_",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Count KA",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio_2023_bis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb ka in scope",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CO2eq",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Exists_nb_validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Measure_to_filter_scopevar",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Exists_nb_total",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Scope_Level",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_gid",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_delta ISOs_bridge",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP Sc1+Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio_2022_bis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_denom INN02",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_DelegatedZone",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% of total Selected Measure GR sc1 + sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "end_date",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Quarters selected",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Climate Asset",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mutual",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_2_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CO2Eq Scope GHG",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_Exit_neg",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% Actual Vs budget_recalc",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Order",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test_sum_6",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Millions tCO2eq",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Scope_all_climate_assets",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GP sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_Date",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test5",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "E_3_9_Scope 1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_delta CO2eq",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_gid",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Value_Unit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_gid",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "test_zone_periodicity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb_total minus others",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Unit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CO2eq N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_Entry",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GR Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc1+sc2 PY",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_4_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_Entity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_today",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Order",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "business_level_3_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% Actual Vs budget_cl.a",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_ISO N",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Quarter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_blank period N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "value",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GR Sc1+Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "longitude",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Value_Unit_",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_gid",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "shared",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Cl.GR Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Min Selected Period",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_id",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "start_date",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Year__",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Comment",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Measure_to_filter_bis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Min KPI Selected Period_01",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ISO N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Trigram Activity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Date_n-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 Sc1+Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Sort",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "label_N",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Climate assets",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Activité ",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Scope",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ScopeVariation CO2eq N Entry",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CO2Eq Scope GHG_cl.a",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_bridge step intermediaire",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_unicode",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "source_system_update_time",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_end",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "datasource",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Exit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_denom E-4-7-3",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "E_3_4_4_Scope 2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_gid",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_percent validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Description",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_end",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Scope",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected unit Baseline 2021 kilotons _Total veolia",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc2 PY",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Unit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "NewAxis1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "bu_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Quarter_",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ISO N",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Baseline 2022 Millions",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Cl.GR Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "country",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR CLA sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "kpi_value_id",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure sc1+sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_denom_bis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ScopeVariation CO2eq N-1 ISO",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% Difference from Baseline",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "max period end",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Emission for climate asset",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test6",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CTD ClA Budget",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb shared",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "gpdp_context_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "value",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Budget value_cl.a",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "value_budget_per_climate_asset",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_end",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_BU",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_denom",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% of total budget",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb_total",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "New Axis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_4_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "key_asset_for_climate",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "date_finale_nextyear",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_CO2eq N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "MIN_DATE N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "kpi_is_validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "NewAxis1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "MAX_DATE",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_3_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Activity Climate assets",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Measure_to_filter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "last_day",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Month Number",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 PY Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_ISO N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Unit Baseline 2021 Millions",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_4_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_start",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Column1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_CO2eq",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ScopeVariation CO2eq N ISO",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Indicator name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Scope_Level_GR",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "kpi_value_id",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_nb_total minus validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_num",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_Exit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Baseline 2021 Millions",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "New Axis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Measure_to_filter_dim_entity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CL.GP Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Emissions (Numerator)",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio_bis",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_count of ka",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_4_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Activity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "scope_GHG",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Measure",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "asset_typology_climate",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Exists_nb_shared",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "date_finale",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Total Veolia remove filter_Million tons",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "value_",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "latitude",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Sort",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Main emission factor (Denominator)",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_ratio N",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI Value",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GR Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure N-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 PY Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "validated",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_iso n bridge",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "TT_Metric_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "ScopeVariation CO2eq N-1 Exit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "bu_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_4_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Intensity factor_denom INN46",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% Actual Vs budget",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Budget value",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_end",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "managerial_organization_level_3_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "source_system_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc1 PY",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Total velio Millions tons",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Parameter",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Program.1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Max KPI Selected Period",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_percentage ClA published",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Date",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Month",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "CL.GP Sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Min KPI Selected Period",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_blank period N",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "GP1 PY Sc1+Sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "periodicity",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "MAX_DATE_N-1_YTD",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "bu_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test4",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "selectedUnit",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Year",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "operational_reporting_entity_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Program",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Trigram_",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_name",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR CLA sc1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "_Entry_bridge",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Mesure_test_sum_4",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "KPI_Exists",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GR sc1 + sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% of total Selected Measure GR sc1 + sc2 PY",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Nb of Climate Asset having published",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Total KA for climate_latest",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Year n-1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "unit_code",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Zone",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "value",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "% Emission Climate asset VsTotal asset_Veolia",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Selected Measure GP sc2",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "Entry",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        },
        {
          "name": "period_end +1",
          "model": {
            "name": "GHG Monitoring Dashboard - PROD",
            "path": "/__VIZ_MODEL__/IST-VE-GHG Monitoring Dashboard - PROD/GHG Monitoring Dashboard - PROD"
          },
          "type": "FIELD"
        }
      ]
    }
    ```

---

## Dashboard Q&A

While you view a dashboard in a supported BI tool, Dashboard Q&A answers natural-language questions about how that dashboard is built, where its data comes from, and which Catalog assets power it. You use it from the Catalog [Browser Extension][] on the page you already have open.

## Before You Begin

Confirm the following before you expect the **Dashboard Q&A** tab to appear:

- **Browser extension** - Install the Catalog extension in Chrome or Edge. See [Browser Extensions][] for install steps and supported BI tools.
- **Catalog sign-in** - Sign in to your Catalog Workspace in the same browser profile you use for the BI tool so the extension can load metadata you are allowed to see.
- **Dashboard in Catalog** - The dashboard you are viewing must be ingested through your organization's BI integration, and extraction must be complete. If the dashboard was added recently, wait for the next sync or run your extraction process according to your setup.
- **Supported dashboard URL** - The extension matches the open page to a dashboard asset in Catalog. Unsupported hosts, or dashboards that load only inside embedded iframes, can block overlays. See [Browser Extensions][] troubleshooting if panels stay empty.
- **Workspace capability** - Some Workspaces include Dashboard Q&A as a licensed capability. If your Workspace does not include it, the **Dashboard Q&A** tab may not appear.
- **Catalog AI enabled** - An administrator must request AI activation for your account before AI features send metadata to the AI provider. Until activation completes, AI capabilities may be unavailable. See [Catalog AI Safety][].

When the extension cannot match your current page to a dashboard in Catalog, you may see other extension capabilities, such as **SQL Copilot** or an **Info** tab, instead of **Dashboard Q&A**.

## What Dashboard Q&A Does

Dashboard Q&A is scoped to one dashboard: the page you have open in your BI tool. It uses deeper metadata for that dashboard (descriptions, SQL, lineage-related fields, pinned assets, owners, and related context) to answer your question. Answers are based on **metadata in Catalog**, not live query results from your warehouse.

Here are example questions you can ask:

- "What data is used in this dashboard?"
- "What is the definition of this metric?"
- "What related assets should you check to get more context around this dashboard?"
- "Show the SQL query for this visualization"
- "What elements are included in this dashboard?"
- "Which tables and columns are used here?"

<details>

<summary>Questions that are out of scope or limited today</summary>

- Org-wide discovery, such as "Do we have a dashboard about churn?" In that case, use [AI Search][] in the Catalog App, Slack, or Microsoft Teams instead.
- Impact of failed data quality tests on this dashboard. Use data quality surfaces in Catalog where your organization has them configured.
- Broad lineage traversal, asset counts across a schema, or how to use Catalog in the Coalesce App. Many of the same limits apply to [AI Search][]; see the AI Assistant page for the current list.

</details>

## How To Use It

1. Install and enable the Catalog extension. Follow [How to Install the Extension][] on the Browser Extensions page.
2. Open a dashboard in a supported BI tool (Tableau, Looker, Power BI, Domo, Metabase, Sigma, Strategy, Qlik, Looker Studio, and others listed on [Browser Extensions][]).
3. Sign in to Catalog in the same browser if you are not already signed in.
4. Open the extension from the browser toolbar. The panel opens beside your BI content.
5. Select the **Dashboard Q&A** tab when it is available, then type your question in natural language.

<figure><figcaption>Dashboard Q&A tab in the Catalog browser extension</figcaption></figure>

When the extension recognizes the open dashboard, use the **Info** tab for metadata such as owners and descriptions, and **Dashboard Q&A** for questions about that dashboard. If you do not see **Dashboard Q&A**, work through [Before You Begin][] and [Troubleshooting][].

You can refine answers in a conversation. If you change topics, reset, or start a fresh conversation so earlier context does not carry over. Conversation history is shared across extension AI features in the same session.

For Looker-specific URL matching and verification, see [Looker and the Catalog Browser Extension][].

## Choose Dashboard Q&A or AI Search

Use this guide to pick the right capability for your question:

- **Dashboard Q&A** - You are on a dashboard in a supported BI tool and your question is about this dashboard: build, metrics, tables, or SQL for a visualization.
- **AI Search** - You are in the Catalog App, Slack, or Microsoft Teams and need discovery across many assets or org-wide definitions.

| Situation | Use |
| --- | --- |
| On a dashboard in Tableau, Looker, Power BI, or another supported tool | Dashboard Q&A |
| Question is about the dashboard you are viewing | Dashboard Q&A |
| Need depth on one asset (tables, columns, SQL for this dashboard) | Dashboard Q&A |
| In the Catalog App, Slack, or Teams | AI Search |
| Question spans many dashboards, tables, columns, or Knowledge pages | AI Search |
| Need a list of relevant assets plus an answer | AI Search |

For availability, input, output, and context in more detail, see [Comparison Between Dashboard Q&A and AI Search][].

## How It Works

Dashboard Q&A sends your question together with metadata for the dashboard the extension matched from your current URL. That includes dashboard fields, descriptions, SQL source queries, parent tables, pinned, and mentioned assets, owners, editors, tags, and certification state. For the full field list, see [Dashboard Q&A Context][].

To improve answer quality:

- Write clear descriptions on the dashboard and on tables or terms it uses.
- Certify trusted assets and deprecate outdated ones so scoring favors the right content.
- Pin related Knowledge pages and assets to the dashboard.

The **Describe with AI** action on a dashboard uses the same contextual metadata as Dashboard Q&A. See [Describe with AI][] for how descriptions are generated.

Changes you make in Catalog follow your BI extraction and Catalog indexing cadence. New or edited descriptions may not appear in answers until the dashboard metadata is synced and available to the extension.

:::info[Metadata Only]

Dashboard Q&A does not run queries against your warehouse. It answers from Catalog metadata only. For what is sent to the AI provider and when, see [Catalog AI Safety][].

:::

## Confirm It Is Working

Run through this checklist when you first use Dashboard Q&A or after large BI changes:

- [ ] You opened a dashboard that exists in Catalog (search Catalog first if you are unsure).
- [ ] The extension is installed and enabled in Chrome or Edge, and you are signed in to the correct Catalog Workspace and region.
- [ ] The extension **Info** tab shows metadata (owners, description, popularity) for **this** dashboard, not a different asset.
- [ ] The **Dashboard Q&A** tab is visible when your Workspace includes the capability.
- [ ] A simple factual question, such as "What tables power this dashboard?", returns an answer that aligns with Catalog lineage or metadata for that dashboard.
- [ ] If you changed topics, you started a new conversation or reset the thread before asking again.

If metadata on **Info** looks wrong or empty, fix extraction and URL matching before you rely on Dashboard Q&A answers. For Looker, follow [Looker and the Catalog Browser Extension][].

## Troubleshooting

Most issues come from extension setup, dashboard matching, or metadata gaps rather than the model alone.

### No Dashboard Q&A Tab

The extension did not match your current page to a dashboard in Catalog, or your Workspace does not include Dashboard Q&A.

1. Confirm the dashboard exists in Catalog and BI extraction has finished.
2. Open the dashboard in a top-level browser tab when your BI tool embeds dashboards in iframes. See [Browser Extensions][] troubleshooting.
3. Refresh the BI tab and open the extension again after sync completes.
4. Confirm your Workspace includes Dashboard Q&A and that an administrator enabled Catalog AI. See [Before You Begin][] and [Catalog AI Safety][].

Seeing **SQL Copilot** without **Dashboard Q&A** is expected when no Catalog dashboard matches the open URL.

### Metadata Panel Is Empty or Describes the Wrong Dashboard

1. Sign in to Catalog in the same browser profile and confirm you use the correct region or tenant for your organization.
2. Compare the open URL to the dashboard asset in Catalog (name, path, source).
3. Check **Settings > Integrations** that the BI source completed ingestion.
4. For Looker URL patterns, see [Looker and the Catalog Browser Extension][].

For sign-in failures, wrong region, or empty toolbar panels, see [Browser Extensions][] troubleshooting.

### Answers Look Wrong or Too Generic

1. Open the dashboard in Catalog and confirm descriptions, owners, and pinned assets match what you asked about.
2. Verify the extension matched **this** dashboard on the **Info** tab before you trust a Dashboard Q&A reply.
3. Certify trusted assets and deprecate misleading ones. See [AI FAQ][] for scoring impact and question patterns.
4. Rephrase with terms that appear in asset titles or descriptions.
5. Start a new conversation when you switch to a different subject.

If the answer still looks off, enrich documentation on the dashboard and its parent tables, then ask again after metadata syncs.

### Questions About Data Quality Test Failures

Dashboard Q&A does not answer which failed tests affect this dashboard or similar data quality impact questions today. Use your organization's data quality views in Catalog where available.

## What's Next?

- [Comparison Between Dashboard Q&A and AI Search][] for a full side-by-side table.
- [Dashboard Q&A Context][] for metadata fields used in answers.
- [AI FAQ][] for differences between Dashboard Q&A and AI Search, and accuracy best practices.
- [AI Assistant][] for AI Search in the Catalog App, Slack, and Teams.
- [Catalog AI Safety][] for what metadata is shared with the AI provider, activation requirements, and privacy commitments.
- [Browser Extensions][] for install, supported technologies, and extension-wide troubleshooting.
- [Looker and the Catalog Browser Extension][] for Looker prerequisites and URL matching.

---

[AI Search]: /docs/catalog/ai-assistant/ai-assistant
[AI Assistant]: /docs/catalog/ai-assistant/ai-assistant
[AI FAQ]: /docs/catalog/ai-assistant/ai-faq
[Before You Begin]: #before-you-begin
[Browser Extension]: /docs/catalog/navigate/browser-extensions
[Browser Extensions]: /docs/catalog/navigate/browser-extensions
[Catalog AI Safety]: /docs/catalog/support/catalog-ai-safety
[Comparison Between Dashboard Q&A and AI Search]: /docs/catalog/ai-assistant/comparison-between-dashboard-q-and-a-and-ai-search
[Dashboard Q&A Context]: /docs/catalog/ai-assistant/dashboard-q-and-a/dashboard-q-and-a-context
[Describe with AI]: /docs/catalog/document-your-data/catalog-scribe/describe-with-ai
[How to Install the Extension]: /docs/catalog/navigate/browser-extensions#how-to-install-the-extension
[Looker and the Catalog Browser Extension]: /docs/catalog/integrations/data-viz/looker/looker-chrome-extension
[Troubleshooting]: #troubleshooting

---

## Introduction to the Catalog AI Assistant

The Catalog AI Assistant has three capabilities: AI Search, SQL Copilot, and Dashboard Q&A. Use them to find data, write SQL faster, and understand your dashboards.

## AI Search: Find Data You Need

Have you ever felt that finding information in a data catalog was tricky for people on the business side? The AI Search feature helps. Use it in the [web app][], in the Catalog [Slack][] or [Microsoft Teams][] apps.

### Search Assistant and Natural Language Filters

In AI Search, the Search Assistant can turn plain language into filters so you don't have to open the filter panel for basic narrowing. It recognizes these dimensions:

- **Owner:** Filter by owner for individual contributors or teams; partial or full match.
- **Tag:** Filter by tags on assets and on columns where column tags apply.
- **Name:** Filter by asset name; partial or full match.

For example: _Which tables are owned by Alice?_, _Show dashboards tagged GDPR_, or _Find assets named customer_.

The Catalog search bar also offers **Advanced search** filters that are not driven through this conversational path: result types such as tables, columns, dashboards, visualization models, fields, or Knowledge; **Certified**; **PII**; and tag or owner logic with **AND**, **OR**, or **exclusion**. Use those when you need that level of control. See [Search Your Asset][].

You can still ask broader discovery questions without naming a filter, for example below.

> Question example: _What is ARR?_

<figure><figcaption>AI Search in the Catalog</figcaption></figure>

Find out more in [AI Search][].

## SQL Copilot: Write SQL Faster

The SQL Copilot helps answer your data needs. It finds the right tables and writes SQL with them. It is embedded directly in your SQL cloud editors. Ask your questions, and it will get you to the right SQL query. Use it in the [Catalog web app][] or in the Catalog [Chrome extension][].

> Question example: _Get the number of users per customer_

<figure><figcaption>SQL Copilot in the Catalog</figcaption></figure>

Find out more in [SQL Copilot][].

## Dashboard Q&A: Understand Your Dashboards

When looking at a dashboard on Tableau, Looker, Power BI, and more, you can ask any question about how the dashboard is built, where the data comes from, right from the Catalog browser extension. Use it in the Catalog [browser extension][].

> Question example: _How is the dashboard built?_

<figure><figcaption>Dashboard Q&A from the extension</figcaption></figure>

Find out more in [Dashboard Q&A][].

## Privacy Concerns

Refer to the [Catalog AI safety][] notice for more information.

[web app]: https://app.castordoc.com/ai/search
[Catalog web app]: https://app.castordoc.com/ai/copilot
[Slack]: /docs/catalog/integrations/communication/slack
[Microsoft Teams]: /docs/catalog/integrations/communication/microsoft-teams
[Chrome extension]: /docs/catalog/navigate/browser-extensions#how-to-install-the-extension
[browser extension]: /docs/catalog/navigate/browser-extensions#how-to-install-the-extension
[AI Search]: /docs/catalog/ai-assistant/ai-assistant
[SQL Copilot]: /docs/catalog/ai-assistant/ai-assistant/sql-copilot
[Dashboard Q&A]: /docs/catalog/ai-assistant/dashboard-q-and-a
[Catalog AI safety]: /docs/catalog/support/catalog-ai-safety
[Search Your Asset]: /docs/catalog/navigate/search

---

## addTeamUsers

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="addTeamUsers" /></>}>

<OperationEndpoint kind="mutation" name="addTeamUsers" />

Add users to a Team

```graphql
addTeamUsers(
  data: TeamUsersInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="addTeamUsers" />

</CatalogGraphQLDocLayout>

---

## attachTags

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="attachTags" /></>}>

<OperationEndpoint kind="mutation" name="attachTags" />

Attach tags to entities by using their tag labels, and create a new tag if it does not already exist.

```graphql
attachTags(
  data: [BaseTagEntityInput!]!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="attachTags" />

</CatalogGraphQLDocLayout>

---

## createColumns

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createColumns" /></>}>

<OperationEndpoint kind="mutation" name="createColumns" />

Create many columns

```graphql
createColumns(
  data: [CreateColumnInput!]!
): [Column!]!
```

<OperationSchemaDetails kind="mutation" operation="createColumns" />

</CatalogGraphQLDocLayout>

---

## createDatabases

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createDatabases" /></>}>

<OperationEndpoint kind="mutation" name="createDatabases" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation will allow you to create batches of databases to administrate through this API.

Note that in order not to conflict with Catalog’s extraction system, you will only be able to create databases on warehouses created from this API.

You will also need to craft and provide custom external ids used as technical identifiers.

```graphql
createDatabases(
  data: [CreateDatabaseInput!]!
): [Database!]!
```

<OperationSchemaDetails kind="mutation" operation="createDatabases" />

</CatalogGraphQLDocLayout>

---

## createExternalLinks

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createExternalLinks" /></>}>

<OperationEndpoint kind="mutation" name="createExternalLinks" />

Create many External Links

```graphql
createExternalLinks(
  data: [CreateExternalLinkInput!]!
): [ExternalLink!]!
```

<OperationSchemaDetails kind="mutation" operation="createExternalLinks" />

</CatalogGraphQLDocLayout>

---

## createSchemas

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createSchemas" /></>}>

<OperationEndpoint kind="mutation" name="createSchemas" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation will allow you to create batches of schemas to administrate through this API.

Note that in order not to conflict with Catalog’s extraction system, you will only be able to create schemas on databases and warehouses created through this API.

You will also need to craft and provide custom external ids used as technical identifiers.

```graphql
createSchemas(
  data: [CreateSchemaInput!]!
): [Schema!]!
```

<OperationSchemaDetails kind="mutation" operation="createSchemas" />

</CatalogGraphQLDocLayout>

---

## createSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createSource" /></>}>

<OperationEndpoint kind="mutation" name="createSource" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation allows you to create a single custom warehouse source to administrate through this API.

This comes with some restrictions:

- The created source will have a few fields with enforced values:
  - `origin` set to `API`
  - `type` set to `WAREHOUSE`
  - `technology` set to `GENERIC_WAREHOUSE`
- You will only be allowed &#x002A;&#x002A;a single source&#x002A;&#x002A; from this API (use the `updateSource` mutation to edit or delete it)

```graphql
createSource(
  data: CreateSourceInput!
): Source!
```

<OperationSchemaDetails kind="mutation" operation="createSource" />

</CatalogGraphQLDocLayout>

---

## createTables

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createTables" /></>}>

<OperationEndpoint kind="mutation" name="createTables" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation will allow you to create batches of tables to administrate through this API.

Note that in order not to conflict with Catalog’s extraction system, you will only be able to create tables on schemas, databases and warehouses created through this API.

You will also need to craft and provide custom external ids used as technical identifiers.

```graphql
createTables(
  data: [CreateTableInput!]!
): [Table!]!
```

<OperationSchemaDetails kind="mutation" operation="createTables" />

</CatalogGraphQLDocLayout>

---

## createTerm

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="createTerm" /></>}>

<OperationEndpoint kind="mutation" name="createTerm" />

Create a Term either at root level (`parentTermId` is `null`) or as children of a given Term.

- `name` is mandatory
- `description` is mandatory and supports markdown
- `linkedTagId` will override any linked term on the given tag

```graphql
createTerm(
  data: CreateTermInput!
): Term!
```

<OperationSchemaDetails kind="mutation" operation="createTerm" />

</CatalogGraphQLDocLayout>

---

## deleteExternalLinks

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="deleteExternalLinks" /></>}>

<OperationEndpoint kind="mutation" name="deleteExternalLinks" />

Delete many External Links

```graphql
deleteExternalLinks(
  data: [DeleteExternalLinkInput!]!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="deleteExternalLinks" />

</CatalogGraphQLDocLayout>

---

## deleteLineages

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="deleteLineages" /></>}>

<OperationEndpoint kind="mutation" name="deleteLineages" />

Delete many Lineages

```graphql
deleteLineages(
  data: [DeleteLineageInput!]!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="deleteLineages" />

</CatalogGraphQLDocLayout>

---

## deleteTerm

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="deleteTerm" /></>}>

<OperationEndpoint kind="mutation" name="deleteTerm" />

Delete an existing Term, `id` of a valid Term is necessary. This operation cannot be reversed.

```graphql
deleteTerm(
  data: DeleteTermInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="deleteTerm" />

</CatalogGraphQLDocLayout>

---

## detachTags

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="detachTags" /></>}>

<OperationEndpoint kind="mutation" name="detachTags" />

Detach tags from entities by using tag labels, and delete the tag if there are no entities attached to it anymore.

```graphql
detachTags(
  data: [BaseTagEntityInput!]!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="detachTags" />

</CatalogGraphQLDocLayout>

---

## removeDataQualities

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="removeDataQualities" /></>}>

<OperationEndpoint kind="mutation" name="removeDataQualities" />

Delete many Quality Checks

```graphql
removeDataQualities(
  data: RemoveQualityChecksInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="removeDataQualities" />

</CatalogGraphQLDocLayout>

---

## removePinnedAssets

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="removePinnedAssets" /></>}>

<OperationEndpoint kind="mutation" name="removePinnedAssets" />

Delete many pinned assets

```graphql
removePinnedAssets(
  data: [EntitiesLinkInput!]!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="removePinnedAssets" />

</CatalogGraphQLDocLayout>

---

## removeTeamOwners

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="removeTeamOwners" /></>}>

<OperationEndpoint kind="mutation" name="removeTeamOwners" />

Remove team ownership to assets in Catalog

```graphql
removeTeamOwners(
  data: TeamOwnerInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="removeTeamOwners" />

</CatalogGraphQLDocLayout>

---

## removeTeamUsers

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="removeTeamUsers" /></>}>

<OperationEndpoint kind="mutation" name="removeTeamUsers" />

Remove users from a Team

```graphql
removeTeamUsers(
  data: TeamUsersInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="removeTeamUsers" />

</CatalogGraphQLDocLayout>

---

## removeUserOwners

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="removeUserOwners" /></>}>

<OperationEndpoint kind="mutation" name="removeUserOwners" />

Delete many User Owners

```graphql
removeUserOwners(
  data: OwnerInput!
): Boolean!
```

<OperationSchemaDetails kind="mutation" operation="removeUserOwners" />

</CatalogGraphQLDocLayout>

---

## updateColumnDescriptions

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateColumnDescriptions" /></>}>

<OperationEndpoint kind="mutation" name="updateColumnDescriptions" />

:::warning[DEPRECATED]

Use updateColumnsMetadata instead

:::

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

Update many column external descriptions

```graphql
updateColumnDescriptions(
  data: [UpdateColumnDescriptionInput!]!
): [Column!]! @deprecated
```

<OperationSchemaDetails kind="mutation" operation="updateColumnDescriptions" />

</CatalogGraphQLDocLayout>

---

## updateColumnsMetadata

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateColumnsMetadata" /></>}>

<OperationEndpoint kind="mutation" name="updateColumnsMetadata" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

Update many column metadata (description, external description, PII, primary key)

```graphql
updateColumnsMetadata(
  data: [UpdateColumnsMetadataInput!]!
): [Column!]!
```

<OperationSchemaDetails kind="mutation" operation="updateColumnsMetadata" />

</CatalogGraphQLDocLayout>

---

## updateColumns

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateColumns" /></>}>

<OperationEndpoint kind="mutation" name="updateColumns" />

Update many columns

```graphql
updateColumns(
  data: [UpdateColumnInput!]!
): [Column!]!
```

<OperationSchemaDetails kind="mutation" operation="updateColumns" />

</CatalogGraphQLDocLayout>

---

## updateDatabases

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateDatabases" /></>}>

<OperationEndpoint kind="mutation" name="updateDatabases" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation will allow you to batch update the databases you created.

Note that in order not to mess with data ingested by Catalog, only the databases belonging to sources created from the API will be editable.

```graphql
updateDatabases(
  data: [UpdateDatabaseInput!]!
): [Database!]!
```

<OperationSchemaDetails kind="mutation" operation="updateDatabases" />

</CatalogGraphQLDocLayout>

---

## updateExternalLinks

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateExternalLinks" /></>}>

<OperationEndpoint kind="mutation" name="updateExternalLinks" />

Update many External Links

```graphql
updateExternalLinks(
  data: [UpdateExternalLinkInput!]!
): [ExternalLink!]!
```

<OperationSchemaDetails kind="mutation" operation="updateExternalLinks" />

</CatalogGraphQLDocLayout>

---

## updateSchemas

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateSchemas" /></>}>

<OperationEndpoint kind="mutation" name="updateSchemas" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation will allow you to batch update the schemas you created.

Note that in order not to mess with data ingested by Catalog, only the schemas belonging to sources and databases created from this API will be editable.

```graphql
updateSchemas(
  data: [UpdateSchemaInput!]!
): [Schema!]!
```

<OperationSchemaDetails kind="mutation" operation="updateSchemas" />

</CatalogGraphQLDocLayout>

---

## updateSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateSource" /></>}>

<OperationEndpoint kind="mutation" name="updateSource" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

This mutation allows you to update the API source you created. 
Note that in order not to mess with data ingested by Catalog, only the source created from the API will be editable.

```graphql
updateSource(
  data: UpdateSourceInput!
): Source!
```

<OperationSchemaDetails kind="mutation" operation="updateSource" />

</CatalogGraphQLDocLayout>

---

## updateTableDescriptions

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateTableDescriptions" /></>}>

<OperationEndpoint kind="mutation" name="updateTableDescriptions" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

Update many tables external description

```graphql
updateTableDescriptions(
  data: [UpdateTableDescriptionInput!]!
): [Table!]!
```

<OperationSchemaDetails kind="mutation" operation="updateTableDescriptions" />

</CatalogGraphQLDocLayout>

---

## updateTables

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateTables" /></>}>

<OperationEndpoint kind="mutation" name="updateTables" />

:::warning[Restricted Access]

This endpoint access is restricted. Please reach out to your point of contact at Catalog if you are interested.
:::

Update many tables

```graphql
updateTables(
  data: [UpdateTableInput!]!
): [Table!]!
```

<OperationSchemaDetails kind="mutation" operation="updateTables" />

</CatalogGraphQLDocLayout>

---

## updateTerm

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="updateTerm" /></>}>

<OperationEndpoint kind="mutation" name="updateTerm" />

Update an existing Term, `id` of a valid Term is necessary, all other fields are optional.

- `description` supports markdown
- `linkedTagId` will override any linked term on the given tag

```graphql
updateTerm(
  data: UpdateTermInput!
): Term!
```

<OperationSchemaDetails kind="mutation" operation="updateTerm" />

</CatalogGraphQLDocLayout>

---

## upsertDataQualities

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertDataQualities" /></>}>

<OperationEndpoint kind="mutation" name="upsertDataQualities" />

With this mutation you can add or update multiple quality checks on a single table.

Quality checks are unique per `tableId` and `externalId` combination.

Thus, for a given `tableId` and `externalId` pair, we only keep the latest test run.

This means that for the provided checks, if the `runAt` value is set after the existing one, the new value will replace it. If however one test exists on those keys and has a `runAt` after the one you’re trying to insert, it will not be added.

```graphql
upsertDataQualities(
  data: UpsertQualityChecksInput!
): [QualityCheck!]!
```

<OperationSchemaDetails kind="mutation" operation="upsertDataQualities" />

</CatalogGraphQLDocLayout>

---

## upsertLineages

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertLineages" /></>}>

<OperationEndpoint kind="mutation" name="upsertLineages" />

Upsert many Lineages

```graphql
upsertLineages(
  data: [UpsertLineageInput!]!
): [Lineage!]!
```

<OperationSchemaDetails kind="mutation" operation="upsertLineages" />

</CatalogGraphQLDocLayout>

---

## upsertPinnedAssets

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertPinnedAssets" /></>}>

<OperationEndpoint kind="mutation" name="upsertPinnedAssets" />

Upsert many pinned assets

```graphql
upsertPinnedAssets(
  data: [EntitiesLinkInput!]!
): [EntitiesLink!]!
```

<OperationSchemaDetails kind="mutation" operation="upsertPinnedAssets" />

</CatalogGraphQLDocLayout>

---

## upsertTeamOwners

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertTeamOwners" /></>}>

<OperationEndpoint kind="mutation" name="upsertTeamOwners" />

Add team ownership to assets in Catalog

```graphql
upsertTeamOwners(
  data: TeamOwnerInput!
): [TeamOwnerEntity!]!
```

<OperationSchemaDetails kind="mutation" operation="upsertTeamOwners" />

</CatalogGraphQLDocLayout>

---

## upsertTeam

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertTeam" /></>}>

<OperationEndpoint kind="mutation" name="upsertTeam" />

Upsert a Team. If the name do not exist, then it will create a new team otherwise it will update the existing team.

```graphql
upsertTeam(
  data: UpsertTeamInput!
): Team!
```

<OperationSchemaDetails kind="mutation" operation="upsertTeam" />

</CatalogGraphQLDocLayout>

---

## upsertUserOwners

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="mutation" operation="upsertUserOwners" /></>}>

<OperationEndpoint kind="mutation" name="upsertUserOwners" />

Upsert many User Owners

```graphql
upsertUserOwners(
  data: OwnerInput!
): [OwnerEntity!]!
```

<OperationSchemaDetails kind="mutation" operation="upsertUserOwners" />

</CatalogGraphQLDocLayout>

---

## addAiAssistantJob

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="addAiAssistantJob" /></>}>

<OperationEndpoint kind="query" name="addAiAssistantJob" />

Starts an AI Assistant job and returns its `jobId`. Use the [`getAiAssistantJobResult`](./get-ai-assistant-job-result.mdx) query to poll for the job result.

```graphql
addAiAssistantJob(
  data: AddAiAssistantJobInput!
): AddAiAssistantJobOutput!
```

<OperationSchemaDetails kind="query" operation="addAiAssistantJob" />

</CatalogGraphQLDocLayout>

---

## getAiAssistantJobResult

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getAiAssistantJobResult" /></>}>

<OperationEndpoint kind="query" name="getAiAssistantJobResult" />

Retrieves the result of an AI Assistant job by its `jobId`. Use the [`addAiAssistantJob`](./add-ai-assistant-job.mdx) query to start a job.

```graphql
getAiAssistantJobResult(
  data: JobResultInput!
): GetAiAssistantJobResultOutput!
```

<OperationSchemaDetails kind="query" operation="getAiAssistantJobResult" />

</CatalogGraphQLDocLayout>

---

## getColumnJoins

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getColumnJoins" /></>}>

<OperationEndpoint kind="query" name="getColumnJoins" />

Get many column joins, with optional scoping, sorting and pagination

```graphql
getColumnJoins(
  pagination: PublicPagination
  sorting: [ColumnJoinSorting!]
  scope: GetColumnJoinsScope
): GetColumnJoinsOutput!
```

<OperationSchemaDetails kind="query" operation="getColumnJoins" />

</CatalogGraphQLDocLayout>

---

## getColumns

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getColumns" /></>}>

<OperationEndpoint kind="query" name="getColumns" />

Get many columns, with optional scoping, sorting and pagination

```graphql
getColumns(
  pagination: PublicPagination
  sorting: [ColumnSorting!]
  scope: GetColumnsScope
): GetColumnsOutput!
```

<OperationSchemaDetails kind="query" operation="getColumns" />

</CatalogGraphQLDocLayout>

---

## getDashboards

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getDashboards" /></>}>

<OperationEndpoint kind="query" name="getDashboards" />

Get many dashboards, with optional scoping, sorting and pagination

```graphql
getDashboards(
  pagination: PublicPagination
  sorting: [DashboardSorting!]
  scope: GetDashboardsScope
): GetDashboardsOutput!
```

<OperationSchemaDetails kind="query" operation="getDashboards" />

</CatalogGraphQLDocLayout>

---

## getDataProducts

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getDataProducts" /></>}>

<OperationEndpoint kind="query" name="getDataProducts" />

Get many Data Products, with optional scoping, sorting and pagination

```graphql
getDataProducts(
  pagination: PublicPagination
  sorting: [DataProductSorting!]
  scope: GetDataProductScope
): GetDataProductOutput!
```

<OperationSchemaDetails kind="query" operation="getDataProducts" />

</CatalogGraphQLDocLayout>

---

## getDataQualities

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getDataQualities" /></>}>

<OperationEndpoint kind="query" name="getDataQualities" />

Get many Quality Checks, with optional scoping, sorting and pagination

```graphql
getDataQualities(
  pagination: PublicPagination
  scope: GetQualityChecksScope
): GetQualityChecksOutput!
```

<OperationSchemaDetails kind="query" operation="getDataQualities" />

</CatalogGraphQLDocLayout>

---

## getDatabases

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getDatabases" /></>}>

<OperationEndpoint kind="query" name="getDatabases" />

Get many databases, with optional scoping, sorting and pagination

```graphql
getDatabases(
  pagination: PublicPagination
  scope: GetDatabasesScope
): GetDatabasesOutput!
```

<OperationSchemaDetails kind="query" operation="getDatabases" />

</CatalogGraphQLDocLayout>

---

## getFieldLineages

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getFieldLineages" /></>}>

<OperationEndpoint kind="query" name="getFieldLineages" />

Get many Field Lineages, with optional scoping, sorting and pagination

```graphql
getFieldLineages(
  pagination: PublicPagination
  sorting: [FieldLineageSorting!]
  scope: GetFieldLineagesScope!
): GetFieldLineagesOutput!
```

<OperationSchemaDetails kind="query" operation="getFieldLineages" />

</CatalogGraphQLDocLayout>

---

## getLineages

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getLineages" /></>}>

<OperationEndpoint kind="query" name="getLineages" />

Get many Lineages, with optional scoping, sorting and pagination

```graphql
getLineages(
  pagination: PublicPagination
  sorting: [LineageSorting!]
  scope: GetLineagesScope
): GetLineagesOutput!
```

<OperationSchemaDetails kind="query" operation="getLineages" />

</CatalogGraphQLDocLayout>

---

## getPinnedAssets

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getPinnedAssets" /></>}>

<OperationEndpoint kind="query" name="getPinnedAssets" />

Get many Pinned Assets, with optional scoping, sorting and pagination

```graphql
getPinnedAssets(
  pagination: PublicPagination
  sorting: [EntitiesLinkSorting!]
  scope: GetEntitiesLinksScope
): GetEntitiesLinkOutput!
```

<OperationSchemaDetails kind="query" operation="getPinnedAssets" />

</CatalogGraphQLDocLayout>

---

## getSchemas

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getSchemas" /></>}>

<OperationEndpoint kind="query" name="getSchemas" />

Get many schemas, with optional scoping, sorting and pagination

```graphql
getSchemas(
  pagination: PublicPagination
  scope: GetSchemasScope
): GetSchemasOutput!
```

<OperationSchemaDetails kind="query" operation="getSchemas" />

</CatalogGraphQLDocLayout>

---

## getSources

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getSources" /></>}>

<OperationEndpoint kind="query" name="getSources" />

Get many sources, with optional scoping, sorting and pagination

```graphql
getSources(
  pagination: PublicPagination
  scope: GetSourcesScope
): GetSourcesOutput!
```

<OperationSchemaDetails kind="query" operation="getSources" />

</CatalogGraphQLDocLayout>

---

## getTableQueries

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getTableQueries" /></>}>

<OperationEndpoint kind="query" name="getTableQueries" />

Get queries associated with tables with scoping

```graphql
getTableQueries(
  pagination: PublicPagination
  sorting: [QuerySorting!]
  scope: GetTableQueriesScope
): GetTableQueriesOutput!
```

<OperationSchemaDetails kind="query" operation="getTableQueries" />

</CatalogGraphQLDocLayout>

---

## getTables

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getTables" /></>}>

<OperationEndpoint kind="query" name="getTables" />

Get many tables, with optional scoping, sorting and pagination

```graphql
getTables(
  pagination: PublicPagination
  sorting: [TableSorting!]
  scope: GetTablesScope
): GetTablesOutput!
```

<OperationSchemaDetails kind="query" operation="getTables" />

</CatalogGraphQLDocLayout>

---

## getTags

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getTags" /></>}>

<OperationEndpoint kind="query" name="getTags" />

Get many tags, with optional scoping, sorting and pagination

All tags will be retrieved, including tags on soft deleted and hidden entities. This means tags on:

- soft deleted columns, tables, dashboards, terms and any other kind of assets
- hidden assets: databases, schemas, etc

```graphql
getTags(
  pagination: PublicPagination
  sorting: [TagSorting!]
  scope: GetTagsScope
): GetTagsOutput!
```

<OperationSchemaDetails kind="query" operation="getTags" />

</CatalogGraphQLDocLayout>

---

## getTeams

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getTeams" /></>}>

<OperationEndpoint kind="query" name="getTeams" />

Get many teams, with optional scoping, sorting and pagination

```graphql
getTeams(
  pagination: PublicPagination
): [GetTeamsOutput!]!
```

<OperationSchemaDetails kind="query" operation="getTeams" />

</CatalogGraphQLDocLayout>

---

## getTerms

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getTerms" /></>}>

<OperationEndpoint kind="query" name="getTerms" />

Get many Terms, with optional scoping, sorting and pagination

```graphql
getTerms(
  pagination: PublicPagination
  sorting: [TermSorting!]
  scope: GetTermsScope
): GetTermsOutput!
```

<OperationSchemaDetails kind="query" operation="getTerms" />

</CatalogGraphQLDocLayout>

---

## getUsers

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="getUsers" /></>}>

<OperationEndpoint kind="query" name="getUsers" />

Get many user, with optional scoping, sorting and pagination

```graphql
getUsers(
  pagination: PublicPagination
): [GetUsersOutput!]!
```

<OperationSchemaDetails kind="query" operation="getUsers" />

</CatalogGraphQLDocLayout>

---

## searchQueries

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

<CatalogGraphQLDocLayout tryIt={<><CatalogGraphQLTryIt kind="query" operation="searchQueries" /></>}>

<OperationEndpoint kind="query" name="searchQueries" />

Perform a semantic search on your ingested SQL queries, using natural language and return 10 relevant results

```graphql
searchQueries(
  scope: SearchQueriesScope
  data: SearchQueriesInput!
): SearchQueriesOutput!
```

<OperationSchemaDetails kind="query" operation="searchQueries" />

</CatalogGraphQLDocLayout>

---

## Colors

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Available colors for tag

```graphql
enum Colors {
  ALERT_COLOR
  AQUA_BLUE
  BLUE_RED
  CERULEAN_BLUE
  CHARTREUSE
  MAGENTA
  ORANGE_RED
  ORANGE_YELLOW
  RED_VIOLET
  SPRING_GREEN
  VIOLET
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>Colors.ALERT&#x005F;COLOR</code>](#alert-color) {/* #alert-color */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.AQUA&#x005F;BLUE</code>](#aqua-blue) {/* #aqua-blue */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.BLUE&#x005F;RED</code>](#blue-red) {/* #blue-red */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.CERULEAN&#x005F;BLUE</code>](#cerulean-blue) {/* #cerulean-blue */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.CHARTREUSE</code>](#chartreuse) {/* #chartreuse */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.MAGENTA</code>](#magenta) {/* #magenta */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.ORANGE&#x005F;RED</code>](#orange-red) {/* #orange-red */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.ORANGE&#x005F;YELLOW</code>](#orange-yellow) {/* #orange-yellow */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.RED&#x005F;VIOLET</code>](#red-violet) {/* #red-violet */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.SPRING&#x005F;GREEN</code>](#spring-green) {/* #spring-green */}
#### [<code style={{ fontWeight: 'normal' }}>Colors.VIOLET</code>](#violet) {/* #violet */}
### Member Of

- [`Tag`](/docs/catalog/api/general/types/objects/tag)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ColumnExternalDescriptionSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Column External Sources of documentation available

```graphql
enum ColumnExternalDescriptionSource {
  API
  COALESCE
  DBT
  DATABASE
  CASTOR
  LOOKER
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.API</code>](#api) {/* #api */}
Documentation from Catalog public API

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.COALESCE</code>](#coalesce) {/* #coalesce */}
Documentation from your Coalesce

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.DBT</code>](#dbt) {/* #dbt */}
Documentation from your dbt

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.DATABASE</code>](#database) {/* #database */}
Documentation from your warehouse

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.CASTOR</code>](#castor) {/* #castor */}
Documentation generated by Catalog

#### [<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource.LOOKER</code>](#looker) {/* #looker */}
Documentation from your Looker

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ColumnJoinSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum ColumnJoinSortingKey {
  count
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoinSortingKey.count</code>](#count) {/* #count */}
Sort by the number of join occurrences

### Member Of

- [`ColumnJoinSorting`](/docs/catalog/api/general/types/inputs/column-join-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## ColumnSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum ColumnSortingKey {
  name
  sourceOrder
  tableName
  tablePopularity
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>ColumnSortingKey.name</code>](#name) {/* #name */}
#### [<code style={{ fontWeight: 'normal' }}>ColumnSortingKey.sourceOrder</code>](#source-order) {/* #source-order */}
#### [<code style={{ fontWeight: 'normal' }}>ColumnSortingKey.tableName</code>](#table-name) {/* #table-name */}
#### [<code style={{ fontWeight: 'normal' }}>ColumnSortingKey.tablePopularity</code>](#table-popularity) {/* #table-popularity */}
### Member Of

- [`ColumnSorting`](/docs/catalog/api/general/types/inputs/column-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## DashboardExternalDescriptionSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Dashboard External Sources of documentation available

```graphql
enum DashboardExternalDescriptionSource {
  CASTOR
  VISUALIZATION
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>DashboardExternalDescriptionSource.CASTOR</code>](#castor) {/* #castor */}
Generated by Catalog

#### [<code style={{ fontWeight: 'normal' }}>DashboardExternalDescriptionSource.VISUALIZATION</code>](#visualization) {/* #visualization */}
From your visualization tool

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## DashboardSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum DashboardSortingKey {
  name
  ownersAndTeamOwnersCount
  popularity
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>DashboardSortingKey.name</code>](#name) {/* #name */}
Sort by the dashboard name

#### [<code style={{ fontWeight: 'normal' }}>DashboardSortingKey.ownersAndTeamOwnersCount</code>](#owners-and-team-owners-count) {/* #owners-and-team-owners-count */}
Sort by the number of owners and team owners

#### [<code style={{ fontWeight: 'normal' }}>DashboardSortingKey.popularity</code>](#popularity) {/* #popularity */}
Sort by the dashboard popularity

### Member Of

- [`DashboardSorting`](/docs/catalog/api/general/types/inputs/dashboard-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## DashboardType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The flavour of dashboard (e.g. tableau)

```graphql
enum DashboardType {
  DASHBOARD
  TILE
  VIZ_MODEL
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>DashboardType.DASHBOARD</code>](#dashboard) {/* #dashboard */}
Visualization composed of one or more tiles

#### [<code style={{ fontWeight: 'normal' }}>DashboardType.TILE</code>](#tile) {/* #tile */}
Atomic visualization component

#### [<code style={{ fontWeight: 'normal' }}>DashboardType.VIZ&#x005F;MODEL</code>](#viz-model) {/* #viz-model */}
Visualization modeling layer with fields

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## DataProductSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields for Data Products

```graphql
enum DataProductSortingKey {
  tableName
  dashboardName
  termName
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>DataProductSortingKey.tableName</code>](#table-name) {/* #table-name */}
Sort by the table name

#### [<code style={{ fontWeight: 'normal' }}>DataProductSortingKey.dashboardName</code>](#dashboard-name) {/* #dashboard-name */}
Sort by the dashboard name

#### [<code style={{ fontWeight: 'normal' }}>DataProductSortingKey.termName</code>](#term-name) {/* #term-name */}
Sort by the term name

### Member Of

- [`DataProductSorting`](/docs/catalog/api/general/types/inputs/data-product-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## EntitiesLinkSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Entities links possible sort fields

```graphql
enum EntitiesLinkSortingKey {
  createdAt
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkSortingKey.createdAt</code>](#created-at) {/* #created-at */}
Sort by the entities link creation date

### Member Of

- [`EntitiesLinkSorting`](/docs/catalog/api/general/types/inputs/entities-link-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## EntitiesLinkTargetType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The type of entity as the target for an EntityLink

```graphql
enum EntitiesLinkTargetType {
  COLUMN
  DASHBOARD
  DASHBOARD_FIELD
  TABLE
  TERM
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType.COLUMN</code>](#column) {/* #column */}
Column entity

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType.DASHBOARD</code>](#dashboard) {/* #dashboard */}
Dashboard entity

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType.DASHBOARD&#x005F;FIELD</code>](#dashboard-field) {/* #dashboard-field */}
#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType.TABLE</code>](#table) {/* #table */}
Table entity

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType.TERM</code>](#term) {/* #term */}
Knowledge entity

### Member Of

- [`EntitiesLinkTargetInput`](/docs/catalog/api/general/types/inputs/entities-link-target-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## EntityTargetType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The scope of the action

```graphql
enum EntityTargetType {
  DASHBOARD
  TABLE
  TERM
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>EntityTargetType.DASHBOARD</code>](#dashboard) {/* #dashboard */}
Dashboard entity

#### [<code style={{ fontWeight: 'normal' }}>EntityTargetType.TABLE</code>](#table) {/* #table */}
Table entity

#### [<code style={{ fontWeight: 'normal' }}>EntityTargetType.TERM</code>](#term) {/* #term */}
Knowledge entity

### Member Of

- [`EntityTarget`](/docs/catalog/api/general/types/inputs/entity-target)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDataProductScope`](/docs/catalog/api/general/types/inputs/get-data-product-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## ExternalLinkTechnology

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible external link technologies

```graphql
enum ExternalLinkTechnology {
  AIRFLOW
  GITHUB
  GITLAB
  OTHER
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology.AIRFLOW</code>](#airflow) {/* #airflow */}
Apache Airflow workflow management platform

#### [<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology.GITHUB</code>](#github) {/* #github */}
GitHub version control and collaboration platform

#### [<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology.GITLAB</code>](#gitlab) {/* #gitlab */}
GitLab version control and DevOps platform

#### [<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology.OTHER</code>](#other) {/* #other */}
Other external link type

### Member Of

- [`CreateExternalLinkInput`](/docs/catalog/api/general/types/inputs/create-external-link-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`ExternalLink`](/docs/catalog/api/general/types/objects/external-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## FieldLineageSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields for field lineages

```graphql
enum FieldLineageSortingKey {
  childDashboardPopularity
  id
  popularity
  type
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSortingKey.childDashboardPopularity</code>](#child-dashboard-popularity) {/* #child-dashboard-popularity */}
Sort by the child dashboard popularity

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSortingKey.id</code>](#id) {/* #id */}
Sort by the field lineage id

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSortingKey.popularity</code>](#popularity) {/* #popularity */}
Sort by the field lineage popularity

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSortingKey.type</code>](#type) {/* #type */}
Sort by the field lineage type

### Member Of

- [`FieldLineageSorting`](/docs/catalog/api/general/types/inputs/field-lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## FilterTablesMode

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

How filter should capture provided table IDs

```graphql
enum FilterTablesMode {
  ANY
  ALL
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>FilterTablesMode.ANY</code>](#any) {/* #any */}
At least one provided table IDs must be used by retrieved queries

#### [<code style={{ fontWeight: 'normal' }}>FilterTablesMode.ALL</code>](#all) {/* #all */}
All provided table IDs must be used by retrieved queries

### Member Of

- [`SearchQueriesScope`](/docs/catalog/api/general/types/inputs/search-queries-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## JobStatus

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The status of the worker job

```graphql
enum JobStatus {
  ADDED
  ACTIVE
  COMPLETED
  FAILED
  RETRIES_EXHAUSTED
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>JobStatus.ADDED</code>](#added) {/* #added */}
Job has been added to the queue

#### [<code style={{ fontWeight: 'normal' }}>JobStatus.ACTIVE</code>](#active) {/* #active */}
Job is currently being processed

#### [<code style={{ fontWeight: 'normal' }}>JobStatus.COMPLETED</code>](#completed) {/* #completed */}
Job has finished successfully

#### [<code style={{ fontWeight: 'normal' }}>JobStatus.FAILED</code>](#failed) {/* #failed */}
Job has failed during execution

#### [<code style={{ fontWeight: 'normal' }}>JobStatus.RETRIES&#x005F;EXHAUSTED</code>](#retries-exhausted) {/* #retries-exhausted */}
Job has failed after all retry attempts

### Member Of

- [`AiAssistantJobResult`](/docs/catalog/api/general/types/objects/ai-assistant-job-result)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## LineageAssetType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The asset type for the lineage

```graphql
enum LineageAssetType {
  COLUMN
  DASHBOARD
  DASHBOARD_FIELD
  TABLE
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>LineageAssetType.COLUMN</code>](#column) {/* #column */}
#### [<code style={{ fontWeight: 'normal' }}>LineageAssetType.DASHBOARD</code>](#dashboard) {/* #dashboard */}
#### [<code style={{ fontWeight: 'normal' }}>LineageAssetType.DASHBOARD&#x005F;FIELD</code>](#dashboard-field) {/* #dashboard-field */}
#### [<code style={{ fontWeight: 'normal' }}>LineageAssetType.TABLE</code>](#table) {/* #table */}
### Member Of

- [`GetFieldLineagesScope`](/docs/catalog/api/general/types/inputs/get-field-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetLineagesScope`](/docs/catalog/api/general/types/inputs/get-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## LineageSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields for lineages

```graphql
enum LineageSortingKey {
  id
  popularity
  type
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>LineageSortingKey.id</code>](#id) {/* #id */}
Sort by the lineage id

#### [<code style={{ fontWeight: 'normal' }}>LineageSortingKey.popularity</code>](#popularity) {/* #popularity */}
Sort by the lineage popularity (based on the parent and child assets popularity)

#### [<code style={{ fontWeight: 'normal' }}>LineageSortingKey.type</code>](#type) {/* #type */}
Sort by the lineage type

### Member Of

- [`LineageSorting`](/docs/catalog/api/general/types/inputs/lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## LineageType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The type of the lineage

```graphql
enum LineageType {
  AUTOMATIC
  MANUAL_CUSTOMER
  MANUAL_OPS
  OTHER_TECHNOS
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>LineageType.AUTOMATIC</code>](#automatic) {/* #automatic */}
Detected automatically by Catalog

#### [<code style={{ fontWeight: 'normal' }}>LineageType.MANUAL&#x005F;CUSTOMER</code>](#manual-customer) {/* #manual-customer */}
Created via the public API

#### [<code style={{ fontWeight: 'normal' }}>LineageType.MANUAL&#x005F;OPS</code>](#manual-ops) {/* #manual-ops */}
Created by the Catalog operations team

#### [<code style={{ fontWeight: 'normal' }}>LineageType.OTHER&#x005F;TECHNOS</code>](#other-technos) {/* #other-technos */}
Imported from other technologies

### Member Of

- [`FieldLineage`](/docs/catalog/api/general/types/objects/field-lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetFieldLineagesScope`](/docs/catalog/api/general/types/inputs/get-field-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetLineagesScope`](/docs/catalog/api/general/types/inputs/get-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Lineage`](/docs/catalog/api/general/types/objects/lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Origin

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The origin of the job

```graphql
enum Origin {
  API
  APP
  DUST
  EXTENSION
  GOOGLE_CHAT_BOT
  MS_TEAMS
  SLACK_BOT
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>Origin.API</code>](#api) {/* #api */}
Job originated from the API

#### [<code style={{ fontWeight: 'normal' }}>Origin.APP</code>](#app) {/* #app */}
Job originated from the main application

#### [<code style={{ fontWeight: 'normal' }}>Origin.DUST</code>](#dust) {/* #dust */}
Job originated from the Dust agent

#### [<code style={{ fontWeight: 'normal' }}>Origin.EXTENSION</code>](#extension) {/* #extension */}
Job originated from the chrome extension

#### [<code style={{ fontWeight: 'normal' }}>Origin.GOOGLE&#x005F;CHAT&#x005F;BOT</code>](#google-chat-bot) {/* #google-chat-bot */}
Job originated from the Google Chat app

#### [<code style={{ fontWeight: 'normal' }}>Origin.MS&#x005F;TEAMS</code>](#ms-teams) {/* #ms-teams */}
Job originated from the Microsoft Teams extension

#### [<code style={{ fontWeight: 'normal' }}>Origin.SLACK&#x005F;BOT</code>](#slack-bot) {/* #slack-bot */}
Job originated from the Slack app

### Member Of

- [`AddAiAssistantJobInput`](/docs/catalog/api/general/types/inputs/add-ai-assistant-job-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## QualityStatus

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The status of quality check

```graphql
enum QualityStatus {
  ALERT
  WARNING
  SUCCESS
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>QualityStatus.ALERT</code>](#alert) {/* #alert */}
Critical quality issue detected

#### [<code style={{ fontWeight: 'normal' }}>QualityStatus.WARNING</code>](#warning) {/* #warning */}
Non-critical quality issue detected

#### [<code style={{ fontWeight: 'normal' }}>QualityStatus.SUCCESS</code>](#success) {/* #success */}
Quality check passed successfully

### Member Of

- [`BaseQualityCheckInput`](/docs/catalog/api/general/types/inputs/base-quality-check-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## QuerySortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields for queries

```graphql
enum QuerySortingKey {
  hash
  queryType
  timestamp
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>QuerySortingKey.hash</code>](#hash) {/* #hash */}
Sort by the query hash

#### [<code style={{ fontWeight: 'normal' }}>QuerySortingKey.queryType</code>](#query-type) {/* #query-type */}
Sort by the query type (SELECT or WRITE)

#### [<code style={{ fontWeight: 'normal' }}>QuerySortingKey.timestamp</code>](#timestamp) {/* #timestamp */}
Sort by the query timestamp

### Member Of

- [`QuerySorting`](/docs/catalog/api/general/types/inputs/query-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## QueryType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The type of the query action

```graphql
enum QueryType {
  SELECT
  WRITE
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>QueryType.SELECT</code>](#select) {/* #select */}
Read-only query that retrieves data

#### [<code style={{ fontWeight: 'normal' }}>QueryType.WRITE</code>](#write) {/* #write */}
Query that modifies or writes data

### Member Of

- [`GetTableQueriesScope`](/docs/catalog/api/general/types/inputs/get-table-queries-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SearchArrayFilterMode

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

How search array filter should capture provided array values

```graphql
enum SearchArrayFilterMode {
  ALL
  ANY
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SearchArrayFilterMode.ALL</code>](#all) {/* #all */}
All values of the provided array must be present in the target array

#### [<code style={{ fontWeight: 'normal' }}>SearchArrayFilterMode.ANY</code>](#any) {/* #any */}
At least one value of the provided array must be present in the target array

### Member Of

- [`GetTableQueriesScope`](/docs/catalog/api/general/types/inputs/get-table-queries-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## SortingDirectionEnum

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort directions : ASC/DESC, default: asc

```graphql
enum SortingDirectionEnum {
  ASC
  DESC
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum.ASC</code>](#asc) {/* #asc */}
Sort in ascending order

#### [<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum.DESC</code>](#desc) {/* #desc */}
Sort in descending order

### Member Of

- [`ColumnJoinSorting`](/docs/catalog/api/general/types/inputs/column-join-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`ColumnSorting`](/docs/catalog/api/general/types/inputs/column-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DashboardSorting`](/docs/catalog/api/general/types/inputs/dashboard-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DataProductSorting`](/docs/catalog/api/general/types/inputs/data-product-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`EntitiesLinkSorting`](/docs/catalog/api/general/types/inputs/entities-link-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`FieldLineageSorting`](/docs/catalog/api/general/types/inputs/field-lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`LineageSorting`](/docs/catalog/api/general/types/inputs/lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`QuerySorting`](/docs/catalog/api/general/types/inputs/query-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TableSorting`](/docs/catalog/api/general/types/inputs/table-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TagSorting`](/docs/catalog/api/general/types/inputs/tag-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TermSorting`](/docs/catalog/api/general/types/inputs/term-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## SortingNullsPriority

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The position of the nulls first/last, default: last

```graphql
enum SortingNullsPriority {
  FIRST
  LAST
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SortingNullsPriority.FIRST</code>](#first) {/* #first */}
Place null values at the beginning of sorted results

#### [<code style={{ fontWeight: 'normal' }}>SortingNullsPriority.LAST</code>](#last) {/* #last */}
Place null values at the end of sorted results

### Member Of

- [`ColumnJoinSorting`](/docs/catalog/api/general/types/inputs/column-join-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`ColumnSorting`](/docs/catalog/api/general/types/inputs/column-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DashboardSorting`](/docs/catalog/api/general/types/inputs/dashboard-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DataProductSorting`](/docs/catalog/api/general/types/inputs/data-product-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`EntitiesLinkSorting`](/docs/catalog/api/general/types/inputs/entities-link-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`FieldLineageSorting`](/docs/catalog/api/general/types/inputs/field-lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`LineageSorting`](/docs/catalog/api/general/types/inputs/lineage-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`QuerySorting`](/docs/catalog/api/general/types/inputs/query-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TableSorting`](/docs/catalog/api/general/types/inputs/table-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TagSorting`](/docs/catalog/api/general/types/inputs/tag-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TermSorting`](/docs/catalog/api/general/types/inputs/term-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## SourceOrigin

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The source origin

```graphql
enum SourceOrigin {
  API
  EXTRACTION
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SourceOrigin.API</code>](#api) {/* #api */}
Data pushed via the public API

#### [<code style={{ fontWeight: 'normal' }}>SourceOrigin.EXTRACTION</code>](#extraction) {/* #extraction */}
Data originating from a customer data source

### Member Of

- [`GetSourcesScope`](/docs/catalog/api/general/types/inputs/get-sources-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SourceTechnology

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

No description

```graphql
enum SourceTechnology {
  GOOGLE_CHAT
  MS_TEAMS
  SLACK
  CONFLUENCE
  NOTION
  ANOMALO
  COALESCE_QUALITY
  DBT_TEST
  GREAT_EXPECTATIONS
  MONTE_CARLO
  SIFFLET
  SODA
  COALESCE
  DBT
  COGNOS
  DOMO
  GENERIC_VISUALIZATION
  LOOKER
  LOOKER_STUDIO
  METABASE
  MIXPANEL
  MODE
  OMNI
  PERISCOPE
  POWERBI
  QLIK_SENSE
  REDASH
  SALESFORCE_REPORTING
  SIGMA
  SISENSE
  SSRS
  STRATEGY
  SUPERSET
  TABLEAU
  THOUGHTSPOT
  ZOHO
  AMAZON_ATHENA
  BIGQUERY
  DATABRICKS
  DELTALAKE
  DOMO_DATA
  DREMIO
  DRUID
  DYNAMODB
  EXASOL
  FIREBOLT
  FIVETRAN
  GENERIC_WAREHOUSE
  GLUE
  HIVE
  KAFKA
  KEBOOLA
  MARIADB
  MYSQL
  ORACLE
  POSTGRES
  PRESTODB
  REDSHIFT
  SALESFORCE
  SNAPLOGIC
  SNOWFLAKE
  SQLSERVER
  SYNAPSE
  S3
  TERADATA
  TRINO
  VERTICA
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.GOOGLE&#x005F;CHAT</code>](#google-chat) {/* #google-chat */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MS&#x005F;TEAMS</code>](#ms-teams) {/* #ms-teams */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SLACK</code>](#slack) {/* #slack */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.CONFLUENCE</code>](#confluence) {/* #confluence */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.NOTION</code>](#notion) {/* #notion */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.ANOMALO</code>](#anomalo) {/* #anomalo */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.COALESCE&#x005F;QUALITY</code>](#coalesce-quality) {/* #coalesce-quality */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DBT&#x005F;TEST</code>](#dbt-test) {/* #dbt-test */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.GREAT&#x005F;EXPECTATIONS</code>](#great-expectations) {/* #great-expectations */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MONTE&#x005F;CARLO</code>](#monte-carlo) {/* #monte-carlo */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SIFFLET</code>](#sifflet) {/* #sifflet */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SODA</code>](#soda) {/* #soda */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.COALESCE</code>](#coalesce) {/* #coalesce */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DBT</code>](#dbt) {/* #dbt */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.COGNOS</code>](#cognos) {/* #cognos */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DOMO</code>](#domo) {/* #domo */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.GENERIC&#x005F;VISUALIZATION</code>](#generic-visualization) {/* #generic-visualization */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.LOOKER</code>](#looker) {/* #looker */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.LOOKER&#x005F;STUDIO</code>](#looker-studio) {/* #looker-studio */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.METABASE</code>](#metabase) {/* #metabase */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MIXPANEL</code>](#mixpanel) {/* #mixpanel */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MODE</code>](#mode) {/* #mode */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.OMNI</code>](#omni) {/* #omni */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.PERISCOPE</code>](#periscope) {/* #periscope */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.POWERBI</code>](#powerbi) {/* #powerbi */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.QLIK&#x005F;SENSE</code>](#qlik-sense) {/* #qlik-sense */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.REDASH</code>](#redash) {/* #redash */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SALESFORCE&#x005F;REPORTING</code>](#salesforce-reporting) {/* #salesforce-reporting */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SIGMA</code>](#sigma) {/* #sigma */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SISENSE</code>](#sisense) {/* #sisense */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SSRS</code>](#ssrs) {/* #ssrs */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.STRATEGY</code>](#strategy) {/* #strategy */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SUPERSET</code>](#superset) {/* #superset */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.TABLEAU</code>](#tableau) {/* #tableau */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.THOUGHTSPOT</code>](#thoughtspot) {/* #thoughtspot */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.ZOHO</code>](#zoho) {/* #zoho */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.AMAZON&#x005F;ATHENA</code>](#amazon-athena) {/* #amazon-athena */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.BIGQUERY</code>](#bigquery) {/* #bigquery */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DATABRICKS</code>](#databricks) {/* #databricks */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DELTALAKE</code>](#deltalake) {/* #deltalake */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DOMO&#x005F;DATA</code>](#domo-data) {/* #domo-data */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DREMIO</code>](#dremio) {/* #dremio */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DRUID</code>](#druid) {/* #druid */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.DYNAMODB</code>](#dynamodb) {/* #dynamodb */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.EXASOL</code>](#exasol) {/* #exasol */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.FIREBOLT</code>](#firebolt) {/* #firebolt */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.FIVETRAN</code>](#fivetran) {/* #fivetran */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.GENERIC&#x005F;WAREHOUSE</code>](#generic-warehouse) {/* #generic-warehouse */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.GLUE</code>](#glue) {/* #glue */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.HIVE</code>](#hive) {/* #hive */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.KAFKA</code>](#kafka) {/* #kafka */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.KEBOOLA</code>](#keboola) {/* #keboola */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MARIADB</code>](#mariadb) {/* #mariadb */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.MYSQL</code>](#mysql) {/* #mysql */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.ORACLE</code>](#oracle) {/* #oracle */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.POSTGRES</code>](#postgres) {/* #postgres */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.PRESTODB</code>](#prestodb) {/* #prestodb */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.REDSHIFT</code>](#redshift) {/* #redshift */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SALESFORCE</code>](#salesforce) {/* #salesforce */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SNAPLOGIC</code>](#snaplogic) {/* #snaplogic */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SNOWFLAKE</code>](#snowflake) {/* #snowflake */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SQLSERVER</code>](#sqlserver) {/* #sqlserver */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.SYNAPSE</code>](#synapse) {/* #synapse */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.S3</code>](#s3) {/* #s3 */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.TERADATA</code>](#teradata) {/* #teradata */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.TRINO</code>](#trino) {/* #trino */}
#### [<code style={{ fontWeight: 'normal' }}>SourceTechnology.VERTICA</code>](#vertica) {/* #vertica */}
### Member Of

- [`GetSourcesScope`](/docs/catalog/api/general/types/inputs/get-sources-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SourceType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

type of source

```graphql
enum SourceType {
  COMMUNICATION
  KNOWLEDGE
  QUALITY
  TRANSFORMATION
  VISUALIZATION
  WAREHOUSE
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>SourceType.COMMUNICATION</code>](#communication) {/* #communication */}
Source for communication and messaging systems

#### [<code style={{ fontWeight: 'normal' }}>SourceType.KNOWLEDGE</code>](#knowledge) {/* #knowledge */}
Source for knowledge management and documentation

#### [<code style={{ fontWeight: 'normal' }}>SourceType.QUALITY</code>](#quality) {/* #quality */}
Source for data quality and validation

#### [<code style={{ fontWeight: 'normal' }}>SourceType.TRANSFORMATION</code>](#transformation) {/* #transformation */}
Source for data transformation and processing

#### [<code style={{ fontWeight: 'normal' }}>SourceType.VISUALIZATION</code>](#visualization) {/* #visualization */}
Source for data visualization and reporting

#### [<code style={{ fontWeight: 'normal' }}>SourceType.WAREHOUSE</code>](#warehouse) {/* #warehouse */}
Source for data warehouse and storage

### Member Of

- [`GetSourcesScope`](/docs/catalog/api/general/types/inputs/get-sources-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TableExternalDescriptionSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Table External Sources of documentation available

```graphql
enum TableExternalDescriptionSource {
  API
  COALESCE
  DBT
  DATABASE
  CASTOR
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource.API</code>](#api) {/* #api */}
Description pushed via the public API

#### [<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource.COALESCE</code>](#coalesce) {/* #coalesce */}
Description originating from coalesce

#### [<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource.DBT</code>](#dbt) {/* #dbt */}
Description originating from dbt

#### [<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource.DATABASE</code>](#database) {/* #database */}
Description originating from a customer warehouse

#### [<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource.CASTOR</code>](#castor) {/* #castor */}
Description generated by Catalog

### Member Of

- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TableSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum TableSortingKey {
  levelOfCompletion
  name
  nameLength
  ownersAndTeamOwnersCount
  popularity
  schemaName
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.levelOfCompletion</code>](#level-of-completion) {/* #level-of-completion */}
Sort by the level of completion

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.name</code>](#name) {/* #name */}
Sort by the table name

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.nameLength</code>](#name-length) {/* #name-length */}
Sort by the table name length

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.ownersAndTeamOwnersCount</code>](#owners-and-team-owners-count) {/* #owners-and-team-owners-count */}
Sort by the number of owners and team owners

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.popularity</code>](#popularity) {/* #popularity */}
Sort by the table popularity

#### [<code style={{ fontWeight: 'normal' }}>TableSortingKey.schemaName</code>](#schema-name) {/* #schema-name */}
Sort by the schema name

### Member Of

- [`TableSorting`](/docs/catalog/api/general/types/inputs/table-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## TableType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible types for a table

```graphql
enum TableType {
  DYNAMIC_TABLE
  EXTERNAL
  TABLE
  TOPIC
  VIEW
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TableType.DYNAMIC&#x005F;TABLE</code>](#dynamic-table) {/* #dynamic-table */}
A Snowflake specific table type, is dynamically updated depending on some criteria

#### [<code style={{ fontWeight: 'normal' }}>TableType.EXTERNAL</code>](#external) {/* #external */}
Table from an external source

#### [<code style={{ fontWeight: 'normal' }}>TableType.TABLE</code>](#table) {/* #table */}
Standard table

#### [<code style={{ fontWeight: 'normal' }}>TableType.TOPIC</code>](#topic) {/* #topic */}
Topic or subject-specific table

#### [<code style={{ fontWeight: 'normal' }}>TableType.VIEW</code>](#view) {/* #view */}
Virtual table defined by a query

### Member Of

- [`CreateTableInput`](/docs/catalog/api/general/types/inputs/create-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UpdateTableInput`](/docs/catalog/api/general/types/inputs/update-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## TagEntityOrigin

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible author of a tag entity

```graphql
enum TagEntityOrigin {
  EXTERNAL
  INTERNAL
  USER
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TagEntityOrigin.EXTERNAL</code>](#external) {/* #external */}
Tag originating for a customer source

#### [<code style={{ fontWeight: 'normal' }}>TagEntityOrigin.INTERNAL</code>](#internal) {/* #internal */}
Tag created by Catalog

#### [<code style={{ fontWeight: 'normal' }}>TagEntityOrigin.USER</code>](#user) {/* #user */}
Tag created by a user

### Member Of

- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TagEntityType

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Tag's target entity type

```graphql
enum TagEntityType {
  COLUMN
  DASHBOARD
  DASHBOARD_FIELD
  TABLE
  TERM
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TagEntityType.COLUMN</code>](#column) {/* #column */}
The column entity

#### [<code style={{ fontWeight: 'normal' }}>TagEntityType.DASHBOARD</code>](#dashboard) {/* #dashboard */}
The dashboard entity

#### [<code style={{ fontWeight: 'normal' }}>TagEntityType.DASHBOARD&#x005F;FIELD</code>](#dashboard-field) {/* #dashboard-field */}
The dashboard field entity

#### [<code style={{ fontWeight: 'normal' }}>TagEntityType.TABLE</code>](#table) {/* #table */}
The table entity

#### [<code style={{ fontWeight: 'normal' }}>TagEntityType.TERM</code>](#term) {/* #term */}
The knowledge entity

### Member Of

- [`BaseTagEntityInput`](/docs/catalog/api/general/types/inputs/base-tag-entity-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## TagSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum TagSortingKey {
  label
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TagSortingKey.label</code>](#label) {/* #label */}
Sort by the tag label

### Member Of

- [`TagSorting`](/docs/catalog/api/general/types/inputs/tag-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## TermExternalDescriptionSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Term available external documentation sources

```graphql
enum TermExternalDescriptionSource {
  CASTOR
  KNOWLEDGE
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TermExternalDescriptionSource.CASTOR</code>](#castor) {/* #castor */}
Description generated by Catalog

#### [<code style={{ fontWeight: 'normal' }}>TermExternalDescriptionSource.KNOWLEDGE</code>](#knowledge) {/* #knowledge */}
Description originating from a customer knowledge base

### Member Of

- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TermSortingKey

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The possible sort fields

```graphql
enum TermSortingKey {
  name
  ownersAndTeamOwnersCount
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TermSortingKey.name</code>](#name) {/* #name */}
Sort by the term name

#### [<code style={{ fontWeight: 'normal' }}>TermSortingKey.ownersAndTeamOwnersCount</code>](#owners-and-team-owners-count) {/* #owners-and-team-owners-count */}
Sort by the number of owners and team owners

### Member Of

- [`TermSorting`](/docs/catalog/api/general/types/inputs/term-sorting)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## TransformationSource

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Transformation source technologies availables

```graphql
enum TransformationSource {
  COALESCE
  DBT
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>TransformationSource.COALESCE</code>](#coalesce) {/* #coalesce */}
Description originating from coalesce

#### [<code style={{ fontWeight: 'normal' }}>TransformationSource.DBT</code>](#dbt) {/* #dbt */}
Description originating from dbt

### Member Of

- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## UserRole

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The role of a Catalog user restricting access to Catalog features.

```graphql
enum UserRole {
  ADMIN
  CONTRIBUTOR
  STEWARD
  VIEWER
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>UserRole.ADMIN</code>](#admin) {/* #admin */}
Full administrative access with all privileges

#### [<code style={{ fontWeight: 'normal' }}>UserRole.CONTRIBUTOR</code>](#contributor) {/* #contributor */}
Can view and contribute to content

#### [<code style={{ fontWeight: 'normal' }}>UserRole.STEWARD</code>](#steward) {/* #steward */}
Partial administrative access

#### [<code style={{ fontWeight: 'normal' }}>UserRole.VIEWER</code>](#viewer) {/* #viewer */}
Read-only access to content

### Member Of

- [`GetUsersOutput`](/docs/catalog/api/general/types/objects/get-users-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UnifiedUser`](/docs/catalog/api/general/types/objects/unified-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## UserStatus

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The current status of the user (e.g. pending, active)

```graphql
enum UserStatus {
  ACTIVATED
  PENDING
  SUSPENDED
}
```

### Values

#### [<code style={{ fontWeight: 'normal' }}>UserStatus.ACTIVATED</code>](#activated) {/* #activated */}
#### [<code style={{ fontWeight: 'normal' }}>UserStatus.PENDING</code>](#pending) {/* #pending */}
#### [<code style={{ fontWeight: 'normal' }}>UserStatus.SUSPENDED</code>](#suspended) {/* #suspended */}
### Member Of

- [`GetUsersOutput`](/docs/catalog/api/general/types/objects/get-users-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UnifiedUser`](/docs/catalog/api/general/types/objects/unified-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## AddAiAssistantJobInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input AddAiAssistantJobInput {
  email: String!
  externalConversationId: String!
  origin: Origin
  question: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>AddAiAssistantJobInput.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The user's email. A Catalog account is required to interact with the AI Assistant, if the user doesn't have one, an error will be raised

#### [<code style={{ fontWeight: 'normal' }}>AddAiAssistantJobInput.externalConversationId</code>](#external-conversation-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-conversation-id */}
The external identifier acts as a unique key that links a conversation across multiple messages.       This enables the AI Assistant to maintain continuity and reuse context, allowing it to deliver       consistent and relevant responses throughout the conversation.

#### [<code style={{ fontWeight: 'normal' }}>AddAiAssistantJobInput.origin</code>](#origin)<Bullet />[<code style={{ fontWeight: 'normal' }}>Origin</code>](/docs/catalog/api/general/types/enums/origin) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #origin */}
Origin of the job (only API or DUST allowed)

#### [<code style={{ fontWeight: 'normal' }}>AddAiAssistantJobInput.question</code>](#question)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #question */}
Question that will be used to find related assets. Must be less than 10000 characters

### Example

<TypeExample />

### Member Of

- [`addAiAssistantJob`](/docs/catalog/api/general/operations/queries/add-ai-assistant-job)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## BaseQualityCheckInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input BaseQualityCheckInput {
  columnId: String
  description: String
  externalId: String!
  name: String!
  runAt: DateTime!
  status: QualityStatus!
  url: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.columnId</code>](#column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #column-id */}
Column id linked to the quality check.

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
Quality check description

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical quality check identifier

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Quality check name

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.runAt</code>](#run-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime!</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #run-at */}
Time at which the quality check ran

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>QualityStatus!</code>](/docs/catalog/api/general/types/enums/quality-status) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
Status of the quality check

#### [<code style={{ fontWeight: 'normal' }}>BaseQualityCheckInput.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
Url of the quality check

### Example

<TypeExample />

### Member Of

- [`UpsertQualityChecksInput`](/docs/catalog/api/general/types/inputs/upsert-quality-checks-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## BaseTagEntityInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input BaseTagEntityInput {
  entityType: TagEntityType!
  entityId: String!
  label: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>BaseTagEntityInput.entityType</code>](#entity-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>TagEntityType!</code>](/docs/catalog/api/general/types/enums/tag-entity-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-type */}
The type of the tagged entity

#### [<code style={{ fontWeight: 'normal' }}>BaseTagEntityInput.entityId</code>](#entity-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-id */}
The id of the tagged entity

#### [<code style={{ fontWeight: 'normal' }}>BaseTagEntityInput.label</code>](#label)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #label */}
The label of the tag

### Example

<TypeExample />

### Member Of

- [`attachTags`](/docs/catalog/api/general/operations/mutations/attach-tags)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`detachTags`](/docs/catalog/api/general/operations/mutations/detach-tags)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## ColumnJoinSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input ColumnJoinSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: ColumnJoinSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoinSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoinSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoinSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>ColumnJoinSortingKey!</code>](/docs/catalog/api/general/types/enums/column-join-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getColumnJoins`](/docs/catalog/api/general/operations/queries/get-column-joins)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## ColumnSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input ColumnSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: ColumnSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>ColumnSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>ColumnSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>ColumnSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>ColumnSortingKey!</code>](/docs/catalog/api/general/types/enums/column-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getColumns`](/docs/catalog/api/general/operations/queries/get-columns)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## CreateColumnInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateColumnInput {
  deletedAt: DateTime
  externalDescription: String
  sourceOrder: Int
  dataType: String!
  externalId: String!
  isNullable: Boolean!
  name: String!
  tableId: String!
  isPii: Boolean
  isPrimaryKey: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the column was deleted

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The external documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.sourceOrder</code>](#source-order)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-order */}
The original column position index as imported from the source warehouse

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.dataType</code>](#data-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #data-type */}
The data type for the column. For example: INTEGER

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical column identifier from the warehouse internal structure

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.isNullable</code>](#is-nullable)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-nullable */}
True if this column is nullable

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The column name

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
The column belongs to this table ID

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.isPii</code>](#is-pii)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-pii */}
The PII status to apply to the column

#### [<code style={{ fontWeight: 'normal' }}>CreateColumnInput.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
The PrimaryKey status to apply to the column

### Example

<TypeExample />

### Member Of

- [`createColumns`](/docs/catalog/api/general/operations/mutations/create-columns)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateDatabaseInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateDatabaseInput {
  deletedAt: DateTime
  description: String
  isHidden: Boolean
  externalId: String!
  name: String!
  sourceId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the database was deleted

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this database (max length: 500)

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Indicate if the database should be hidden

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical database identifier from the warehouse structure

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The database name

#### [<code style={{ fontWeight: 'normal' }}>CreateDatabaseInput.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
The id of the parent warehouse

### Example

<TypeExample />

### Member Of

- [`createDatabases`](/docs/catalog/api/general/operations/mutations/create-databases)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateExternalLinkInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateExternalLinkInput {
  tableId: String!
  technology: ExternalLinkTechnology!
  url: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateExternalLinkInput.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
The table to which the link refers

#### [<code style={{ fontWeight: 'normal' }}>CreateExternalLinkInput.technology</code>](#technology)<Bullet />[<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology!</code>](/docs/catalog/api/general/types/enums/external-link-technology) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #technology */}
The origin of the link

#### [<code style={{ fontWeight: 'normal' }}>CreateExternalLinkInput.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
the url of the link

### Example

<TypeExample />

### Member Of

- [`createExternalLinks`](/docs/catalog/api/general/operations/mutations/create-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateSchemaInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateSchemaInput {
  deletedAt: DateTime
  description: String
  isHidden: Boolean
  databaseId: String!
  externalId: String!
  name: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the schema was deleted

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this schema

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Whether the schema should be hidden

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
The id of the parent database

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical schema identifier from the warehouse schema structure

#### [<code style={{ fontWeight: 'normal' }}>CreateSchemaInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The schema name

### Example

<TypeExample />

### Member Of

- [`createSchemas`](/docs/catalog/api/general/operations/mutations/create-schemas)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateSourceInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateSourceInput {
  name: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateSourceInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the source to add

### Example

<TypeExample />

### Member Of

- [`createSource`](/docs/catalog/api/general/operations/mutations/create-source)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateTableInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateTableInput {
  externalDescription: String
  lastRefreshedAt: DateTime
  numberOfQueries: Int
  popularity: Int
  tableSize: Int
  tableType: TableType
  url: String
  externalId: String!
  name: String!
  schemaId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the source for this table

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.lastRefreshedAt</code>](#last-refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-refreshed-at */}
The last known datetime of refresh for this table

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.numberOfQueries</code>](#number-of-queries)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #number-of-queries */}
The number of queries on which popularity is based

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.popularity</code>](#popularity)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #popularity */}
The table popularity rated out of 1000000

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.tableSize</code>](#table-size)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-size */}
The size of the table (in megabytes)

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.tableType</code>](#table-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>TableType</code>](/docs/catalog/api/general/types/enums/table-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-type */}
The type of this table

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The external url to the table

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical table identifier from the warehouse table structure

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The table name

#### [<code style={{ fontWeight: 'normal' }}>CreateTableInput.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
The schema ID to link with the table that belongs to. Required at table creation. Cannot be modified later

### Example

<TypeExample />

### Member Of

- [`createTables`](/docs/catalog/api/general/operations/mutations/create-tables)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## CreateTermInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input CreateTermInput {
  description: String!
  linkedTagId: String
  name: String!
  parentTermId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>CreateTermInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
Description for the term, supports markdown

#### [<code style={{ fontWeight: 'normal' }}>CreateTermInput.linkedTagId</code>](#linked-tag-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #linked-tag-id */}
ID of the tag to link to the term

#### [<code style={{ fontWeight: 'normal' }}>CreateTermInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Name of the term

#### [<code style={{ fontWeight: 'normal' }}>CreateTermInput.parentTermId</code>](#parent-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-term-id */}
ID of the parent term, no value will create a root term

### Example

<TypeExample />

### Member Of

- [`createTerm`](/docs/catalog/api/general/operations/mutations/create-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## DashboardSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input DashboardSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: DashboardSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DashboardSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>DashboardSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>DashboardSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>DashboardSortingKey!</code>](/docs/catalog/api/general/types/enums/dashboard-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getDashboards`](/docs/catalog/api/general/operations/queries/get-dashboards)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## DataProductSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input DataProductSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: DataProductSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DataProductSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>DataProductSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>DataProductSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>DataProductSortingKey!</code>](/docs/catalog/api/general/types/enums/data-product-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getDataProducts`](/docs/catalog/api/general/operations/queries/get-data-products)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## DeleteExternalLinkInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input DeleteExternalLinkInput {
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DeleteExternalLinkInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The id of the link to delete

### Example

<TypeExample />

### Member Of

- [`deleteExternalLinks`](/docs/catalog/api/general/operations/mutations/delete-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## DeleteLineageInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input DeleteLineageInput {
  childDashboardId: String
  childTableId: String
  parentDashboardId: String
  parentTableId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DeleteLineageInput.childDashboardId</code>](#child-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-id */}
The id of the children dashboard of the lineage

#### [<code style={{ fontWeight: 'normal' }}>DeleteLineageInput.childTableId</code>](#child-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-table-id */}
The id of the children table of the lineage

#### [<code style={{ fontWeight: 'normal' }}>DeleteLineageInput.parentDashboardId</code>](#parent-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-id */}
The id of the parent dashboard of the lineage

#### [<code style={{ fontWeight: 'normal' }}>DeleteLineageInput.parentTableId</code>](#parent-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-table-id */}
The id of the parent table of the lineage

### Example

<TypeExample />

### Member Of

- [`deleteLineages`](/docs/catalog/api/general/operations/mutations/delete-lineages)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## DeleteTermInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input DeleteTermInput {
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DeleteTermInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
id of the term to delete

### Example

<TypeExample />

### Member Of

- [`deleteTerm`](/docs/catalog/api/general/operations/mutations/delete-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## EntitiesLinkInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input EntitiesLinkInput {
  from: EntitiesLinkTargetInput!
  to: EntitiesLinkTargetInput!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkInput.from</code>](#from)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetInput!</code>](/docs/catalog/api/general/types/inputs/entities-link-target-input) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #from */}
The entity from which the entitiesLink is issued

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkInput.to</code>](#to)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetInput!</code>](/docs/catalog/api/general/types/inputs/entities-link-target-input) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #to */}
The entity to which the entitiesLink points

### Example

```graphql
{
  "from": {}
}
```

### Member Of

- [`removePinnedAssets`](/docs/catalog/api/general/operations/mutations/remove-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`upsertPinnedAssets`](/docs/catalog/api/general/operations/mutations/upsert-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## EntitiesLinkSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input EntitiesLinkSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: EntitiesLinkSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntitiesLinkSortingKey!</code>](/docs/catalog/api/general/types/enums/entities-link-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getPinnedAssets`](/docs/catalog/api/general/operations/queries/get-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## EntitiesLinkTargetInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input EntitiesLinkTargetInput {
  id: String!
  type: EntitiesLinkTargetType!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The target entity id

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetInput.type</code>](#type)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntitiesLinkTargetType!</code>](/docs/catalog/api/general/types/enums/entities-link-target-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #type */}
The target entity type

### Example

<TypeExample />

### Member Of

- [`EntitiesLinkInput`](/docs/catalog/api/general/types/inputs/entities-link-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## EntityTarget

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input EntityTarget {
  entityId: String
  entityType: EntityTargetType
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntityTarget.entityId</code>](#entity-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-id */}
The id of the entity

#### [<code style={{ fontWeight: 'normal' }}>EntityTarget.entityType</code>](#entity-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntityTargetType</code>](/docs/catalog/api/general/types/enums/entity-target-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-type */}
The type of the entity

### Example

<TypeExample />

### Member Of

- [`OwnerInput`](/docs/catalog/api/general/types/inputs/owner-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TeamOwnerInput`](/docs/catalog/api/general/types/inputs/team-owner-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## FieldLineageSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input FieldLineageSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: FieldLineageSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>FieldLineageSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>FieldLineageSortingKey!</code>](/docs/catalog/api/general/types/enums/field-lineage-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getFieldLineages`](/docs/catalog/api/general/operations/queries/get-field-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetColumnJoinsScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetColumnJoinsScope {
  columnIds: [String!]
  ids: [String!]
  tableIds: [String!]
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsScope.columnIds</code>](#column-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #column-ids */}
Filter upon first and second column ids

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Filter upon column join ids

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsScope.tableIds</code>](#table-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids */}
Filter upon table ids

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getColumnJoins`](/docs/catalog/api/general/operations/queries/get-column-joins)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetColumnsScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetColumnsScope {
  name: String
  nameContains: String
  description: String
  databaseId: String
  databaseIds: [String!]
  ids: [String!]
  hasColumnJoins: Boolean
  isDocumented: Boolean
  isPii: Boolean
  isPrimaryKey: Boolean
  schemaId: String
  schemaIds: [String!]
  sourceId: String
  sourceIds: [String!]
  tableId: String
  tableIds: [String!]
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Scope by column name

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the column name

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
Scope by description

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
Scope by database

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.databaseIds</code>](#database-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-ids */}
Scope by database IDs

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Scope by ids

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.hasColumnJoins</code>](#has-column-joins)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #has-column-joins */}
Scope by hasColumnJoins

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.isDocumented</code>](#is-documented)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-documented */}
Scope by documentation status (either documented in the Catalog or an external source)

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.isPii</code>](#is-pii)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-pii */}
Scope by isPii (personally identifiable information)

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
Scope by isPrimaryKey

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
Scope by schema ID

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.schemaIds</code>](#schema-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-ids */}
Scope by schema IDs

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
Scope by source ID

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.sourceIds</code>](#source-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-ids */}
Scope by source IDs

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Scope by table ID

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.tableIds</code>](#table-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids */}
Scope by table IDs

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getColumns`](/docs/catalog/api/general/operations/queries/get-columns)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDashboardsScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetDashboardsScope {
  folderPath: String
  ids: [String!]
  nameContains: String
  sourceId: String
  withDeleted: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsScope.folderPath</code>](#folder-path)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #folder-path */}
Scope by the dashboard folder path, start from the root folder

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Scope by multiple dashboard IDs

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the dashboard name, case insensitive

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsScope.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
Scope by source ID

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

### Example

<TypeExample />

### Member Of

- [`getDashboards`](/docs/catalog/api/general/operations/queries/get-dashboards)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDataProductScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetDataProductScope {
  withTagId: String
  entityType: EntityTargetType
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductScope.withTagId</code>](#with-tag-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-tag-id */}
Scope by tag or domainid

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductScope.entityType</code>](#entity-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>EntityTargetType</code>](/docs/catalog/api/general/types/enums/entity-target-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-type */}
The type of the entities

### Example

<TypeExample />

### Member Of

- [`getDataProducts`](/docs/catalog/api/general/operations/queries/get-data-products)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDatabasesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetDatabasesScope {
  nameContains: String
  sourceIds: [String!]
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the field name

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesScope.sourceIds</code>](#source-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-ids */}
Scope by a list of source ids

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getDatabases`](/docs/catalog/api/general/operations/queries/get-databases)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetEntitiesLinksScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

No description

```graphql
input GetEntitiesLinksScope {
  fromDashboardId: String
  fromTableId: String
  fromTermId: String
  fromTermIds: [String!]
  toColumnsOfTableId: String
  toDashboardId: String
  toDashboardFieldId: String
  toFieldsOfDashboardId: String
  toTableId: String
  toTermId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.fromDashboardId</code>](#from-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-dashboard-id */}
Scope by dashboard the entities link is issued from

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.fromTableId</code>](#from-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-table-id */}
Scope by table the entities link is issued from

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.fromTermId</code>](#from-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-term-id */}
Scope by term the entities link is issued from

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.fromTermIds</code>](#from-term-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-term-ids */}
Scope by multiple term IDs from which the entity links are issued

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toColumnsOfTableId</code>](#to-columns-of-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-columns-of-table-id */}
Scope by the parent table of columns pointed to by the entities

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toDashboardId</code>](#to-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard-id */}
Scope by dashboard the entities link points to

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toDashboardFieldId</code>](#to-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard-field-id */}
Scope by dashboard field the entities link points to

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toFieldsOfDashboardId</code>](#to-fields-of-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-fields-of-dashboard-id */}
Scope by the parent dashboard of fields pointed to by the entities

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toTableId</code>](#to-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-table-id */}
Scope by table the entities link points to

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinksScope.toTermId</code>](#to-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-term-id */}
Scope by term the entities link points to

### Example

<TypeExample />

### Member Of

- [`getPinnedAssets`](/docs/catalog/api/general/operations/queries/get-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetFieldLineagesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetFieldLineagesScope {
  childColumnId: String
  childDashboardFieldId: String
  childDashboardFieldSourceId: String
  childDashboardSourceId: String
  columnSourceId: String
  hasDashboardChild: Boolean
  lineageType: LineageType
  parentColumnId: String
  parentDashboardFieldId: String
  withChildAssetType: LineageAssetType
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.childColumnId</code>](#child-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-column-id */}
Scope by child column

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.childDashboardFieldId</code>](#child-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-field-id */}
Scope by child dashboard field

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.childDashboardFieldSourceId</code>](#child-dashboard-field-source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-field-source-id */}
Scope by child dashboard field source

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.childDashboardSourceId</code>](#child-dashboard-source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-source-id */}
Scope by child dashboard source

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.columnSourceId</code>](#column-source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #column-source-id */}
Scope by column source (PARENT / CHILD)

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.hasDashboardChild</code>](#has-dashboard-child)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #has-dashboard-child */}
Scope on lineages with a dashboard child

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.lineageType</code>](#lineage-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageType</code>](/docs/catalog/api/general/types/enums/lineage-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #lineage-type */}
Scope on lineages type

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.parentColumnId</code>](#parent-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-column-id */}
Scope by parent column

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.parentDashboardFieldId</code>](#parent-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-field-id */}
Scope by parent dashboard field

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesScope.withChildAssetType</code>](#with-child-asset-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageAssetType</code>](/docs/catalog/api/general/types/enums/lineage-asset-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-child-asset-type */}
Scope to filter on child asset type

### Example

<TypeExample />

### Member Of

- [`getFieldLineages`](/docs/catalog/api/general/operations/queries/get-field-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetLineagesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetLineagesScope {
  childDashboardId: String
  childSourceId: String
  childTableId: String
  lineageIds: [String!]
  lineageType: LineageType
  parentDashboardId: String
  parentSourceId: String
  parentTableId: String
  withChildAssetType: LineageAssetType
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.childDashboardId</code>](#child-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-id */}
Scope by child dashboard ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.childSourceId</code>](#child-source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-source-id */}
Scope by child source ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.childTableId</code>](#child-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-table-id */}
Scope by child table ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.lineageIds</code>](#lineage-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #lineage-ids */}
Scope by multiple lineage IDs

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.lineageType</code>](#lineage-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageType</code>](/docs/catalog/api/general/types/enums/lineage-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #lineage-type */}
Scope by lineage type

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.parentDashboardId</code>](#parent-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-id */}
Scope by parent dashboard ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.parentSourceId</code>](#parent-source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-source-id */}
Scope by parent source ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.parentTableId</code>](#parent-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-table-id */}
Scope by parent table ID

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.withChildAssetType</code>](#with-child-asset-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageAssetType</code>](/docs/catalog/api/general/types/enums/lineage-asset-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-child-asset-type */}
Scope by the type of the child asset

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getLineages`](/docs/catalog/api/general/operations/queries/get-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetQualityChecksScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetQualityChecksScope {
  tableId: String
  withDeleted: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksScope.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Scope by table ID

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

### Example

<TypeExample />

### Member Of

- [`getDataQualities`](/docs/catalog/api/general/operations/queries/get-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetSchemasScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetSchemasScope {
  databaseIds: [String!]
  nameContains: String
  sourceIds: [String!]
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasScope.databaseIds</code>](#database-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-ids */}
Scope by a list of database ids

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the field name

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasScope.sourceIds</code>](#source-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-ids */}
Scope by a list of source ids

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getSchemas`](/docs/catalog/api/general/operations/queries/get-schemas)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetSourcesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetSourcesScope {
  nameContains: String
  origin: SourceOrigin
  technology: SourceTechnology
  type: SourceType
  withDeleted: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the source name

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesScope.origin</code>](#origin)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceOrigin</code>](/docs/catalog/api/general/types/enums/source-origin) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #origin */}
Scope by source origin

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesScope.technology</code>](#technology)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceTechnology</code>](/docs/catalog/api/general/types/enums/source-technology) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #technology */}
Scope by source technology

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesScope.type</code>](#type)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceType</code>](/docs/catalog/api/general/types/enums/source-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #type */}
Scope by source type

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

### Example

<TypeExample />

### Member Of

- [`getSources`](/docs/catalog/api/general/operations/queries/get-sources)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTableQueriesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetTableQueriesScope {
  databaseId: String
  queryType: QueryType
  schemaId: String
  tableIds: [String!]
  tableIdsFilterMode: SearchArrayFilterMode
  warehouseId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
Scope by database ID

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.queryType</code>](#query-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>QueryType</code>](/docs/catalog/api/general/types/enums/query-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #query-type */}
Filter by query type (SELECT or WRITE)

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
Scope by schema ID

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.tableIds</code>](#table-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids */}
Scope by specific table IDs (max 50 ids)

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.tableIdsFilterMode</code>](#table-ids-filter-mode)<Bullet />[<code style={{ fontWeight: 'normal' }}>SearchArrayFilterMode</code>](/docs/catalog/api/general/types/enums/search-array-filter-mode) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids-filter-mode */}
how to filter by table IDs - default is ALL

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesScope.warehouseId</code>](#warehouse-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #warehouse-id */}
Scope by warehouse ID

### Example

<TypeExample />

### Member Of

- [`getTableQueries`](/docs/catalog/api/general/operations/queries/get-table-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTablesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetTablesScope {
  databaseId: String
  ids: [String!]
  nameContains: String
  pathContains: String
  schemaId: String
  warehouseId: String
  withDeleted: Boolean
  withHidden: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
Scope by database ID

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Scope by multiple table IDs

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the table name, case insensitive

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.pathContains</code>](#path-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #path-contains */}
Scope by a substring of the table path, case insensitive

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
Scope by schema ID

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.warehouseId</code>](#warehouse-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #warehouse-id */}
Scope by warehouse ID

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.withDeleted</code>](#with-deleted)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-deleted */}
Scope by withDeleted, whether to include deleted assets

#### [<code style={{ fontWeight: 'normal' }}>GetTablesScope.withHidden</code>](#with-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #with-hidden */}
Scope by withHidden, whether to include hidden assets

### Example

<TypeExample />

### Member Of

- [`getTables`](/docs/catalog/api/general/operations/queries/get-tables)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTagsScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetTagsScope {
  ids: [String!]
  labelContains: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTagsScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Scope by multiple tag IDs

#### [<code style={{ fontWeight: 'normal' }}>GetTagsScope.labelContains</code>](#label-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #label-contains */}
Scope by a substring of the tag label, case insensitive

### Example

<TypeExample />

### Member Of

- [`getTags`](/docs/catalog/api/general/operations/queries/get-tags)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTermsScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input GetTermsScope {
  ids: [String!]
  nameContains: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTermsScope.ids</code>](#ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #ids */}
Scope by multiple term IDs

#### [<code style={{ fontWeight: 'normal' }}>GetTermsScope.nameContains</code>](#name-contains)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name-contains */}
Scope by a substring of the term name, case insensitive

### Example

<TypeExample />

### Member Of

- [`getTerms`](/docs/catalog/api/general/operations/queries/get-terms)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## JobResultInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input JobResultInput {
  delaySeconds: Int
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>JobResultInput.delaySeconds</code>](#delay-seconds)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #delay-seconds */}
The number of seconds to delay before returning the result if the job is not finished

#### [<code style={{ fontWeight: 'normal' }}>JobResultInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Job ID to retrieve the result for

### Example

<TypeExample />

### Member Of

- [`getAiAssistantJobResult`](/docs/catalog/api/general/operations/queries/get-ai-assistant-job-result)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## LineageSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input LineageSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: LineageSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>LineageSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>LineageSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>LineageSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageSortingKey!</code>](/docs/catalog/api/general/types/enums/lineage-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getLineages`](/docs/catalog/api/general/operations/queries/get-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## OwnerInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input OwnerInput {
  targetEntities: [EntityTarget!]
  userId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>OwnerInput.targetEntities</code>](#target-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntityTarget!]</code>](/docs/catalog/api/general/types/inputs/entity-target) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #target-entities */}
Scope by multiple target entities (dashboard, table, term)

#### [<code style={{ fontWeight: 'normal' }}>OwnerInput.userId</code>](#user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #user-id */}
The id of the user

### Example

```graphql
{
  "targetEntities": [
    {}
  ]
}
```

### Member Of

- [`removeUserOwners`](/docs/catalog/api/general/operations/mutations/remove-user-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`upsertUserOwners`](/docs/catalog/api/general/operations/mutations/upsert-user-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## PublicPagination

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Pagination options for the result

```graphql
input PublicPagination {
  greaterThanId: String
  nbPerPage: Int!
  page: Int
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>PublicPagination.greaterThanId</code>](#greater-than-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greater-than-id */}
Fetch entities with an id strictly greater than this. Enables cursor-based pagination. Cannot be used together with page.

#### [<code style={{ fontWeight: 'normal' }}>PublicPagination.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page (max 500)

#### [<code style={{ fontWeight: 'normal' }}>PublicPagination.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
Fetch entities at this page, start at page 0. Cannot be used together with greaterThanId.

### Example

<TypeExample />

### Member Of

- [`getColumnJoins`](/docs/catalog/api/general/operations/queries/get-column-joins)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getColumns`](/docs/catalog/api/general/operations/queries/get-columns)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getDashboards`](/docs/catalog/api/general/operations/queries/get-dashboards)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getDatabases`](/docs/catalog/api/general/operations/queries/get-databases)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getDataProducts`](/docs/catalog/api/general/operations/queries/get-data-products)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getDataQualities`](/docs/catalog/api/general/operations/queries/get-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getFieldLineages`](/docs/catalog/api/general/operations/queries/get-field-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getLineages`](/docs/catalog/api/general/operations/queries/get-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getPinnedAssets`](/docs/catalog/api/general/operations/queries/get-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getSchemas`](/docs/catalog/api/general/operations/queries/get-schemas)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getSources`](/docs/catalog/api/general/operations/queries/get-sources)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getTableQueries`](/docs/catalog/api/general/operations/queries/get-table-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getTables`](/docs/catalog/api/general/operations/queries/get-tables)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getTags`](/docs/catalog/api/general/operations/queries/get-tags)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getTeams`](/docs/catalog/api/general/operations/queries/get-teams)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getTerms`](/docs/catalog/api/general/operations/queries/get-terms)  <Badge class="badge badge--secondary badge--relation" text="query"/>
- [`getUsers`](/docs/catalog/api/general/operations/queries/get-users)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## QualityCheckInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Quality check identifier

```graphql
input QualityCheckInput {
  externalId: String!
  tableId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>QualityCheckInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical quality check identifier

#### [<code style={{ fontWeight: 'normal' }}>QualityCheckInput.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Table id the target quality check is linked to

### Example

<TypeExample />

### Member Of

- [`RemoveQualityChecksInput`](/docs/catalog/api/general/types/inputs/remove-quality-checks-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## QuerySorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input QuerySorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: QuerySortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>QuerySorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>QuerySorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>QuerySorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>QuerySortingKey!</code>](/docs/catalog/api/general/types/enums/query-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getTableQueries`](/docs/catalog/api/general/operations/queries/get-table-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## RemoveQualityChecksInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input RemoveQualityChecksInput {
  qualityChecks: [QualityCheckInput!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>RemoveQualityChecksInput.qualityChecks</code>](#quality-checks)<Bullet />[<code style={{ fontWeight: 'normal' }}>[QualityCheckInput!]!</code>](/docs/catalog/api/general/types/inputs/quality-check-input) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #quality-checks */}
Array of quality checks tableId/externalId keys

### Example

```graphql
{
  "qualityChecks": [
    {}
  ]
}
```

### Member Of

- [`removeDataQualities`](/docs/catalog/api/general/operations/mutations/remove-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## SearchQueriesInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input SearchQueriesInput {
  question: String!
  translateQuestionToEnglish: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SearchQueriesInput.question</code>](#question)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #question */}
Question used to find related queries. Must be less than 256 words and 1024 characters

#### [<code style={{ fontWeight: 'normal' }}>SearchQueriesInput.translateQuestionToEnglish</code>](#translate-question-to-english)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #translate-question-to-english */}
Enforce translation of the provided question to english. Can enhance results precision, however take more time

### Example

<TypeExample />

### Member Of

- [`searchQueries`](/docs/catalog/api/general/operations/queries/search-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## SearchQueriesScope

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Allow to refine selection of the results using multiple scopes. Note: all scopes are additive

```graphql
input SearchQueriesScope {
  filterMode: FilterTablesMode
  tableIds: [String!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SearchQueriesScope.filterMode</code>](#filter-mode)<Bullet />[<code style={{ fontWeight: 'normal' }}>FilterTablesMode</code>](/docs/catalog/api/general/types/enums/filter-tables-mode) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #filter-mode */}
How filter should capture provided table IDs. Default to "ALL"

#### [<code style={{ fontWeight: 'normal' }}>SearchQueriesScope.tableIds</code>](#table-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids */}
Filter queries using provided table IDs (max 10 ids)

### Example

```graphql
{
  "tableIds": []
}
```

### Member Of

- [`searchQueries`](/docs/catalog/api/general/operations/queries/search-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## TableSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input TableSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: TableSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TableSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>TableSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>TableSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>TableSortingKey!</code>](/docs/catalog/api/general/types/enums/table-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getTables`](/docs/catalog/api/general/operations/queries/get-tables)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## TagSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input TagSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: TagSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TagSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>TagSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>TagSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>TagSortingKey!</code>](/docs/catalog/api/general/types/enums/tag-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getTags`](/docs/catalog/api/general/operations/queries/get-tags)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## TeamOwnerInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input TeamOwnerInput {
  targetEntities: [EntityTarget!]
  teamId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerInput.targetEntities</code>](#target-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntityTarget!]</code>](/docs/catalog/api/general/types/inputs/entity-target) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #target-entities */}
Scope by multiple tag IDs

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerInput.teamId</code>](#team-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-id */}
The id of the team

### Example

```graphql
{
  "targetEntities": [
    {}
  ]
}
```

### Member Of

- [`removeTeamOwners`](/docs/catalog/api/general/operations/mutations/remove-team-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`upsertTeamOwners`](/docs/catalog/api/general/operations/mutations/upsert-team-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## TeamUsersInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input TeamUsersInput {
  emails: [String!]!
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TeamUsersInput.emails</code>](#emails)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #emails */}
Emails of users to add or remove from the team

#### [<code style={{ fontWeight: 'normal' }}>TeamUsersInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The id of the team

### Example

```graphql
{
  "emails": []
}
```

### Member Of

- [`addTeamUsers`](/docs/catalog/api/general/operations/mutations/add-team-users)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removeTeamUsers`](/docs/catalog/api/general/operations/mutations/remove-team-users)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## TermSorting

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Sorting options for results

```graphql
input TermSorting {
  direction: SortingDirectionEnum
  nullsPriority: SortingNullsPriority
  sortingKey: TermSortingKey!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TermSorting.direction</code>](#direction)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingDirectionEnum</code>](/docs/catalog/api/general/types/enums/sorting-direction-enum) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #direction */}
The direction to sort the results: by `ASC` or `DESC`

#### [<code style={{ fontWeight: 'normal' }}>TermSorting.nullsPriority</code>](#nulls-priority)<Bullet />[<code style={{ fontWeight: 'normal' }}>SortingNullsPriority</code>](/docs/catalog/api/general/types/enums/sorting-nulls-priority) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #nulls-priority */}
The position of null values in the results: `FIRST` or `LAST`

#### [<code style={{ fontWeight: 'normal' }}>TermSorting.sortingKey</code>](#sorting-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>TermSortingKey!</code>](/docs/catalog/api/general/types/enums/term-sorting-key) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #sorting-key */}
Available attributes to sort the results by

### Example

<TypeExample />

### Member Of

- [`getTerms`](/docs/catalog/api/general/operations/queries/get-terms)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## UpdateColumnDescriptionInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateColumnDescriptionInput {
  externalDescription: String
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnDescriptionInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The external description for this column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnDescriptionInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
the id of the column we want to update

### Example

<TypeExample />

### Member Of

- [`updateColumnDescriptions`](/docs/catalog/api/general/operations/mutations/update-column-descriptions)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateColumnInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateColumnInput {
  deletedAt: DateTime
  externalDescription: String
  sourceOrder: Int
  dataType: String
  externalId: String
  id: String!
  isNullable: Boolean
  isPii: Boolean
  isPrimaryKey: Boolean
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the column was deleted

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The external documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.sourceOrder</code>](#source-order)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-order */}
The original column position index as imported from the source warehouse

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.dataType</code>](#data-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #data-type */}
The data type for the column. For example: INTEGER

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical column identifier from the warehouse structure

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The id of the column you are looking for

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.isNullable</code>](#is-nullable)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-nullable */}
True if this column is nullable

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.isPii</code>](#is-pii)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-pii */}
The PII status to apply to the column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
The PrimaryKey status to apply to the column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The column name

### Example

<TypeExample />

### Member Of

- [`updateColumns`](/docs/catalog/api/general/operations/mutations/update-columns)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateColumnsMetadataInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateColumnsMetadataInput {
  externalDescription: String
  id: String!
  descriptionRaw: String
  isPii: Boolean
  isPrimaryKey: Boolean
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnsMetadataInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The external description for this column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnsMetadataInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
the id of the column we want to update

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnsMetadataInput.descriptionRaw</code>](#description-raw)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-raw */}
The user documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnsMetadataInput.isPii</code>](#is-pii)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-pii */}
True if this column is PII related

#### [<code style={{ fontWeight: 'normal' }}>UpdateColumnsMetadataInput.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
True if this column is a primary key

### Example

<TypeExample />

### Member Of

- [`updateColumnsMetadata`](/docs/catalog/api/general/operations/mutations/update-columns-metadata)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateDatabaseInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateDatabaseInput {
  deletedAt: DateTime
  description: String
  isHidden: Boolean
  externalId: String
  id: String!
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the database was deleted

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this database (max length: 500)

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Indicate if the database should be hidden

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical database identifier from the warehouse structure

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The database id

#### [<code style={{ fontWeight: 'normal' }}>UpdateDatabaseInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The database name

### Example

<TypeExample />

### Member Of

- [`updateDatabases`](/docs/catalog/api/general/operations/mutations/update-databases)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateExternalLinkInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateExternalLinkInput {
  id: String!
  url: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateExternalLinkInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
the id of the link to update

#### [<code style={{ fontWeight: 'normal' }}>UpdateExternalLinkInput.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
the url of the link

### Example

<TypeExample />

### Member Of

- [`updateExternalLinks`](/docs/catalog/api/general/operations/mutations/update-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateSchemaInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateSchemaInput {
  deletedAt: DateTime
  description: String
  isHidden: Boolean
  externalId: String
  id: String!
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the schema was deleted

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this schema

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Whether the schema should be hidden

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical schema identifier from the warehouse schema structure

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The schema id

#### [<code style={{ fontWeight: 'normal' }}>UpdateSchemaInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The schema name

### Example

<TypeExample />

### Member Of

- [`updateSchemas`](/docs/catalog/api/general/operations/mutations/update-schemas)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateSourceInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateSourceInput {
  deletedAt: DateTime
  id: String!
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateSourceInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
The time at which the source was soft deleted

#### [<code style={{ fontWeight: 'normal' }}>UpdateSourceInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The id of the source to update

#### [<code style={{ fontWeight: 'normal' }}>UpdateSourceInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the source to update

### Example

<TypeExample />

### Member Of

- [`updateSource`](/docs/catalog/api/general/operations/mutations/update-source)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateTableDescriptionInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateTableDescriptionInput {
  externalDescription: String
  id: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableDescriptionInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The external description for the table

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableDescriptionInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The table id

### Example

<TypeExample />

### Member Of

- [`updateTableDescriptions`](/docs/catalog/api/general/operations/mutations/update-table-descriptions)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateTableInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateTableInput {
  externalDescription: String
  lastRefreshedAt: DateTime
  numberOfQueries: Int
  popularity: Int
  tableSize: Int
  tableType: TableType
  url: String
  deletedAt: DateTime
  externalId: String
  id: String!
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the source for this table

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.lastRefreshedAt</code>](#last-refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-refreshed-at */}
The last known datetime of refresh for this table

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.numberOfQueries</code>](#number-of-queries)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #number-of-queries */}
The number of queries on which popularity is based

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.popularity</code>](#popularity)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #popularity */}
The table popularity rated out of 1000000

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.tableSize</code>](#table-size)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-size */}
The size of the table (in megabytes)

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.tableType</code>](#table-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>TableType</code>](/docs/catalog/api/general/types/enums/table-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-type */}
The type of this table

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The external url to the table

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>DateTime</code>](/docs/catalog/api/general/types/scalars/date-time) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
If present, indicates when the table was deleted

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical table identifier from the warehouse table structure

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The table id

#### [<code style={{ fontWeight: 'normal' }}>UpdateTableInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The table name

### Example

<TypeExample />

### Member Of

- [`updateTables`](/docs/catalog/api/general/operations/mutations/update-tables)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpdateTermInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpdateTermInput {
  description: String
  id: String!
  linkedTagId: String
  name: String
  parentTermId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpdateTermInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
Description for the term, supports markdown

#### [<code style={{ fontWeight: 'normal' }}>UpdateTermInput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
ID of the term to update

#### [<code style={{ fontWeight: 'normal' }}>UpdateTermInput.linkedTagId</code>](#linked-tag-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #linked-tag-id */}
ID of the tag to link to the term, remove by sending null

#### [<code style={{ fontWeight: 'normal' }}>UpdateTermInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Name of the term

#### [<code style={{ fontWeight: 'normal' }}>UpdateTermInput.parentTermId</code>](#parent-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-term-id */}
ID of the parent term, null will set as root term

### Example

<TypeExample />

### Member Of

- [`updateTerm`](/docs/catalog/api/general/operations/mutations/update-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpsertLineageInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpsertLineageInput {
  childDashboardId: String
  childTableId: String
  parentDashboardId: String
  parentTableId: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpsertLineageInput.childDashboardId</code>](#child-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-id */}
The id of the children dashboard of the lineage

#### [<code style={{ fontWeight: 'normal' }}>UpsertLineageInput.childTableId</code>](#child-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-table-id */}
The id of the children table of the lineage

#### [<code style={{ fontWeight: 'normal' }}>UpsertLineageInput.parentDashboardId</code>](#parent-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-id */}
The id of the parent dashboard of the lineage

#### [<code style={{ fontWeight: 'normal' }}>UpsertLineageInput.parentTableId</code>](#parent-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-table-id */}
The id of the parent table of the lineage

### Example

<TypeExample />

### Member Of

- [`upsertLineages`](/docs/catalog/api/general/operations/mutations/upsert-lineages)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpsertQualityChecksInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpsertQualityChecksInput {
  tableId: String!
  qualityChecks: [BaseQualityCheckInput!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpsertQualityChecksInput.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Table id linked to quality checks array.

#### [<code style={{ fontWeight: 'normal' }}>UpsertQualityChecksInput.qualityChecks</code>](#quality-checks)<Bullet />[<code style={{ fontWeight: 'normal' }}>[BaseQualityCheckInput!]!</code>](/docs/catalog/api/general/types/inputs/base-quality-check-input) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="input"/> <Badge class="badge badge--secondary " text="general"/> {/* #quality-checks */}
Array of quality checks

### Example

```graphql
{
  "qualityChecks": [
    {}
  ]
}
```

### Member Of

- [`upsertDataQualities`](/docs/catalog/api/general/operations/mutations/upsert-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## UpsertTeamInput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Input object expected by the query or mutation

```graphql
input UpsertTeamInput {
  description: String
  email: String
  name: String!
  storedPrivateSlackChannel: String
  slackChannel: String
  slackGroup: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The description of the team

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the team

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the team: unique across your account

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.storedPrivateSlackChannel</code>](#stored-private-slack-channel)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #stored-private-slack-channel */}
Private slack channel of the team. Used for notifications (only available for admin or service).  Don't forget to invite the slack bot into this channel!

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.slackChannel</code>](#slack-channel)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-channel */}
The slack channel of the team (start with #)

#### [<code style={{ fontWeight: 'normal' }}>UpsertTeamInput.slackGroup</code>](#slack-group)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-group */}
The slack group of the team (start with @)

### Example

<TypeExample />

### Member Of

- [`upsertTeam`](/docs/catalog/api/general/operations/mutations/upsert-team)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

---

## AddAiAssistantJobOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Response from the executed query or mutation

```graphql
type AddAiAssistantJobOutput {
  data: ConverseWithAssistantOutput!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>AddAiAssistantJobOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>ConverseWithAssistantOutput!</code>](/docs/catalog/api/general/types/objects/converse-with-assistant-output) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": {}
}
```

### Returned By

- [`addAiAssistantJob`](/docs/catalog/api/general/operations/queries/add-ai-assistant-job)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## AiAssistantJobResult

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Status and result of the AI Assistant job

```graphql
type AiAssistantJobResult {
  answer: String!
  assets: [AssetWithInternalLink!]!
  status: JobStatus!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>AiAssistantJobResult.answer</code>](#answer)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #answer */}
The answer to the question

#### [<code style={{ fontWeight: 'normal' }}>AiAssistantJobResult.assets</code>](#assets)<Bullet />[<code style={{ fontWeight: 'normal' }}>[AssetWithInternalLink!]!</code>](/docs/catalog/api/general/types/objects/asset-with-internal-link) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #assets */}
The list of assets referenced to generate the answer

#### [<code style={{ fontWeight: 'normal' }}>AiAssistantJobResult.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>JobStatus!</code>](/docs/catalog/api/general/types/enums/job-status) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
The current status of the job

### Example

```graphql
{
  "assets": [
    {}
  ]
}
```

### Member Of

- [`GetAiAssistantJobResultOutput`](/docs/catalog/api/general/types/objects/get-ai-assistant-job-result-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## AssetWithInternalLink

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Assets referenced in the AI Assistant job result

```graphql
type AssetWithInternalLink {
  id: String!
  internalLink: String!
  name: String!
  url: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>AssetWithInternalLink.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
ID of the asset

#### [<code style={{ fontWeight: 'normal' }}>AssetWithInternalLink.internalLink</code>](#internal-link)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #internal-link */}
The internal link of the asset (link to the asset in Catalog)

#### [<code style={{ fontWeight: 'normal' }}>AssetWithInternalLink.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the asset

#### [<code style={{ fontWeight: 'normal' }}>AssetWithInternalLink.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The url of the asset

### Example

<TypeExample />

### Member Of

- [`AiAssistantJobResult`](/docs/catalog/api/general/types/objects/ai-assistant-job-result)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ColumnJoin

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Represents a join relationship between two columns, including the number of times the join occurs. Each ColumnJoin links two columns and provides metadata about their association.

```graphql
type ColumnJoin {
  id: ID!
  count: Int!
  firstColumnId: String!
  firstColumn: Column!
  secondColumnId: String!
  secondColumn: Column!
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.count</code>](#count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #count */}
The number of join linking the columns

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.firstColumnId</code>](#first-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #first-column-id */}
The first column id on which the join happens

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.firstColumn</code>](#first-column)<Bullet />[<code style={{ fontWeight: 'normal' }}>Column!</code>](/docs/catalog/api/general/types/objects/column) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #first-column */}
The first column on which the join happens

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.secondColumnId</code>](#second-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #second-column-id */}
The second column id on which the join happens

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.secondColumn</code>](#second-column)<Bullet />[<code style={{ fontWeight: 'normal' }}>Column!</code>](/docs/catalog/api/general/types/objects/column) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #second-column */}
The second column on which the join happens

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>ColumnJoin.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "firstColumn": {
    "table": {
      "entityEditors": [
        {
          "sourceUser": {
            "unifiedUser": {
              "teamIds": []
            }
          }
        }
      ],
      "externalLinks": [
        {}
      ],
      "ownerEntities": [
        {
          "ownerLabel": {},
          "user": {
            "ownerEntities": []
          }
        }
      ],
      "schema": {
        "database": {
          "warehouse": {}
        }
      },
      "tagEntities": [
        {
          "tag": {}
        }
      ],
      "teamOwnerEntities": [
        {
          "team": {
            "teamOwnerEntities": []
          }
        }
      ]
    },
    "tagEntities": []
  }
}
```

### Member Of

- [`GetColumnJoinsOutput`](/docs/catalog/api/general/types/objects/get-column-joins-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Column

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A column represents a single field within a table. Each column is linked to a table and contains metadata such as data type, description, and technical identifiers.

```graphql
type Column {
  id: ID!
  dataType: String!
  descriptionRaw: String
  externalDescription: String
  externalDescriptionSource: ColumnExternalDescriptionSource
  externalId: String!
  isDescriptionGenerated: Boolean
  isNullable: Boolean!
  isPii: Boolean
  isPrimaryKey: Boolean
  name: String!
  sourceOrder: Int
  accountId: String!
  databaseId: String!
  describedByColumnId: String
  schemaId: String!
  sourceId: String!
  tableId: String!
  table: Table!
  tagEntities: [TagEntity!]!
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
  description: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Column.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Column.dataType</code>](#data-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #data-type */}
The data type for the column. For example: INTEGER

#### [<code style={{ fontWeight: 'normal' }}>Column.descriptionRaw</code>](#description-raw)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-raw */}
The Catalog documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>Column.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the source for this column

#### [<code style={{ fontWeight: 'normal' }}>Column.externalDescriptionSource</code>](#external-description-source)<Bullet />[<code style={{ fontWeight: 'normal' }}>ColumnExternalDescriptionSource</code>](/docs/catalog/api/general/types/enums/column-external-description-source) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description-source */}
The source of the documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>Column.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical column identifier from the warehouse internal structure

#### [<code style={{ fontWeight: 'normal' }}>Column.isDescriptionGenerated</code>](#is-description-generated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-description-generated */}
True if the description has been generated, false otherwise

#### [<code style={{ fontWeight: 'normal' }}>Column.isNullable</code>](#is-nullable)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-nullable */}
True if this column is nullable

#### [<code style={{ fontWeight: 'normal' }}>Column.isPii</code>](#is-pii)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-pii */}
True if this column is PII related

#### [<code style={{ fontWeight: 'normal' }}>Column.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
True if this column is a primary key

#### [<code style={{ fontWeight: 'normal' }}>Column.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The column name

#### [<code style={{ fontWeight: 'normal' }}>Column.sourceOrder</code>](#source-order)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-order */}
The original column position index as imported from the source warehouse

#### [<code style={{ fontWeight: 'normal' }}>Column.accountId</code>](#account-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #account-id */}
The column belongs to this account ID

#### [<code style={{ fontWeight: 'normal' }}>Column.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
The column belongs to this database ID

#### [<code style={{ fontWeight: 'normal' }}>Column.describedByColumnId</code>](#described-by-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #described-by-column-id */}
The column is described by this column id

#### [<code style={{ fontWeight: 'normal' }}>Column.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
The column belongs to this schema ID

#### [<code style={{ fontWeight: 'normal' }}>Column.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
The column belongs to this source ID

#### [<code style={{ fontWeight: 'normal' }}>Column.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
The column belongs to this table ID

#### [<code style={{ fontWeight: 'normal' }}>Column.table</code>](#table)<Bullet />[<code style={{ fontWeight: 'normal' }}>Table!</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #table */}
The column belongs to this table ID

#### [<code style={{ fontWeight: 'normal' }}>Column.tagEntities</code>](#tag-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TagEntity!]!</code>](/docs/catalog/api/general/types/objects/tag-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag-entities */}
The tag entities belonging to this column

#### [<code style={{ fontWeight: 'normal' }}>Column.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Column.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Column.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

#### [<code style={{ fontWeight: 'normal' }}>Column.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The column custom description or the external one if empty

### Example

```graphql
{
  "table": {
    "entityEditors": [
      {
        "sourceUser": {
          "unifiedUser": {
            "teamIds": []
          }
        }
      }
    ],
    "externalLinks": [
      {}
    ],
    "ownerEntities": [
      {
        "ownerLabel": {},
        "user": {
          "ownerEntities": []
        }
      }
    ],
    "schema": {
      "database": {
        "warehouse": {}
      }
    },
    "tagEntities": [
      {
        "tag": {}
      }
    ],
    "teamOwnerEntities": [
      {
        "team": {
          "teamOwnerEntities": []
        }
      }
    ]
  },
  "tagEntities": []
}
```

### Returned By

- [`createColumns`](/docs/catalog/api/general/operations/mutations/create-columns)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateColumnDescriptions`](/docs/catalog/api/general/operations/mutations/update-column-descriptions)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateColumns`](/docs/catalog/api/general/operations/mutations/update-columns)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateColumnsMetadata`](/docs/catalog/api/general/operations/mutations/update-columns-metadata)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`ColumnJoin`](/docs/catalog/api/general/types/objects/column-join)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnsOutput`](/docs/catalog/api/general/types/objects/get-columns-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ConverseWithAssistantOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Information about the started AI Assistant job

```graphql
type ConverseWithAssistantOutput {
  id: String!
  jobId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>ConverseWithAssistantOutput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The assistant message ID (answer)

#### [<code style={{ fontWeight: 'normal' }}>ConverseWithAssistantOutput.jobId</code>](#job-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #job-id */}
ID of the job processing the response. Can be used to pull the job result

### Example

<TypeExample />

### Member Of

- [`AddAiAssistantJobOutput`](/docs/catalog/api/general/types/objects/add-ai-assistant-job-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## DashboardField

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A dashboard field represents a single field within a dashboard. Each dashboard field is linked to a dashboard and contains metadata such as data type, description, and technical identifier.

```graphql
type DashboardField {
  id: ID!
  dataType: String!
  description: String
  externalId: String!
  isPrimaryKey: Boolean
  label: String!
  name: String!
  popularity: Int
  slug: String!
  viewLabel: String
  viewName: String
  dashboardId: String!
  dashboard: Dashboard!
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique identifier

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.dataType</code>](#data-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #data-type */}
The data type for the dashboard field

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this column

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical identifier within the source

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.isPrimaryKey</code>](#is-primary-key)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-primary-key */}
True if this field is a primary key

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.label</code>](#label)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #label */}
The dashboard field label

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The dashboard field name

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.popularity</code>](#popularity)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #popularity */}
Field popularity

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
The dashboard field slug

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.viewLabel</code>](#view-label)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #view-label */}
Name of the view used for display

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.viewName</code>](#view-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #view-name */}
Technical name of the view

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.dashboardId</code>](#dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard-id */}
Id of the dashboard this field belongs to

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.dashboard</code>](#dashboard)<Bullet />[<code style={{ fontWeight: 'normal' }}>Dashboard!</code>](/docs/catalog/api/general/types/objects/dashboard) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard */}
The field belongs to this dashboard

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Timestamp of when the record was soft-deleted

#### [<code style={{ fontWeight: 'normal' }}>DashboardField.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

```graphql
{
  "dashboard": {
    "entityEditors": [
      {
        "sourceUser": {
          "unifiedUser": {
            "teamIds": []
          }
        }
      }
    ],
    "ownerEntities": [
      {
        "ownerLabel": {},
        "user": {
          "ownerEntities": []
        }
      }
    ],
    "source": {},
    "tagEntities": [
      {
        "tag": {}
      }
    ],
    "teamOwnerEntities": [
      {
        "team": {
          "teamOwnerEntities": []
        }
      }
    ]
  }
}
```

### Member Of

- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Dashboard

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A dashboard represents a collection of visualizations and reports, providing insights into data. Each dashboard contains metadata, documentation, and links to its source and related entities.

```graphql
type Dashboard {
  id: ID!
  deprecatedAt: Timestamp
  descriptionRaw: String
  descriptionStateLexical: JSON
  externalDescription: String
  externalDescriptionSource: DashboardExternalDescriptionSource
  externalId: String!
  externalSlug: String
  folderPath: String!
  folderUrl: String
  isDescriptionGenerated: Boolean
  name: String!
  popularity: Int
  slug: String!
  type: DashboardType!
  url: String!
  verifiedAt: Timestamp
  deprecatedByUserId: String
  entityEditors: [EntityEditor!]!
  lastDescribedByUserId: String
  ownerEntities: [OwnerEntity!]!
  sourceId: String!
  source: Source!
  tagEntities: [TagEntity!]!
  teamOwnerEntities: [TeamOwnerEntity!]!
  verifiedByUserId: String
  verifiedBy: User
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
  description: String!
  isDeprecated: Boolean!
  isVerified: Boolean!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.deprecatedAt</code>](#deprecated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-at */}
The time at which the dashboard was deprecated

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.descriptionRaw</code>](#description-raw)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-raw */}
The Catalog documentation for this dashboard, as raw text. Can contains Markdown.

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.descriptionStateLexical</code>](#description-state-lexical)<Bullet />[<code style={{ fontWeight: 'normal' }}>JSON</code>](/docs/catalog/api/general/types/scalars/json) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-state-lexical */}
Rich-text description of this dashboard as lexical JSON

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the source for this dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.externalDescriptionSource</code>](#external-description-source)<Bullet />[<code style={{ fontWeight: 'normal' }}>DashboardExternalDescriptionSource</code>](/docs/catalog/api/general/types/enums/dashboard-external-description-source) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description-source */}
The source of the documentation for this dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The external ID of the dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.externalSlug</code>](#external-slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-slug */}
The external slug of the dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.folderPath</code>](#folder-path)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #folder-path */}
The folder path inside the external data source. Format is root/folder1/folder2

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.folderUrl</code>](#folder-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #folder-url */}
The url of the folder where the dashboard lies in the source

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.isDescriptionGenerated</code>](#is-description-generated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-description-generated */}
True if the description has been generated, false otherwise

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The dashboard name

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.popularity</code>](#popularity)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #popularity */}
·The dashboard popularity rated out of 1000000

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
Unique Catalog slug of the resource

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.type</code>](#type)<Bullet />[<code style={{ fontWeight: 'normal' }}>DashboardType!</code>](/docs/catalog/api/general/types/enums/dashboard-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #type */}
The dashboard's type

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The dashboard url in the source

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.verifiedAt</code>](#verified-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-at */}
The time at which the dashboard was verified

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.deprecatedByUserId</code>](#deprecated-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-by-user-id */}
The dashboard was deprecated by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.entityEditors</code>](#entity-editors)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntityEditor!]!</code>](/docs/catalog/api/general/types/objects/entity-editor) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-editors */}
The editors associated to this dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.lastDescribedByUserId</code>](#last-described-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-described-by-user-id */}
The dashboard was last described be the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.ownerEntities</code>](#owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[OwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-entities */}
The individual owners of the dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
The dashboard belongs to this data source ID

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.source</code>](#source)<Bullet />[<code style={{ fontWeight: 'normal' }}>Source!</code>](/docs/catalog/api/general/types/objects/source) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #source */}
The dashboard belongs to this data source

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.tagEntities</code>](#tag-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TagEntity!]!</code>](/docs/catalog/api/general/types/objects/tag-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag-entities */}
The tag entities belonging to this dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.teamOwnerEntities</code>](#team-owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TeamOwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/team-owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-owner-entities */}
The team owners of the dashboard

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.verifiedByUserId</code>](#verified-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by-user-id */}
The dashboard is verified by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.verifiedBy</code>](#verified-by)<Bullet />[<code style={{ fontWeight: 'normal' }}>User</code>](/docs/catalog/api/general/types/objects/user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by */}
The dashboard is verified by this user

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Timestamp of when the record was soft-deleted

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The dashboard custom rawDescription or the external one if empty

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.isDeprecated</code>](#is-deprecated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-deprecated */}
Whether this dashboard has been marked as deprecated

#### [<code style={{ fontWeight: 'normal' }}>Dashboard.isVerified</code>](#is-verified)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-verified */}
Whether this dashboard has been marked as verified

### Example

```graphql
{
  "entityEditors": [
    {
      "sourceUser": {
        "unifiedUser": {
          "teamIds": []
        }
      }
    }
  ],
  "ownerEntities": [
    {
      "ownerLabel": {},
      "user": {
        "ownerEntities": []
      }
    }
  ],
  "source": {},
  "tagEntities": [
    {
      "tag": {}
    }
  ],
  "teamOwnerEntities": [
    {
      "team": {
        "teamOwnerEntities": []
      }
    }
  ]
}
```

### Member Of

- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDashboardsOutput`](/docs/catalog/api/general/types/objects/get-dashboards-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## DataProduct

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

No description

```graphql
type DataProduct {
  id: ID!
  coverUrl: String
  dashboardId: String
  dashboard: Dashboard
  tableId: String
  table: Table
  termId: String
  term: Term
  createdAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.coverUrl</code>](#cover-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #cover-url */}
The data product’s cover URL

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.dashboardId</code>](#dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard-id */}
The dashboard id marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.dashboard</code>](#dashboard)<Bullet />[<code style={{ fontWeight: 'normal' }}>Dashboard</code>](/docs/catalog/api/general/types/objects/dashboard) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard */}
The dashboard marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
The table id marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.table</code>](#table)<Bullet />[<code style={{ fontWeight: 'normal' }}>Table</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #table */}
The table marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.termId</code>](#term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #term-id */}
The knowledge page id marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.term</code>](#term)<Bullet />[<code style={{ fontWeight: 'normal' }}>Term</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #term */}
The knowledge page marked as data product

#### [<code style={{ fontWeight: 'normal' }}>DataProduct.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

### Example

```graphql
{
  "dashboard": {
    "entityEditors": [
      {
        "sourceUser": {
          "unifiedUser": {
            "teamIds": []
          }
        }
      }
    ],
    "ownerEntities": [
      {
        "ownerLabel": {},
        "user": {
          "ownerEntities": []
        }
      }
    ],
    "source": {},
    "tagEntities": [
      {
        "tag": {}
      }
    ],
    "teamOwnerEntities": [
      {
        "team": {
          "teamOwnerEntities": []
        }
      }
    ]
  },
  "table": {
    "entityEditors": [],
    "externalLinks": [
      {}
    ],
    "ownerEntities": [],
    "schema": {
      "database": {}
    },
    "tagEntities": [],
    "teamOwnerEntities": []
  },
  "term": {
    "childrenTerms": [],
    "entityEditors": [],
    "ownerEntities": [],
    "tagEntities": [],
    "teamOwnerEntities": []
  }
}
```

### Member Of

- [`GetDataProductOutput`](/docs/catalog/api/general/types/objects/get-data-product-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Database

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A database represents a collection of tables and views, providing insights into data. Each database contains metadata, documentation, and links to its source and related entities.

```graphql
type Database {
  id: ID!
  description: String
  externalId: String!
  isHidden: Boolean!
  name: String!
  slug: String!
  sourceId: String!
  warehouse: Source!
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Database.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Database.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this database

#### [<code style={{ fontWeight: 'normal' }}>Database.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical database identifier from the warehouse database structure

#### [<code style={{ fontWeight: 'normal' }}>Database.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Whether this database is hidden

#### [<code style={{ fontWeight: 'normal' }}>Database.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The technical database identifier from the warehouse database structure

#### [<code style={{ fontWeight: 'normal' }}>Database.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
Unique Catalog slug of the resource

#### [<code style={{ fontWeight: 'normal' }}>Database.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
The source (warehouse) ID of the database

#### [<code style={{ fontWeight: 'normal' }}>Database.warehouse</code>](#warehouse)<Bullet />[<code style={{ fontWeight: 'normal' }}>Source!</code>](/docs/catalog/api/general/types/objects/source) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #warehouse */}
The warehouse of the database

#### [<code style={{ fontWeight: 'normal' }}>Database.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Database.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Database.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "warehouse": {}
}
```

### Returned By

- [`createDatabases`](/docs/catalog/api/general/operations/mutations/create-databases)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateDatabases`](/docs/catalog/api/general/operations/mutations/update-databases)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`GetDatabasesOutput`](/docs/catalog/api/general/types/objects/get-databases-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Schema`](/docs/catalog/api/general/types/objects/schema)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## EntitiesLink

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

No description

```graphql
type EntitiesLink {
  id: ID!
  fromDashboardId: String
  fromDashboard: Dashboard
  fromTableId: String
  fromTable: Table
  fromTermId: String
  fromTerm: Term
  toColumnId: String
  toColumn: Column
  toDashboardId: String
  toDashboard: Dashboard
  toDashboardFieldId: String
  toDashboardField: DashboardField
  toTableId: String
  toTable: Table
  toTermId: String
  toTerm: Term
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromDashboardId</code>](#from-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-dashboard-id */}
The dashboard id the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromDashboard</code>](#from-dashboard)<Bullet />[<code style={{ fontWeight: 'normal' }}>Dashboard</code>](/docs/catalog/api/general/types/objects/dashboard) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-dashboard */}
The dashboard the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromTableId</code>](#from-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-table-id */}
The table id the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromTable</code>](#from-table)<Bullet />[<code style={{ fontWeight: 'normal' }}>Table</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-table */}
The table the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromTermId</code>](#from-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-term-id */}
The knowledge page id the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.fromTerm</code>](#from-term)<Bullet />[<code style={{ fontWeight: 'normal' }}>Term</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #from-term */}
The knowledge page the link is issued from

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toColumnId</code>](#to-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-column-id */}
The column id the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toColumn</code>](#to-column)<Bullet />[<code style={{ fontWeight: 'normal' }}>Column</code>](/docs/catalog/api/general/types/objects/column) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-column */}
The column the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toDashboardId</code>](#to-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard-id */}
The dashboard id the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toDashboard</code>](#to-dashboard)<Bullet />[<code style={{ fontWeight: 'normal' }}>Dashboard</code>](/docs/catalog/api/general/types/objects/dashboard) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard */}
The dashboard the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toDashboardFieldId</code>](#to-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard-field-id */}
The dashboard field id the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toDashboardField</code>](#to-dashboard-field)<Bullet />[<code style={{ fontWeight: 'normal' }}>DashboardField</code>](/docs/catalog/api/general/types/objects/dashboard-field) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-dashboard-field */}
The dashboard field the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toTableId</code>](#to-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-table-id */}
The table id the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toTable</code>](#to-table)<Bullet />[<code style={{ fontWeight: 'normal' }}>Table</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-table */}
The table the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toTermId</code>](#to-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-term-id */}
The knowledge page id the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.toTerm</code>](#to-term)<Bullet />[<code style={{ fontWeight: 'normal' }}>Term</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #to-term */}
The knowledge page the link points to

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>EntitiesLink.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "fromDashboard": {
    "entityEditors": [
      {
        "sourceUser": {
          "unifiedUser": {
            "teamIds": []
          }
        }
      }
    ],
    "ownerEntities": [
      {
        "ownerLabel": {},
        "user": {
          "ownerEntities": []
        }
      }
    ],
    "source": {},
    "tagEntities": [
      {
        "tag": {}
      }
    ],
    "teamOwnerEntities": [
      {
        "team": {
          "teamOwnerEntities": []
        }
      }
    ]
  },
  "fromTable": {
    "entityEditors": [],
    "externalLinks": [
      {}
    ],
    "ownerEntities": [],
    "schema": {
      "database": {}
    },
    "tagEntities": [],
    "teamOwnerEntities": []
  },
  "fromTerm": {
    "childrenTerms": [],
    "entityEditors": [],
    "ownerEntities": [],
    "tagEntities": [],
    "teamOwnerEntities": []
  },
  "toColumn": {
    "tagEntities": []
  },
  "toDashboardField": {}
}
```

### Returned By

- [`upsertPinnedAssets`](/docs/catalog/api/general/operations/mutations/upsert-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`GetEntitiesLinkOutput`](/docs/catalog/api/general/types/objects/get-entities-link-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## EntityEditor

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Represents a user who is designated as an editor for a specific entity (dashboard, table, or knowledge).

```graphql
type EntityEditor {
  id: ID!
  dashboardId: String
  sourceUserId: String!
  sourceUser: SourceUser!
  tableId: String
  termId: String
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.dashboardId</code>](#dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard-id */}
Id of the associated dashboard

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.sourceUserId</code>](#source-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-user-id */}
ID of the user editor for this entity

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.sourceUser</code>](#source-user)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceUser!</code>](/docs/catalog/api/general/types/objects/source-user) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-user */}
Associated sourceUser

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Id of the associated table

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.termId</code>](#term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #term-id */}
Id of the associated term

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>EntityEditor.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

```graphql
{
  "sourceUser": {
    "unifiedUser": {
      "teamIds": []
    }
  }
}
```

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ExternalLink

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

An external link for a table represents a URL to an external system such as github

```graphql
type ExternalLink {
  id: ID!
  technology: ExternalLinkTechnology!
  url: String!
  tableId: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>ExternalLink.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>ExternalLink.technology</code>](#technology)<Bullet />[<code style={{ fontWeight: 'normal' }}>ExternalLinkTechnology!</code>](/docs/catalog/api/general/types/enums/external-link-technology) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #technology */}
The origin of the url

#### [<code style={{ fontWeight: 'normal' }}>ExternalLink.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The external link url

#### [<code style={{ fontWeight: 'normal' }}>ExternalLink.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
The link belongs to this data table ID

### Example

<TypeExample />

### Returned By

- [`createExternalLinks`](/docs/catalog/api/general/operations/mutations/create-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateExternalLinks`](/docs/catalog/api/general/operations/mutations/update-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## FieldLineage

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Field lineage link between your assets

```graphql
type FieldLineage {
  id: ID!
  lineageType: LineageType
  refreshedAt: Timestamp
  childColumnId: String
  childDashboardId: String
  childDashboardFieldId: String
  parentColumnId: String
  parentDashboardFieldId: String
  createdAt: Timestamp
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.lineageType</code>](#lineage-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageType</code>](/docs/catalog/api/general/types/enums/lineage-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #lineage-type */}
The type of the lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.refreshedAt</code>](#refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #refreshed-at */}
The last date on which this field lineage was recomputed

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.childColumnId</code>](#child-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-column-id */}
The id of the child column if it is a column lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.childDashboardId</code>](#child-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-id */}
The id of the child dashboard if it is a dashboard lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.childDashboardFieldId</code>](#child-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-field-id */}
The id of the child dashboard field if it is a dashboard field lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.parentColumnId</code>](#parent-column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-column-id */}
The id of the parent column if it is a column lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.parentDashboardFieldId</code>](#parent-dashboard-field-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-field-id */}
The id of the parent dashboard field if it is a dashboard field lineage

#### [<code style={{ fontWeight: 'normal' }}>FieldLineage.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

### Example

<TypeExample />

### Member Of

- [`GetFieldLineagesOutput`](/docs/catalog/api/general/types/objects/get-field-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## GetAiAssistantJobResultOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Response from the executed query or mutation

```graphql
type GetAiAssistantJobResultOutput {
  data: AiAssistantJobResult!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetAiAssistantJobResultOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>AiAssistantJobResult!</code>](/docs/catalog/api/general/types/objects/ai-assistant-job-result) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": {
    "assets": [
      {}
    ]
  }
}
```

### Returned By

- [`getAiAssistantJobResult`](/docs/catalog/api/general/operations/queries/get-ai-assistant-job-result)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetColumnJoinsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetColumnJoinsOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [ColumnJoin!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetColumnJoinsOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[ColumnJoin!]!</code>](/docs/catalog/api/general/types/objects/column-join) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "firstColumn": {
        "table": {
          "entityEditors": [
            {
              "sourceUser": {
                "unifiedUser": {
                  "teamIds": []
                }
              }
            }
          ],
          "externalLinks": [
            {}
          ],
          "ownerEntities": [
            {
              "ownerLabel": {},
              "user": {
                "ownerEntities": []
              }
            }
          ],
          "schema": {
            "database": {
              "warehouse": {}
            }
          },
          "tagEntities": [
            {
              "tag": {}
            }
          ],
          "teamOwnerEntities": [
            {
              "team": {
                "teamOwnerEntities": []
              }
            }
          ]
        },
        "tagEntities": []
      }
    }
  ]
}
```

### Returned By

- [`getColumnJoins`](/docs/catalog/api/general/operations/queries/get-column-joins)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetColumnsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetColumnsOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Column!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetColumnsOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Column!]!</code>](/docs/catalog/api/general/types/objects/column) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "table": {
        "entityEditors": [
          {
            "sourceUser": {
              "unifiedUser": {
                "teamIds": []
              }
            }
          }
        ],
        "externalLinks": [
          {}
        ],
        "ownerEntities": [
          {
            "ownerLabel": {},
            "user": {
              "ownerEntities": []
            }
          }
        ],
        "schema": {
          "database": {
            "warehouse": {}
          }
        },
        "tagEntities": [
          {
            "tag": {}
          }
        ],
        "teamOwnerEntities": [
          {
            "team": {
              "teamOwnerEntities": []
            }
          }
        ]
      },
      "tagEntities": []
    }
  ]
}
```

### Returned By

- [`getColumns`](/docs/catalog/api/general/operations/queries/get-columns)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDashboardsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetDashboardsOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Dashboard!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetDashboardsOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Dashboard!]!</code>](/docs/catalog/api/general/types/objects/dashboard) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "entityEditors": [
        {
          "sourceUser": {
            "unifiedUser": {
              "teamIds": []
            }
          }
        }
      ],
      "ownerEntities": [
        {
          "ownerLabel": {},
          "user": {
            "ownerEntities": []
          }
        }
      ],
      "source": {},
      "tagEntities": [
        {
          "tag": {}
        }
      ],
      "teamOwnerEntities": [
        {
          "team": {
            "teamOwnerEntities": []
          }
        }
      ]
    }
  ]
}
```

### Returned By

- [`getDashboards`](/docs/catalog/api/general/operations/queries/get-dashboards)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDataProductOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetDataProductOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [DataProduct!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetDataProductOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[DataProduct!]!</code>](/docs/catalog/api/general/types/objects/data-product) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "dashboard": {
        "entityEditors": [
          {
            "sourceUser": {
              "unifiedUser": {
                "teamIds": []
              }
            }
          }
        ],
        "ownerEntities": [
          {
            "ownerLabel": {},
            "user": {
              "ownerEntities": []
            }
          }
        ],
        "source": {},
        "tagEntities": [
          {
            "tag": {}
          }
        ],
        "teamOwnerEntities": [
          {
            "team": {
              "teamOwnerEntities": []
            }
          }
        ]
      },
      "table": {
        "entityEditors": [],
        "externalLinks": [
          {}
        ],
        "ownerEntities": [],
        "schema": {
          "database": {}
        },
        "tagEntities": [],
        "teamOwnerEntities": []
      },
      "term": {
        "childrenTerms": [],
        "entityEditors": [],
        "ownerEntities": [],
        "tagEntities": [],
        "teamOwnerEntities": []
      }
    }
  ]
}
```

### Returned By

- [`getDataProducts`](/docs/catalog/api/general/operations/queries/get-data-products)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetDatabasesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetDatabasesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Database!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetDatabasesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Database!]!</code>](/docs/catalog/api/general/types/objects/database) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "warehouse": {}
    }
  ]
}
```

### Returned By

- [`getDatabases`](/docs/catalog/api/general/operations/queries/get-databases)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetEntitiesLinkOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetEntitiesLinkOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [EntitiesLink!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinkOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinkOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinkOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinkOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetEntitiesLinkOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntitiesLink!]!</code>](/docs/catalog/api/general/types/objects/entities-link) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "fromDashboard": {
        "entityEditors": [
          {
            "sourceUser": {
              "unifiedUser": {
                "teamIds": []
              }
            }
          }
        ],
        "ownerEntities": [
          {
            "ownerLabel": {},
            "user": {
              "ownerEntities": []
            }
          }
        ],
        "source": {},
        "tagEntities": [
          {
            "tag": {}
          }
        ],
        "teamOwnerEntities": [
          {
            "team": {
              "teamOwnerEntities": []
            }
          }
        ]
      },
      "fromTable": {
        "entityEditors": [],
        "externalLinks": [
          {}
        ],
        "ownerEntities": [],
        "schema": {
          "database": {}
        },
        "tagEntities": [],
        "teamOwnerEntities": []
      },
      "fromTerm": {
        "childrenTerms": [],
        "entityEditors": [],
        "ownerEntities": [],
        "tagEntities": [],
        "teamOwnerEntities": []
      },
      "toColumn": {
        "tagEntities": []
      },
      "toDashboardField": {}
    }
  ]
}
```

### Returned By

- [`getPinnedAssets`](/docs/catalog/api/general/operations/queries/get-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetFieldLineagesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetFieldLineagesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [FieldLineage!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetFieldLineagesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[FieldLineage!]!</code>](/docs/catalog/api/general/types/objects/field-lineage) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {}
  ]
}
```

### Returned By

- [`getFieldLineages`](/docs/catalog/api/general/operations/queries/get-field-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetLineagesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetLineagesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Lineage!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetLineagesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Lineage!]!</code>](/docs/catalog/api/general/types/objects/lineage) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {}
  ]
}
```

### Returned By

- [`getLineages`](/docs/catalog/api/general/operations/queries/get-lineages)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetQualityChecksOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetQualityChecksOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [QualityCheck!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetQualityChecksOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[QualityCheck!]!</code>](/docs/catalog/api/general/types/objects/quality-check) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "column": {
        "table": {
          "entityEditors": [
            {
              "sourceUser": {
                "unifiedUser": {
                  "teamIds": []
                }
              }
            }
          ],
          "externalLinks": [
            {}
          ],
          "ownerEntities": [
            {
              "ownerLabel": {},
              "user": {
                "ownerEntities": []
              }
            }
          ],
          "schema": {
            "database": {
              "warehouse": {}
            }
          },
          "tagEntities": [
            {
              "tag": {}
            }
          ],
          "teamOwnerEntities": [
            {
              "team": {
                "teamOwnerEntities": []
              }
            }
          ]
        },
        "tagEntities": []
      }
    }
  ]
}
```

### Returned By

- [`getDataQualities`](/docs/catalog/api/general/operations/queries/get-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetSchemasOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetSchemasOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Schema!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetSchemasOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Schema!]!</code>](/docs/catalog/api/general/types/objects/schema) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "database": {
        "warehouse": {}
      }
    }
  ]
}
```

### Returned By

- [`getSchemas`](/docs/catalog/api/general/operations/queries/get-schemas)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetSourcesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetSourcesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Source!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetSourcesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Source!]!</code>](/docs/catalog/api/general/types/objects/source) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {}
  ]
}
```

### Returned By

- [`getSources`](/docs/catalog/api/general/operations/queries/get-sources)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTableQueriesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetTableQueriesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [TableQueryOutput!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetTableQueriesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TableQueryOutput!]!</code>](/docs/catalog/api/general/types/objects/table-query-output) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "databaseIds": [],
      "schemaIds": [],
      "tables": [
        {}
      ],
      "warehouseIds": []
    }
  ]
}
```

### Returned By

- [`getTableQueries`](/docs/catalog/api/general/operations/queries/get-table-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTablesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetTablesOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Table!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTablesOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTablesOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetTablesOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTablesOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetTablesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Table!]!</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "entityEditors": [
        {
          "sourceUser": {
            "unifiedUser": {
              "teamIds": []
            }
          }
        }
      ],
      "externalLinks": [
        {}
      ],
      "ownerEntities": [
        {
          "ownerLabel": {},
          "user": {
            "ownerEntities": []
          }
        }
      ],
      "schema": {
        "database": {
          "warehouse": {}
        }
      },
      "tagEntities": [
        {
          "tag": {}
        }
      ],
      "teamOwnerEntities": [
        {
          "team": {
            "teamOwnerEntities": []
          }
        }
      ]
    }
  ]
}
```

### Returned By

- [`getTables`](/docs/catalog/api/general/operations/queries/get-tables)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTagsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetTagsOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Tag!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTagsOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTagsOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetTagsOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTagsOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetTagsOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Tag!]!</code>](/docs/catalog/api/general/types/objects/tag) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {}
  ]
}
```

### Returned By

- [`getTags`](/docs/catalog/api/general/operations/queries/get-tags)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTeamsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Response from the executed query or mutation

```graphql
type GetTeamsOutput {
  createdAt: Timestamp!
  description: String
  email: String
  id: String!
  memberIds: [String!]!
  name: String!
  ownedAssetIds: [String!]!
  slackChannel: String
  slackGroup: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The description of the team

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the team

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
the unique id of the team

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.memberIds</code>](#member-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #member-ids */}
List of users ids that are members

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the team

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.ownedAssetIds</code>](#owned-asset-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #owned-asset-ids */}
List of entities ids owned by the team

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.slackChannel</code>](#slack-channel)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-channel */}
The slack channel of the team (start with #)

#### [<code style={{ fontWeight: 'normal' }}>GetTeamsOutput.slackGroup</code>](#slack-group)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-group */}
The slack group of the team (start with @)

### Example

```graphql
{
  "memberIds": [],
  "ownedAssetIds": []
}
```

### Returned By

- [`getTeams`](/docs/catalog/api/general/operations/queries/get-teams)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetTermsOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Paginated response from the executed query or mutation

```graphql
type GetTermsOutput {
  greatestId: String
  nbPerPage: Int!
  page: Int
  totalCount: Int!
  data: [Term!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetTermsOutput.greatestId</code>](#greatest-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #greatest-id */}
The greatest id returned, to pass as the next 'greaterThanId' when using cursor-based pagination. Returned only when page is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTermsOutput.nbPerPage</code>](#nb-per-page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #nb-per-page */}
Number of entities to return per page

#### [<code style={{ fontWeight: 'normal' }}>GetTermsOutput.page</code>](#page)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #page */}
The page number (start at 0). Returned only when greaterThanId is not used.

#### [<code style={{ fontWeight: 'normal' }}>GetTermsOutput.totalCount</code>](#total-count)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int!</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #total-count */}
The total number of entities

#### [<code style={{ fontWeight: 'normal' }}>GetTermsOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Term!]!</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "childrenTerms": [],
      "entityEditors": [
        {
          "sourceUser": {
            "unifiedUser": {
              "teamIds": []
            }
          }
        }
      ],
      "lastDescribedBy": {
        "ownerEntities": [
          {
            "ownerLabel": {}
          }
        ]
      },
      "linkedTag": {},
      "ownerEntities": [],
      "tagEntities": [
        {}
      ],
      "teamOwnerEntities": [
        {
          "team": {
            "teamOwnerEntities": []
          }
        }
      ]
    }
  ]
}
```

### Returned By

- [`getTerms`](/docs/catalog/api/general/operations/queries/get-terms)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## GetUsersOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Response from the executed query or mutation

```graphql
type GetUsersOutput {
  email: String!
  firstName: String!
  isEmailValidated: Boolean!
  id: String!
  lastName: String!
  role: UserRole!
  status: UserStatus!
  ownedAssetIds: [String!]!
  teamIds: [String!]!
  createdAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.firstName</code>](#first-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #first-name */}
The first name of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.isEmailValidated</code>](#is-email-validated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-email-validated */}
Whether the user has validated their email

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The ID of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.lastName</code>](#last-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-name */}
The last name of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.role</code>](#role)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserRole!</code>](/docs/catalog/api/general/types/enums/user-role) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #role */}
The role of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserStatus!</code>](/docs/catalog/api/general/types/enums/user-status) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
The status of the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.ownedAssetIds</code>](#owned-asset-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #owned-asset-ids */}
List of entities owned by the user

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.teamIds</code>](#team-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-ids */}
ids of the teams this user belongs to

#### [<code style={{ fontWeight: 'normal' }}>GetUsersOutput.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
The date and time the user was created

### Example

```graphql
{
  "ownedAssetIds": [],
  "teamIds": []
}
```

### Returned By

- [`getUsers`](/docs/catalog/api/general/operations/queries/get-users)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## Lineage

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Lineage link between your assets

```graphql
type Lineage {
  id: ID!
  lineageType: LineageType
  refreshedAt: Timestamp
  childDashboardId: String
  childTableId: String
  parentDashboardId: String
  parentTableId: String
  createdAt: Timestamp
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Lineage.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Lineage.lineageType</code>](#lineage-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>LineageType</code>](/docs/catalog/api/general/types/enums/lineage-type) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #lineage-type */}
The type of the lineage

#### [<code style={{ fontWeight: 'normal' }}>Lineage.refreshedAt</code>](#refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #refreshed-at */}
The last date on which this lineage was recomputed

#### [<code style={{ fontWeight: 'normal' }}>Lineage.childDashboardId</code>](#child-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-dashboard-id */}
The id of the child dashboard if it is a dashboard lineage

#### [<code style={{ fontWeight: 'normal' }}>Lineage.childTableId</code>](#child-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #child-table-id */}
The id of the child table if it is a table lineage

#### [<code style={{ fontWeight: 'normal' }}>Lineage.parentDashboardId</code>](#parent-dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-dashboard-id */}
The id of the parent dashboard if it is a dashboard lineage

#### [<code style={{ fontWeight: 'normal' }}>Lineage.parentTableId</code>](#parent-table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-table-id */}
The id of the parent table if it is a table lineage

#### [<code style={{ fontWeight: 'normal' }}>Lineage.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

### Example

<TypeExample />

### Returned By

- [`upsertLineages`](/docs/catalog/api/general/operations/mutations/upsert-lineages)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`GetLineagesOutput`](/docs/catalog/api/general/types/objects/get-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## OwnerEntity

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Represents a user who is designated as the owner of a dashboard, table, or term.

```graphql
type OwnerEntity {
  id: ID!
  dashboardId: String
  ownerLabelId: String
  ownerLabel: OwnerLabel
  tableId: String
  termId: String
  unifiedUser: UnifiedUser!
  userId: String
  user: User
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.dashboardId</code>](#dashboard-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #dashboard-id */}
Id of the dashboard owned

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.ownerLabelId</code>](#owner-label-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-label-id */}
Id of the owner label

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.ownerLabel</code>](#owner-label)<Bullet />[<code style={{ fontWeight: 'normal' }}>OwnerLabel</code>](/docs/catalog/api/general/types/objects/owner-label) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-label */}
The owner label

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Id of the table owned

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.termId</code>](#term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #term-id */}
Id of the term owned

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.unifiedUser</code>](#unified-user)<Bullet />[<code style={{ fontWeight: 'normal' }}>UnifiedUser!</code>](/docs/catalog/api/general/types/objects/unified-user) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #unified-user */}
The unified user linked to the owner

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.userId</code>](#user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #user-id */}
Id of the user who owns the entity

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.user</code>](#user)<Bullet />[<code style={{ fontWeight: 'normal' }}>User</code>](/docs/catalog/api/general/types/objects/user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #user */}
The owner

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>OwnerEntity.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

```graphql
{
  "ownerLabel": {},
  "unifiedUser": {
    "teamIds": []
  },
  "user": {
    "ownerEntities": []
  }
}
```

### Returned By

- [`upsertUserOwners`](/docs/catalog/api/general/operations/mutations/upsert-user-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## OwnerLabel

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

No description

```graphql
type OwnerLabel {
  id: ID!
  label: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>OwnerLabel.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique identifier

#### [<code style={{ fontWeight: 'normal' }}>OwnerLabel.label</code>](#label)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #label */}
The owner label

### Example

<TypeExample />

### Member Of

- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TeamOwnerEntity`](/docs/catalog/api/general/types/objects/team-owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## QualityCheck

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Represents a quality test for a table

```graphql
type QualityCheck {
  id: ID!
  description: String
  externalId: String!
  name: String!
  ownerEmail: String
  result: String
  runAt: Timestamp!
  source: String!
  status: QualityStatus!
  url: String
  columnId: String
  column: Column
  tableId: String!
  table: Table!
  createdAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
Quality check description

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical quality check identifier

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Quality check name

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.ownerEmail</code>](#owner-email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-email */}
Email of the owner of the quality check

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.result</code>](#result)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #result */}
Result is different from the status, it can contain information on why the test failed

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.runAt</code>](#run-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #run-at */}
Time at which the quality check ran

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.source</code>](#source)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source */}
Source that ran the quality check

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>QualityStatus!</code>](/docs/catalog/api/general/types/enums/quality-status) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
Status of the quality check

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
Url of the quality check

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.columnId</code>](#column-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #column-id */}
Column id linked to the quality check.

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.column</code>](#column)<Bullet />[<code style={{ fontWeight: 'normal' }}>Column</code>](/docs/catalog/api/general/types/objects/column) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #column */}
Column linked to the quality check.

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.tableId</code>](#table-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-id */}
Table id linked to the quality check.

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.table</code>](#table)<Bullet />[<code style={{ fontWeight: 'normal' }}>Table!</code>](/docs/catalog/api/general/types/objects/table) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #table */}
Table linked to the quality check.

#### [<code style={{ fontWeight: 'normal' }}>QualityCheck.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

### Example

```graphql
{
  "column": {
    "table": {
      "entityEditors": [
        {
          "sourceUser": {
            "unifiedUser": {
              "teamIds": []
            }
          }
        }
      ],
      "externalLinks": [
        {}
      ],
      "ownerEntities": [
        {
          "ownerLabel": {},
          "user": {
            "ownerEntities": []
          }
        }
      ],
      "schema": {
        "database": {
          "warehouse": {}
        }
      },
      "tagEntities": [
        {
          "tag": {}
        }
      ],
      "teamOwnerEntities": [
        {
          "team": {
            "teamOwnerEntities": []
          }
        }
      ]
    },
    "tagEntities": []
  }
}
```

### Returned By

- [`upsertDataQualities`](/docs/catalog/api/general/operations/mutations/upsert-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`GetQualityChecksOutput`](/docs/catalog/api/general/types/objects/get-quality-checks-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Schema

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A schema represents a collection of tables and views.

```graphql
type Schema {
  id: ID!
  description: String
  externalId: String!
  isHidden: Boolean!
  name: String!
  slug: String!
  databaseId: String!
  database: Database!
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Schema.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Schema.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The Catalog documentation for this schema

#### [<code style={{ fontWeight: 'normal' }}>Schema.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical schema identifier from the warehouse schema structure

#### [<code style={{ fontWeight: 'normal' }}>Schema.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
Whether this schema is hidden

#### [<code style={{ fontWeight: 'normal' }}>Schema.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the schema

#### [<code style={{ fontWeight: 'normal' }}>Schema.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
Unique Catalog slug of the resource

#### [<code style={{ fontWeight: 'normal' }}>Schema.databaseId</code>](#database-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-id */}
The database ID of the schema

#### [<code style={{ fontWeight: 'normal' }}>Schema.database</code>](#database)<Bullet />[<code style={{ fontWeight: 'normal' }}>Database!</code>](/docs/catalog/api/general/types/objects/database) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #database */}
The database linked to this schema

#### [<code style={{ fontWeight: 'normal' }}>Schema.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Schema.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Schema.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "database": {
    "warehouse": {}
  }
}
```

### Returned By

- [`createSchemas`](/docs/catalog/api/general/operations/mutations/create-schemas)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateSchemas`](/docs/catalog/api/general/operations/mutations/update-schemas)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`GetSchemasOutput`](/docs/catalog/api/general/types/objects/get-schemas-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SearchQueriesOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Response from the executed query or mutation

```graphql
type SearchQueriesOutput {
  data: [SearchQueryResult!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SearchQueriesOutput.data</code>](#data)<Bullet />[<code style={{ fontWeight: 'normal' }}>[SearchQueryResult!]!</code>](/docs/catalog/api/general/types/objects/search-query-result) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #data */}
Field containing the response from the executed query or mutation

### Example

```graphql
{
  "data": [
    {
      "author": {},
      "tableIds": []
    }
  ]
}
```

### Returned By

- [`searchQueries`](/docs/catalog/api/general/operations/queries/search-queries)  <Badge class="badge badge--secondary badge--relation" text="query"/>

---

## SearchQueryResultAuthor

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Query author information

```graphql
type SearchQueryResultAuthor {
  email: String
  name: String
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SearchQueryResultAuthor.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
Query author email

#### [<code style={{ fontWeight: 'normal' }}>SearchQueryResultAuthor.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
Query author name

### Example

<TypeExample />

### Member Of

- [`SearchQueryResult`](/docs/catalog/api/general/types/objects/search-query-result)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SearchQueryResult

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A single query result

```graphql
type SearchQueryResult {
  author: SearchQueryResultAuthor
  query: String!
  tableIds: [String!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SearchQueryResult.author</code>](#author)<Bullet />[<code style={{ fontWeight: 'normal' }}>SearchQueryResultAuthor</code>](/docs/catalog/api/general/types/objects/search-query-result-author) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #author */}
Query author information

#### [<code style={{ fontWeight: 'normal' }}>SearchQueryResult.query</code>](#query)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #query */}
Query matching provided question

#### [<code style={{ fontWeight: 'normal' }}>SearchQueryResult.tableIds</code>](#table-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-ids */}
Table IDs used by this query

### Example

```graphql
{
  "author": {},
  "tableIds": []
}
```

### Member Of

- [`SearchQueriesOutput`](/docs/catalog/api/general/types/objects/search-queries-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## SourceUser

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Represents a user imported from an external data source, including their name, email, etc.

```graphql
type SourceUser {
  id: ID!
  avatarUrl: String
  email: String
  externalId: String!
  name: String
  sourceId: String!
  unifiedId: String
  unifiedUser: UnifiedUser
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.avatarUrl</code>](#avatar-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #avatar-url */}
The URL of the user avatar

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The optional email for this source query

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The internal reference to this user in the source

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the user

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.sourceId</code>](#source-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-id */}
The source user belongs to this source ID.

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.unifiedId</code>](#unified-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #unified-id */}
The precomputed unifiedId

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.unifiedUser</code>](#unified-user)<Bullet />[<code style={{ fontWeight: 'normal' }}>UnifiedUser</code>](/docs/catalog/api/general/types/objects/unified-user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #unified-user */}
The unified user associated to that source user.

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Timestamp of when the record was soft-deleted

#### [<code style={{ fontWeight: 'normal' }}>SourceUser.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

```graphql
{
  "unifiedUser": {
    "teamIds": []
  }
}
```

### Member Of

- [`EntityEditor`](/docs/catalog/api/general/types/objects/entity-editor)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Source

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A source represents a data source that can be used to create dashboards, tables, and knowledge. Each source contains metadata, documentation, and links to its related entities.

```graphql
type Source {
  id: ID!
  lastRefreshedAt: Timestamp
  name: String!
  origin: SourceOrigin!
  slug: String!
  technology: SourceTechnology!
  type: SourceType!
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Source.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Source.lastRefreshedAt</code>](#last-refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-refreshed-at */}
The time at which the source was last ingested by service

#### [<code style={{ fontWeight: 'normal' }}>Source.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the source

#### [<code style={{ fontWeight: 'normal' }}>Source.origin</code>](#origin)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceOrigin!</code>](/docs/catalog/api/general/types/enums/source-origin) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #origin */}
Whether the source is issued from the API or the extraction

#### [<code style={{ fontWeight: 'normal' }}>Source.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
Unique Catalog slug of the resource

#### [<code style={{ fontWeight: 'normal' }}>Source.technology</code>](#technology)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceTechnology!</code>](/docs/catalog/api/general/types/enums/source-technology) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #technology */}
The technology of the source

#### [<code style={{ fontWeight: 'normal' }}>Source.type</code>](#type)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceType!</code>](/docs/catalog/api/general/types/enums/source-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #type */}
Type of the source

#### [<code style={{ fontWeight: 'normal' }}>Source.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Source.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Source.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

<TypeExample />

### Returned By

- [`createSource`](/docs/catalog/api/general/operations/mutations/create-source)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateSource`](/docs/catalog/api/general/operations/mutations/update-source)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Database`](/docs/catalog/api/general/types/objects/database)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetSourcesOutput`](/docs/catalog/api/general/types/objects/get-sources-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TableInQueryOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Table data related to a SQL query

```graphql
type TableInQueryOutput {
  id: String!
  name: String!
  path: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TableInQueryOutput.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
The table id

#### [<code style={{ fontWeight: 'normal' }}>TableInQueryOutput.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The table name

#### [<code style={{ fontWeight: 'normal' }}>TableInQueryOutput.path</code>](#path)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #path */}
The table path

### Example

<TypeExample />

### Member Of

- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TableQueryOutput

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A SQL query being run on the warehouse either to READ or WRITE data

```graphql
type TableQueryOutput {
  author: String
  databaseIds: [String!]!
  hash: ID!
  isHidden: Boolean!
  query: String!
  queriesDuration: Float
  queryType: QueryType!
  rowsProduced: Int
  schemaIds: [String!]!
  snowflakeWarehouseSize: String
  sourceUserId: String
  tables: [TableInQueryOutput!]!
  timestamp: Timestamp!
  warehouseIds: [String!]!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.author</code>](#author)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #author */}
The query's author

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.databaseIds</code>](#database-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #database-ids */}
The database ids

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.hash</code>](#hash)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #hash */}
The query identifier

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.isHidden</code>](#is-hidden)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-hidden */}
The query's visibility

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.query</code>](#query)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #query */}
The query's content

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.queriesDuration</code>](#queries-duration)<Bullet />[<code style={{ fontWeight: 'normal' }}>Float</code>](/docs/catalog/api/general/types/scalars/float) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #queries-duration */}
The query duration (in seconds)

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.queryType</code>](#query-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>QueryType!</code>](/docs/catalog/api/general/types/enums/query-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #query-type */}
The query's type

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.rowsProduced</code>](#rows-produced)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #rows-produced */}
The number of rows produced by the query

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.schemaIds</code>](#schema-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-ids */}
The schema ids

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.snowflakeWarehouseSize</code>](#snowflake-warehouse-size)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #snowflake-warehouse-size */}
The snowflake warehouse size used to run the query

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.sourceUserId</code>](#source-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source-user-id */}
The source user id associated to the query

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.tables</code>](#tables)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TableInQueryOutput!]!</code>](/docs/catalog/api/general/types/objects/table-in-query-output) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tables */}
The tables used in the query

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.timestamp</code>](#timestamp)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #timestamp */}
date of the execution of the query

#### [<code style={{ fontWeight: 'normal' }}>TableQueryOutput.warehouseIds</code>](#warehouse-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #warehouse-ids */}
The warehouse ids

### Example

```graphql
{
  "databaseIds": [],
  "schemaIds": [],
  "tables": [
    {}
  ],
  "warehouseIds": []
}
```

### Member Of

- [`GetTableQueriesOutput`](/docs/catalog/api/general/types/objects/get-table-queries-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Table

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A table represents a collection of data stored in a database. Each table contains columns, metadata, documentation, and links to its related entities.

```graphql
type Table {
  id: ID!
  deprecatedAt: Timestamp
  descriptionRaw: String
  descriptionStateLexical: JSON
  externalDescription: String
  externalDescriptionSource: TableExternalDescriptionSource
  externalId: String!
  isDescriptionGenerated: Boolean
  lastQueriedAt: Timestamp
  lastRefreshedAt: Timestamp
  name: String!
  numberOfQueries: Int
  popularity: Int
  slug: String!
  tableSize: Int
  tableType: TableType!
  transformationSource: TransformationSource
  url: String
  verifiedAt: Timestamp
  deprecatedByUserId: String
  entityEditors: [EntityEditor!]!
  externalLinks: [ExternalLink!]
  lastDescribedByUserId: String
  ownerEntities: [OwnerEntity!]!
  schemaId: String!
  schema: Schema!
  tagEntities: [TagEntity!]!
  teamOwnerEntities: [TeamOwnerEntity!]!
  verifiedByUserId: String
  verifiedBy: User
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
  description: String!
  isDeprecated: Boolean!
  isVerified: Boolean!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Table.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Table.deprecatedAt</code>](#deprecated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-at */}
The time at which the table was deprecated

#### [<code style={{ fontWeight: 'normal' }}>Table.descriptionRaw</code>](#description-raw)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-raw */}
The Catalog documentation for this table, as raw text. Can contain Markdown.

#### [<code style={{ fontWeight: 'normal' }}>Table.descriptionStateLexical</code>](#description-state-lexical)<Bullet />[<code style={{ fontWeight: 'normal' }}>JSON</code>](/docs/catalog/api/general/types/scalars/json) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-state-lexical */}
Rich-text description of this table as lexical JSON

#### [<code style={{ fontWeight: 'normal' }}>Table.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the source for this table

#### [<code style={{ fontWeight: 'normal' }}>Table.externalDescriptionSource</code>](#external-description-source)<Bullet />[<code style={{ fontWeight: 'normal' }}>TableExternalDescriptionSource</code>](/docs/catalog/api/general/types/enums/table-external-description-source) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description-source */}
The source of the documentation for this table

#### [<code style={{ fontWeight: 'normal' }}>Table.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
The technical table identifier from the warehouse table structure

#### [<code style={{ fontWeight: 'normal' }}>Table.isDescriptionGenerated</code>](#is-description-generated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-description-generated */}
True if the description has been generated, false otherwise

#### [<code style={{ fontWeight: 'normal' }}>Table.lastQueriedAt</code>](#last-queried-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-queried-at */}
The last datetime at which we queried for this table freshness

#### [<code style={{ fontWeight: 'normal' }}>Table.lastRefreshedAt</code>](#last-refreshed-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-refreshed-at */}
The last known datetime of refresh for this table

#### [<code style={{ fontWeight: 'normal' }}>Table.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The table name

#### [<code style={{ fontWeight: 'normal' }}>Table.numberOfQueries</code>](#number-of-queries)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #number-of-queries */}
The number of queries on which popularity is based

#### [<code style={{ fontWeight: 'normal' }}>Table.popularity</code>](#popularity)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #popularity */}
·The table popularity rated out of 1000000

#### [<code style={{ fontWeight: 'normal' }}>Table.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
The Catalog slug of the table

#### [<code style={{ fontWeight: 'normal' }}>Table.tableSize</code>](#table-size)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-size */}
The size of the table (in megabytes)

#### [<code style={{ fontWeight: 'normal' }}>Table.tableType</code>](#table-type)<Bullet />[<code style={{ fontWeight: 'normal' }}>TableType!</code>](/docs/catalog/api/general/types/enums/table-type) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #table-type */}
The type of table asset: e.g. view, table

#### [<code style={{ fontWeight: 'normal' }}>Table.transformationSource</code>](#transformation-source)<Bullet />[<code style={{ fontWeight: 'normal' }}>TransformationSource</code>](/docs/catalog/api/general/types/enums/transformation-source) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #transformation-source */}
The transformation source technology of this table

#### [<code style={{ fontWeight: 'normal' }}>Table.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The external url to the table

#### [<code style={{ fontWeight: 'normal' }}>Table.verifiedAt</code>](#verified-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-at */}
The time at which the table was certified

#### [<code style={{ fontWeight: 'normal' }}>Table.deprecatedByUserId</code>](#deprecated-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-by-user-id */}
The table was deprecated by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Table.entityEditors</code>](#entity-editors)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntityEditor!]!</code>](/docs/catalog/api/general/types/objects/entity-editor) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-editors */}
The editors associated to this table

#### [<code style={{ fontWeight: 'normal' }}>Table.externalLinks</code>](#external-links)<Bullet />[<code style={{ fontWeight: 'normal' }}>[ExternalLink!]</code>](/docs/catalog/api/general/types/objects/external-link) <Badge class="badge badge--secondary " text="list"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-links */}
External Links of a table

#### [<code style={{ fontWeight: 'normal' }}>Table.lastDescribedByUserId</code>](#last-described-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-described-by-user-id */}
The table was last described be the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Table.ownerEntities</code>](#owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[OwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-entities */}
The individual owners of the table

#### [<code style={{ fontWeight: 'normal' }}>Table.schemaId</code>](#schema-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema-id */}
The table belongs to this schema ID

#### [<code style={{ fontWeight: 'normal' }}>Table.schema</code>](#schema)<Bullet />[<code style={{ fontWeight: 'normal' }}>Schema!</code>](/docs/catalog/api/general/types/objects/schema) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #schema */}
The table belongs to this schema

#### [<code style={{ fontWeight: 'normal' }}>Table.tagEntities</code>](#tag-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TagEntity!]!</code>](/docs/catalog/api/general/types/objects/tag-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag-entities */}
Tag associated to this table

#### [<code style={{ fontWeight: 'normal' }}>Table.teamOwnerEntities</code>](#team-owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TeamOwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/team-owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-owner-entities */}
The team owners of the table

#### [<code style={{ fontWeight: 'normal' }}>Table.verifiedByUserId</code>](#verified-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by-user-id */}
The table is verified by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Table.verifiedBy</code>](#verified-by)<Bullet />[<code style={{ fontWeight: 'normal' }}>User</code>](/docs/catalog/api/general/types/objects/user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by */}
The table is verified by this user

#### [<code style={{ fontWeight: 'normal' }}>Table.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Table.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Table.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

#### [<code style={{ fontWeight: 'normal' }}>Table.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The table custom rawDescription or the external one if empty

#### [<code style={{ fontWeight: 'normal' }}>Table.isDeprecated</code>](#is-deprecated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-deprecated */}
Whether this table has been marked as deprecated

#### [<code style={{ fontWeight: 'normal' }}>Table.isVerified</code>](#is-verified)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-verified */}
Whether this table has been marked as verified

### Example

```graphql
{
  "entityEditors": [
    {
      "sourceUser": {
        "unifiedUser": {
          "teamIds": []
        }
      }
    }
  ],
  "externalLinks": [
    {}
  ],
  "ownerEntities": [
    {
      "ownerLabel": {},
      "user": {
        "ownerEntities": []
      }
    }
  ],
  "schema": {
    "database": {
      "warehouse": {}
    }
  },
  "tagEntities": [
    {
      "tag": {}
    }
  ],
  "teamOwnerEntities": [
    {
      "team": {
        "teamOwnerEntities": []
      }
    }
  ]
}
```

### Returned By

- [`createTables`](/docs/catalog/api/general/operations/mutations/create-tables)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateTableDescriptions`](/docs/catalog/api/general/operations/mutations/update-table-descriptions)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateTables`](/docs/catalog/api/general/operations/mutations/update-tables)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTablesOutput`](/docs/catalog/api/general/types/objects/get-tables-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TagEntity

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A tag entity represents a tag applied to an entity (dashboard, table, column, etc.)

```graphql
type TagEntity {
  id: ID!
  origin: TagEntityOrigin!
  technology: SourceTechnology
  tagId: String!
  tag: Tag!
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.origin</code>](#origin)<Bullet />[<code style={{ fontWeight: 'normal' }}>TagEntityOrigin!</code>](/docs/catalog/api/general/types/enums/tag-entity-origin) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #origin */}
Author of a tag entity

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.technology</code>](#technology)<Bullet />[<code style={{ fontWeight: 'normal' }}>SourceTechnology</code>](/docs/catalog/api/general/types/enums/source-technology) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #technology */}
The source technology that brought the tag (for EXTERNAL tag entities only)

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.tagId</code>](#tag-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag-id */}
Id of the tag concerned

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.tag</code>](#tag)<Bullet />[<code style={{ fontWeight: 'normal' }}>Tag!</code>](/docs/catalog/api/general/types/objects/tag) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag */}
The tag concerned

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>TagEntity.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

```graphql
{
  "tag": {}
}
```

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Tag

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A tag represents a label applied to entities (dashboards, tables, columns, etc.) to categorize and organize them.

```graphql
type Tag {
  id: ID!
  color: Colors
  label: String!
  slug: String!
  linkedTermId: String
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Tag.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Tag.color</code>](#color)<Bullet />[<code style={{ fontWeight: 'normal' }}>Colors</code>](/docs/catalog/api/general/types/enums/colors) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #color */}
The color given to the tag

#### [<code style={{ fontWeight: 'normal' }}>Tag.label</code>](#label)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #label */}
The label given to the tag

#### [<code style={{ fontWeight: 'normal' }}>Tag.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
Unique Catalog slug of the resource

#### [<code style={{ fontWeight: 'normal' }}>Tag.linkedTermId</code>](#linked-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #linked-term-id */}
The ID of the term linked to tag

#### [<code style={{ fontWeight: 'normal' }}>Tag.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

#### [<code style={{ fontWeight: 'normal' }}>Tag.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Timestamp of when the record was last updated

### Example

<TypeExample />

### Member Of

- [`GetTagsOutput`](/docs/catalog/api/general/types/objects/get-tags-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## TeamOwnerEntity

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A team owner entity represents a team that owns an entity (dashboard, table, knowledge).

```graphql
type TeamOwnerEntity {
  id: ID!
  ownerLabelId: String
  ownerLabel: OwnerLabel
  teamId: String!
  team: Team!
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.ownerLabelId</code>](#owner-label-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-label-id */}
The id of the owner label

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.ownerLabel</code>](#owner-label)<Bullet />[<code style={{ fontWeight: 'normal' }}>OwnerLabel</code>](/docs/catalog/api/general/types/objects/owner-label) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-label */}
The owner label

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.teamId</code>](#team-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-id */}
the owning team id

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.team</code>](#team)<Bullet />[<code style={{ fontWeight: 'normal' }}>Team!</code>](/docs/catalog/api/general/types/objects/team) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #team */}
The owning team

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>TeamOwnerEntity.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "ownerLabel": {},
  "team": {
    "teamOwnerEntities": []
  }
}
```

### Returned By

- [`upsertTeamOwners`](/docs/catalog/api/general/operations/mutations/upsert-team-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Team`](/docs/catalog/api/general/types/objects/team)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Team

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A team represents a group of users.

```graphql
type Team {
  id: ID!
  avatarUrl: String
  description: String
  email: String
  externalId: String
  name: String!
  slackChannel: String
  slackGroup: String
  source: String
  teamOwnerEntities: [TeamOwnerEntity!]!
  createdAt: Timestamp!
  updatedAt: Timestamp!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Team.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Team.avatarUrl</code>](#avatar-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #avatar-url */}
The avatar url of the team

#### [<code style={{ fontWeight: 'normal' }}>Team.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The description of the team

#### [<code style={{ fontWeight: 'normal' }}>Team.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the team

#### [<code style={{ fontWeight: 'normal' }}>Team.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
external id of the team when synchronized from an external source

#### [<code style={{ fontWeight: 'normal' }}>Team.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the team

#### [<code style={{ fontWeight: 'normal' }}>Team.slackChannel</code>](#slack-channel)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-channel */}
The slack channel of the team (start with #)

#### [<code style={{ fontWeight: 'normal' }}>Team.slackGroup</code>](#slack-group)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slack-group */}
The slack group of the team (start with @)

#### [<code style={{ fontWeight: 'normal' }}>Team.source</code>](#source)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #source */}
Indicate where the team come from: HR tool, Slack, Google groups...

#### [<code style={{ fontWeight: 'normal' }}>Team.teamOwnerEntities</code>](#team-owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TeamOwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/team-owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-owner-entities */}
The entities owned by the team

#### [<code style={{ fontWeight: 'normal' }}>Team.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Team.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

### Example

```graphql
{
  "teamOwnerEntities": [
    {
      "ownerLabel": {}
    }
  ]
}
```

### Returned By

- [`upsertTeam`](/docs/catalog/api/general/operations/mutations/upsert-team)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`TeamOwnerEntity`](/docs/catalog/api/general/types/objects/team-owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Term

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

Also known as Knowledge page, a Term is used to document your data.

```graphql
type Term {
  id: ID!
  deprecatedAt: Timestamp
  depthLevel: Int
  descriptionRaw: String
  descriptionStateLexical: JSON
  externalDescription: String
  externalDescriptionSource: TermExternalDescriptionSource
  externalId: String
  icon: String
  isDescriptionGenerated: Boolean
  lastEditedAt: Timestamp
  name: String
  slug: String!
  url: String
  verifiedAt: Timestamp
  childrenTerms: [Term!]!
  deprecatedByUserId: String
  entityEditors: [EntityEditor!]!
  lastDescribedByUserId: String
  lastDescribedBy: User
  linkedTag: Tag
  ownerEntities: [OwnerEntity!]!
  parentTermId: String
  parentTerm: Term
  tagEntities: [TagEntity!]!
  teamOwnerEntities: [TeamOwnerEntity!]!
  verifiedByUserId: String
  verifiedBy: User
  createdAt: Timestamp!
  deletedAt: Timestamp
  updatedAt: Timestamp!
  description: String!
  isDeprecated: Boolean!
  isVerified: Boolean!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>Term.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>Term.deprecatedAt</code>](#deprecated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-at */}
The time at which the term was deprecated

#### [<code style={{ fontWeight: 'normal' }}>Term.depthLevel</code>](#depth-level)<Bullet />[<code style={{ fontWeight: 'normal' }}>Int</code>](/docs/catalog/api/general/types/scalars/int) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #depth-level */}
The term depth - Default 0 is highest in hierarchy

#### [<code style={{ fontWeight: 'normal' }}>Term.descriptionRaw</code>](#description-raw)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-raw */}
The Catalog documentation for this term, as raw text

#### [<code style={{ fontWeight: 'normal' }}>Term.descriptionStateLexical</code>](#description-state-lexical)<Bullet />[<code style={{ fontWeight: 'normal' }}>JSON</code>](/docs/catalog/api/general/types/scalars/json) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description-state-lexical */}
Rich-text description of this term as lexical JSON

#### [<code style={{ fontWeight: 'normal' }}>Term.externalDescription</code>](#external-description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description */}
The documentation from the external source for this term

#### [<code style={{ fontWeight: 'normal' }}>Term.externalDescriptionSource</code>](#external-description-source)<Bullet />[<code style={{ fontWeight: 'normal' }}>TermExternalDescriptionSource</code>](/docs/catalog/api/general/types/enums/term-external-description-source) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-description-source */}
The source of the documentation for this term

#### [<code style={{ fontWeight: 'normal' }}>Term.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
External ID of the term when synchronized from a source

#### [<code style={{ fontWeight: 'normal' }}>Term.icon</code>](#icon)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #icon */}
The icon associated to the term

#### [<code style={{ fontWeight: 'normal' }}>Term.isDescriptionGenerated</code>](#is-description-generated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-description-generated */}
True if the description has been generated, false otherwise

#### [<code style={{ fontWeight: 'normal' }}>Term.lastEditedAt</code>](#last-edited-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-edited-at */}
The time at which the term was last edited

#### [<code style={{ fontWeight: 'normal' }}>Term.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The term name

#### [<code style={{ fontWeight: 'normal' }}>Term.slug</code>](#slug)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #slug */}
The unique slug of the term

#### [<code style={{ fontWeight: 'normal' }}>Term.url</code>](#url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #url */}
The term's url in the external source

#### [<code style={{ fontWeight: 'normal' }}>Term.verifiedAt</code>](#verified-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-at */}
The time at which the term was verified

#### [<code style={{ fontWeight: 'normal' }}>Term.childrenTerms</code>](#children-terms)<Bullet />[<code style={{ fontWeight: 'normal' }}>[Term!]!</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #children-terms */}
The children terms of this term

#### [<code style={{ fontWeight: 'normal' }}>Term.deprecatedByUserId</code>](#deprecated-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deprecated-by-user-id */}
The term was deprecated by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Term.entityEditors</code>](#entity-editors)<Bullet />[<code style={{ fontWeight: 'normal' }}>[EntityEditor!]!</code>](/docs/catalog/api/general/types/objects/entity-editor) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #entity-editors */}
The entity editors this term points to

#### [<code style={{ fontWeight: 'normal' }}>Term.lastDescribedByUserId</code>](#last-described-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-described-by-user-id */}
The term was last described be the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Term.lastDescribedBy</code>](#last-described-by)<Bullet />[<code style={{ fontWeight: 'normal' }}>User</code>](/docs/catalog/api/general/types/objects/user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-described-by */}
The term was last described by this user

#### [<code style={{ fontWeight: 'normal' }}>Term.linkedTag</code>](#linked-tag)<Bullet />[<code style={{ fontWeight: 'normal' }}>Tag</code>](/docs/catalog/api/general/types/objects/tag) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #linked-tag */}
The tag to which this term is linked

#### [<code style={{ fontWeight: 'normal' }}>Term.ownerEntities</code>](#owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[OwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-entities */}
The individual owners of the term

#### [<code style={{ fontWeight: 'normal' }}>Term.parentTermId</code>](#parent-term-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-term-id */}
The id of the parent term of this term

#### [<code style={{ fontWeight: 'normal' }}>Term.parentTerm</code>](#parent-term)<Bullet />[<code style={{ fontWeight: 'normal' }}>Term</code>](/docs/catalog/api/general/types/objects/term) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #parent-term */}
The parent term of this term

#### [<code style={{ fontWeight: 'normal' }}>Term.tagEntities</code>](#tag-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TagEntity!]!</code>](/docs/catalog/api/general/types/objects/tag-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #tag-entities */}
The tag entities belonging to this term

#### [<code style={{ fontWeight: 'normal' }}>Term.teamOwnerEntities</code>](#team-owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[TeamOwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/team-owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-owner-entities */}
The team owners of the term

#### [<code style={{ fontWeight: 'normal' }}>Term.verifiedByUserId</code>](#verified-by-user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by-user-id */}
The term is verified by the user belonging to this ID

#### [<code style={{ fontWeight: 'normal' }}>Term.verifiedBy</code>](#verified-by)<Bullet />[<code style={{ fontWeight: 'normal' }}>User</code>](/docs/catalog/api/general/types/objects/user) <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #verified-by */}
The term is verified by this user

#### [<code style={{ fontWeight: 'normal' }}>Term.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>Term.deletedAt</code>](#deleted-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #deleted-at */}
Date and time the resource was deleted

#### [<code style={{ fontWeight: 'normal' }}>Term.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

#### [<code style={{ fontWeight: 'normal' }}>Term.description</code>](#description)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #description */}
The term's raw description or the external one if empty

#### [<code style={{ fontWeight: 'normal' }}>Term.isDeprecated</code>](#is-deprecated)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-deprecated */}
Whether this term has been marked as deprecated

#### [<code style={{ fontWeight: 'normal' }}>Term.isVerified</code>](#is-verified)<Bullet />[<code style={{ fontWeight: 'normal' }}>Boolean!</code>](/docs/catalog/api/general/types/scalars/boolean) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #is-verified */}
Is the term verified?

### Example

```graphql
{
  "childrenTerms": [],
  "entityEditors": [
    {
      "sourceUser": {
        "unifiedUser": {
          "teamIds": []
        }
      }
    }
  ],
  "lastDescribedBy": {
    "ownerEntities": [
      {
        "ownerLabel": {}
      }
    ]
  },
  "linkedTag": {},
  "ownerEntities": [],
  "tagEntities": [
    {}
  ],
  "teamOwnerEntities": [
    {
      "team": {
        "teamOwnerEntities": []
      }
    }
  ]
}
```

### Returned By

- [`createTerm`](/docs/catalog/api/general/operations/mutations/create-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`updateTerm`](/docs/catalog/api/general/operations/mutations/update-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTermsOutput`](/docs/catalog/api/general/types/objects/get-terms-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## UnifiedUser

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

A unified user represents a user that is associated with a Catalog user and a source user. It is used to manage the user across the different systems.

```graphql
type UnifiedUser {
  id: String!
  avatarUrl: String
  email: String!
  name: String
  role: UserRole
  status: UserStatus
  teamIds: [String!]!
  userId: String
  createdAt: Timestamp
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.avatarUrl</code>](#avatar-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #avatar-url */}
The avatar URL of the unified user

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the unified user

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.name</code>](#name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #name */}
The name of the unified user

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.role</code>](#role)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserRole</code>](/docs/catalog/api/general/types/enums/user-role) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #role */}
The role of the unified user in the account

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserStatus</code>](/docs/catalog/api/general/types/enums/user-status) <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
The status of the unified user

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.teamIds</code>](#team-ids)<Bullet />[<code style={{ fontWeight: 'normal' }}>[String!]!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #team-ids */}
List of team id where the unified user is present

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.userId</code>](#user-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #user-id */}
The unified user is associated to this Catalog user ID

#### [<code style={{ fontWeight: 'normal' }}>UnifiedUser.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Timestamp of when the record was created

### Example

```graphql
{
  "teamIds": []
}
```

### Member Of

- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SourceUser`](/docs/catalog/api/general/types/objects/source-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## User

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The user entity that authenticate in the Catalog application

```graphql
type User {
  id: ID!
  avatarUrl: String
  email: String!
  externalId: String
  firstName: String!
  lastName: String!
  role: UserRole!
  status: UserStatus!
  unifiedId: String!
  ownerEntities: [OwnerEntity!]!
  unifiedUser: UnifiedUser!
  createdAt: Timestamp!
  updatedAt: Timestamp!
  fullName: String!
}
```

### Fields

#### [<code style={{ fontWeight: 'normal' }}>User.id</code>](#id)<Bullet />[<code style={{ fontWeight: 'normal' }}>ID!</code>](/docs/catalog/api/general/types/scalars/id) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #id */}
Unique Catalog identifier of the resource

#### [<code style={{ fontWeight: 'normal' }}>User.avatarUrl</code>](#avatar-url)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #avatar-url */}
The avatar url of the user

#### [<code style={{ fontWeight: 'normal' }}>User.email</code>](#email)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #email */}
The email of the user

#### [<code style={{ fontWeight: 'normal' }}>User.externalId</code>](#external-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #external-id */}
external id of the user when synchronized from an external source

#### [<code style={{ fontWeight: 'normal' }}>User.firstName</code>](#first-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #first-name */}
The first name of the user

#### [<code style={{ fontWeight: 'normal' }}>User.lastName</code>](#last-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #last-name */}
The last name of the user

#### [<code style={{ fontWeight: 'normal' }}>User.role</code>](#role)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserRole!</code>](/docs/catalog/api/general/types/enums/user-role) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #role */}
The role of the user in the account

#### [<code style={{ fontWeight: 'normal' }}>User.status</code>](#status)<Bullet />[<code style={{ fontWeight: 'normal' }}>UserStatus!</code>](/docs/catalog/api/general/types/enums/user-status) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="enum"/> <Badge class="badge badge--secondary " text="general"/> {/* #status */}
The status of the user in the account

#### [<code style={{ fontWeight: 'normal' }}>User.unifiedId</code>](#unified-id)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #unified-id */}
The precomputed unifiedId

#### [<code style={{ fontWeight: 'normal' }}>User.ownerEntities</code>](#owner-entities)<Bullet />[<code style={{ fontWeight: 'normal' }}>[OwnerEntity!]!</code>](/docs/catalog/api/general/types/objects/owner-entity) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #owner-entities */}
The entities owned by the user

#### [<code style={{ fontWeight: 'normal' }}>User.unifiedUser</code>](#unified-user)<Bullet />[<code style={{ fontWeight: 'normal' }}>UnifiedUser!</code>](/docs/catalog/api/general/types/objects/unified-user) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="object"/> <Badge class="badge badge--secondary " text="general"/> {/* #unified-user */}
The unified user linked to that user.

#### [<code style={{ fontWeight: 'normal' }}>User.createdAt</code>](#created-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #created-at */}
Date and time the resource was created

#### [<code style={{ fontWeight: 'normal' }}>User.updatedAt</code>](#updated-at)<Bullet />[<code style={{ fontWeight: 'normal' }}>Timestamp!</code>](/docs/catalog/api/general/types/scalars/timestamp) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #updated-at */}
Date and time the resource was last updated

#### [<code style={{ fontWeight: 'normal' }}>User.fullName</code>](#full-name)<Bullet />[<code style={{ fontWeight: 'normal' }}>String!</code>](/docs/catalog/api/general/types/scalars/string) <Badge class="badge badge--secondary badge--non_null" text="non-null"/> <Badge class="badge badge--secondary " text="scalar"/> <Badge class="badge badge--secondary " text="general"/> {/* #full-name */}
The full name of the user

### Example

```graphql
{
  "ownerEntities": [
    {
      "ownerLabel": {},
      "unifiedUser": {
        "teamIds": []
      }
    }
  ]
}
```

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Boolean

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `Boolean` scalar type represents `true` or `false`.

```graphql
scalar Boolean
```

### Returned By

- [`addTeamUsers`](/docs/catalog/api/general/operations/mutations/add-team-users)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`attachTags`](/docs/catalog/api/general/operations/mutations/attach-tags)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`deleteExternalLinks`](/docs/catalog/api/general/operations/mutations/delete-external-links)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`deleteLineages`](/docs/catalog/api/general/operations/mutations/delete-lineages)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`deleteTerm`](/docs/catalog/api/general/operations/mutations/delete-term)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`detachTags`](/docs/catalog/api/general/operations/mutations/detach-tags)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removeDataQualities`](/docs/catalog/api/general/operations/mutations/remove-data-qualities)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removePinnedAssets`](/docs/catalog/api/general/operations/mutations/remove-pinned-assets)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removeTeamOwners`](/docs/catalog/api/general/operations/mutations/remove-team-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removeTeamUsers`](/docs/catalog/api/general/operations/mutations/remove-team-users)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>
- [`removeUserOwners`](/docs/catalog/api/general/operations/mutations/remove-user-owners)  <Badge class="badge badge--secondary badge--relation" text="mutation"/>

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`CreateColumnInput`](/docs/catalog/api/general/types/inputs/create-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateDatabaseInput`](/docs/catalog/api/general/types/inputs/create-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateSchemaInput`](/docs/catalog/api/general/types/inputs/create-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Database`](/docs/catalog/api/general/types/objects/database)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnJoinsScope`](/docs/catalog/api/general/types/inputs/get-column-joins-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetColumnsScope`](/docs/catalog/api/general/types/inputs/get-columns-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDashboardsScope`](/docs/catalog/api/general/types/inputs/get-dashboards-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDatabasesScope`](/docs/catalog/api/general/types/inputs/get-databases-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetFieldLineagesScope`](/docs/catalog/api/general/types/inputs/get-field-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetLineagesScope`](/docs/catalog/api/general/types/inputs/get-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetQualityChecksScope`](/docs/catalog/api/general/types/inputs/get-quality-checks-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetSchemasScope`](/docs/catalog/api/general/types/inputs/get-schemas-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetSourcesScope`](/docs/catalog/api/general/types/inputs/get-sources-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetTablesScope`](/docs/catalog/api/general/types/inputs/get-tables-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetUsersOutput`](/docs/catalog/api/general/types/objects/get-users-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Schema`](/docs/catalog/api/general/types/objects/schema)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SearchQueriesInput`](/docs/catalog/api/general/types/inputs/search-queries-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UpdateColumnInput`](/docs/catalog/api/general/types/inputs/update-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateColumnsMetadataInput`](/docs/catalog/api/general/types/inputs/update-columns-metadata-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateDatabaseInput`](/docs/catalog/api/general/types/inputs/update-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateSchemaInput`](/docs/catalog/api/general/types/inputs/update-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## DateTime

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The javascript `Date` as string. Type represents date and time as the ISO Date string.

```graphql
scalar DateTime
```

### Member Of

- [`BaseQualityCheckInput`](/docs/catalog/api/general/types/inputs/base-quality-check-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateColumnInput`](/docs/catalog/api/general/types/inputs/create-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateDatabaseInput`](/docs/catalog/api/general/types/inputs/create-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateSchemaInput`](/docs/catalog/api/general/types/inputs/create-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateTableInput`](/docs/catalog/api/general/types/inputs/create-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateColumnInput`](/docs/catalog/api/general/types/inputs/update-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateDatabaseInput`](/docs/catalog/api/general/types/inputs/update-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateSchemaInput`](/docs/catalog/api/general/types/inputs/update-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateSourceInput`](/docs/catalog/api/general/types/inputs/update-source-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateTableInput`](/docs/catalog/api/general/types/inputs/update-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## Float

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `Float` scalar type represents signed double-precision fractional values as specified by [IEEE 754](https://en.wikipedia.org/wiki/IEEE&#x005F;floating&#x005F;point).

```graphql
scalar Float
```

### Member Of

- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## ID

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `ID` scalar type represents a unique identifier, often used to refetch an object or as key for a cache. The ID type appears in a JSON response as a String; however, it is not intended to be human-readable. When expected as an input type, any string (such as `"4"`) or integer (such as `4`) input value will be accepted as an ID.

```graphql
scalar ID
```

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ColumnJoin`](/docs/catalog/api/general/types/objects/column-join)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Database`](/docs/catalog/api/general/types/objects/database)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntityEditor`](/docs/catalog/api/general/types/objects/entity-editor)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ExternalLink`](/docs/catalog/api/general/types/objects/external-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`FieldLineage`](/docs/catalog/api/general/types/objects/field-lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Lineage`](/docs/catalog/api/general/types/objects/lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerLabel`](/docs/catalog/api/general/types/objects/owner-label)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Schema`](/docs/catalog/api/general/types/objects/schema)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SourceUser`](/docs/catalog/api/general/types/objects/source-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Tag`](/docs/catalog/api/general/types/objects/tag)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Team`](/docs/catalog/api/general/types/objects/team)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TeamOwnerEntity`](/docs/catalog/api/general/types/objects/team-owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Int

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `Int` scalar type represents non-fractional signed whole numeric values. Int can represent values between -(2^31) and 2^31 - 1.

```graphql
scalar Int
```

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ColumnJoin`](/docs/catalog/api/general/types/objects/column-join)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`CreateColumnInput`](/docs/catalog/api/general/types/inputs/create-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateTableInput`](/docs/catalog/api/general/types/inputs/create-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnJoinsOutput`](/docs/catalog/api/general/types/objects/get-column-joins-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnsOutput`](/docs/catalog/api/general/types/objects/get-columns-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDashboardsOutput`](/docs/catalog/api/general/types/objects/get-dashboards-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDatabasesOutput`](/docs/catalog/api/general/types/objects/get-databases-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDataProductOutput`](/docs/catalog/api/general/types/objects/get-data-product-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetEntitiesLinkOutput`](/docs/catalog/api/general/types/objects/get-entities-link-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetFieldLineagesOutput`](/docs/catalog/api/general/types/objects/get-field-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetLineagesOutput`](/docs/catalog/api/general/types/objects/get-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetQualityChecksOutput`](/docs/catalog/api/general/types/objects/get-quality-checks-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetSchemasOutput`](/docs/catalog/api/general/types/objects/get-schemas-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetSourcesOutput`](/docs/catalog/api/general/types/objects/get-sources-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTableQueriesOutput`](/docs/catalog/api/general/types/objects/get-table-queries-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTablesOutput`](/docs/catalog/api/general/types/objects/get-tables-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTagsOutput`](/docs/catalog/api/general/types/objects/get-tags-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTermsOutput`](/docs/catalog/api/general/types/objects/get-terms-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`JobResultInput`](/docs/catalog/api/general/types/inputs/job-result-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`PublicPagination`](/docs/catalog/api/general/types/inputs/public-pagination)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UpdateColumnInput`](/docs/catalog/api/general/types/inputs/update-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateTableInput`](/docs/catalog/api/general/types/inputs/update-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>

---

## JSON

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `JSON` scalar type represents JSON values as specified by [ECMA-404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf).

```graphql
scalar JSON
```

### <SpecifiedBy url="http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf"/>

### Member Of

- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## String

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The `String` scalar type represents textual data, represented as UTF-8 character sequences. The String type is most often used by GraphQL to represent free-form human-readable text.

```graphql
scalar String
```

### Member Of

- [`AddAiAssistantJobInput`](/docs/catalog/api/general/types/inputs/add-ai-assistant-job-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`AiAssistantJobResult`](/docs/catalog/api/general/types/objects/ai-assistant-job-result)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`AssetWithInternalLink`](/docs/catalog/api/general/types/objects/asset-with-internal-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`BaseQualityCheckInput`](/docs/catalog/api/general/types/inputs/base-quality-check-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`BaseTagEntityInput`](/docs/catalog/api/general/types/inputs/base-tag-entity-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ColumnJoin`](/docs/catalog/api/general/types/objects/column-join)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ConverseWithAssistantOutput`](/docs/catalog/api/general/types/objects/converse-with-assistant-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`CreateColumnInput`](/docs/catalog/api/general/types/inputs/create-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateDatabaseInput`](/docs/catalog/api/general/types/inputs/create-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateExternalLinkInput`](/docs/catalog/api/general/types/inputs/create-external-link-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateSchemaInput`](/docs/catalog/api/general/types/inputs/create-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateSourceInput`](/docs/catalog/api/general/types/inputs/create-source-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateTableInput`](/docs/catalog/api/general/types/inputs/create-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`CreateTermInput`](/docs/catalog/api/general/types/inputs/create-term-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Database`](/docs/catalog/api/general/types/objects/database)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DeleteExternalLinkInput`](/docs/catalog/api/general/types/inputs/delete-external-link-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DeleteLineageInput`](/docs/catalog/api/general/types/inputs/delete-lineage-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`DeleteTermInput`](/docs/catalog/api/general/types/inputs/delete-term-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLinkTargetInput`](/docs/catalog/api/general/types/inputs/entities-link-target-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`EntityEditor`](/docs/catalog/api/general/types/objects/entity-editor)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntityTarget`](/docs/catalog/api/general/types/inputs/entity-target)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`ExternalLink`](/docs/catalog/api/general/types/objects/external-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`FieldLineage`](/docs/catalog/api/general/types/objects/field-lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnJoinsOutput`](/docs/catalog/api/general/types/objects/get-column-joins-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnJoinsScope`](/docs/catalog/api/general/types/inputs/get-column-joins-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetColumnsOutput`](/docs/catalog/api/general/types/objects/get-columns-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetColumnsScope`](/docs/catalog/api/general/types/inputs/get-columns-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDashboardsOutput`](/docs/catalog/api/general/types/objects/get-dashboards-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDashboardsScope`](/docs/catalog/api/general/types/inputs/get-dashboards-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDatabasesOutput`](/docs/catalog/api/general/types/objects/get-databases-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDatabasesScope`](/docs/catalog/api/general/types/inputs/get-databases-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetDataProductOutput`](/docs/catalog/api/general/types/objects/get-data-product-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetDataProductScope`](/docs/catalog/api/general/types/inputs/get-data-product-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetEntitiesLinkOutput`](/docs/catalog/api/general/types/objects/get-entities-link-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetEntitiesLinksScope`](/docs/catalog/api/general/types/inputs/get-entities-links-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetFieldLineagesOutput`](/docs/catalog/api/general/types/objects/get-field-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetFieldLineagesScope`](/docs/catalog/api/general/types/inputs/get-field-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetLineagesOutput`](/docs/catalog/api/general/types/objects/get-lineages-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetLineagesScope`](/docs/catalog/api/general/types/inputs/get-lineages-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetQualityChecksOutput`](/docs/catalog/api/general/types/objects/get-quality-checks-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetQualityChecksScope`](/docs/catalog/api/general/types/inputs/get-quality-checks-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetSchemasOutput`](/docs/catalog/api/general/types/objects/get-schemas-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetSchemasScope`](/docs/catalog/api/general/types/inputs/get-schemas-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetSourcesOutput`](/docs/catalog/api/general/types/objects/get-sources-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetSourcesScope`](/docs/catalog/api/general/types/inputs/get-sources-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetTableQueriesOutput`](/docs/catalog/api/general/types/objects/get-table-queries-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTableQueriesScope`](/docs/catalog/api/general/types/inputs/get-table-queries-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetTablesOutput`](/docs/catalog/api/general/types/objects/get-tables-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTablesScope`](/docs/catalog/api/general/types/inputs/get-tables-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetTagsOutput`](/docs/catalog/api/general/types/objects/get-tags-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTagsScope`](/docs/catalog/api/general/types/inputs/get-tags-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetTeamsOutput`](/docs/catalog/api/general/types/objects/get-teams-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTermsOutput`](/docs/catalog/api/general/types/objects/get-terms-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTermsScope`](/docs/catalog/api/general/types/inputs/get-terms-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`GetUsersOutput`](/docs/catalog/api/general/types/objects/get-users-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`JobResultInput`](/docs/catalog/api/general/types/inputs/job-result-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Lineage`](/docs/catalog/api/general/types/objects/lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerInput`](/docs/catalog/api/general/types/inputs/owner-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`OwnerLabel`](/docs/catalog/api/general/types/objects/owner-label)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`PublicPagination`](/docs/catalog/api/general/types/inputs/public-pagination)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`QualityCheckInput`](/docs/catalog/api/general/types/inputs/quality-check-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Schema`](/docs/catalog/api/general/types/objects/schema)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SearchQueriesInput`](/docs/catalog/api/general/types/inputs/search-queries-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`SearchQueriesScope`](/docs/catalog/api/general/types/inputs/search-queries-scope)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`SearchQueryResult`](/docs/catalog/api/general/types/objects/search-query-result)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SearchQueryResultAuthor`](/docs/catalog/api/general/types/objects/search-query-result-author)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SourceUser`](/docs/catalog/api/general/types/objects/source-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableInQueryOutput`](/docs/catalog/api/general/types/objects/table-in-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Tag`](/docs/catalog/api/general/types/objects/tag)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Team`](/docs/catalog/api/general/types/objects/team)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TeamOwnerEntity`](/docs/catalog/api/general/types/objects/team-owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TeamOwnerInput`](/docs/catalog/api/general/types/inputs/team-owner-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`TeamUsersInput`](/docs/catalog/api/general/types/inputs/team-users-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UnifiedUser`](/docs/catalog/api/general/types/objects/unified-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UpdateColumnDescriptionInput`](/docs/catalog/api/general/types/inputs/update-column-description-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateColumnInput`](/docs/catalog/api/general/types/inputs/update-column-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateColumnsMetadataInput`](/docs/catalog/api/general/types/inputs/update-columns-metadata-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateDatabaseInput`](/docs/catalog/api/general/types/inputs/update-database-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateExternalLinkInput`](/docs/catalog/api/general/types/inputs/update-external-link-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateSchemaInput`](/docs/catalog/api/general/types/inputs/update-schema-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateSourceInput`](/docs/catalog/api/general/types/inputs/update-source-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateTableDescriptionInput`](/docs/catalog/api/general/types/inputs/update-table-description-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateTableInput`](/docs/catalog/api/general/types/inputs/update-table-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpdateTermInput`](/docs/catalog/api/general/types/inputs/update-term-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpsertLineageInput`](/docs/catalog/api/general/types/inputs/upsert-lineage-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpsertQualityChecksInput`](/docs/catalog/api/general/types/inputs/upsert-quality-checks-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`UpsertTeamInput`](/docs/catalog/api/general/types/inputs/upsert-team-input)  <Badge class="badge badge--secondary badge--relation" text="input"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## Timestamp

export const Bullet = () => <>&nbsp;●&nbsp;</>

export const SpecifiedBy = (props) => <>Specification⎘</>

export const Badge = (props) => <>{props.text}</>

The javascript `Date` as integer. Type represents date and time as number of milliseconds from start of UNIX epoch.

```graphql
scalar Timestamp
```

### Member Of

- [`Column`](/docs/catalog/api/general/types/objects/column)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`ColumnJoin`](/docs/catalog/api/general/types/objects/column-join)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Dashboard`](/docs/catalog/api/general/types/objects/dashboard)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DashboardField`](/docs/catalog/api/general/types/objects/dashboard-field)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Database`](/docs/catalog/api/general/types/objects/database)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`DataProduct`](/docs/catalog/api/general/types/objects/data-product)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntitiesLink`](/docs/catalog/api/general/types/objects/entities-link)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`EntityEditor`](/docs/catalog/api/general/types/objects/entity-editor)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`FieldLineage`](/docs/catalog/api/general/types/objects/field-lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetTeamsOutput`](/docs/catalog/api/general/types/objects/get-teams-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`GetUsersOutput`](/docs/catalog/api/general/types/objects/get-users-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Lineage`](/docs/catalog/api/general/types/objects/lineage)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`OwnerEntity`](/docs/catalog/api/general/types/objects/owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`QualityCheck`](/docs/catalog/api/general/types/objects/quality-check)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Schema`](/docs/catalog/api/general/types/objects/schema)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Source`](/docs/catalog/api/general/types/objects/source)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`SourceUser`](/docs/catalog/api/general/types/objects/source-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Table`](/docs/catalog/api/general/types/objects/table)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TableQueryOutput`](/docs/catalog/api/general/types/objects/table-query-output)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Tag`](/docs/catalog/api/general/types/objects/tag)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TagEntity`](/docs/catalog/api/general/types/objects/tag-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Team`](/docs/catalog/api/general/types/objects/team)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`TeamOwnerEntity`](/docs/catalog/api/general/types/objects/team-owner-entity)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`Term`](/docs/catalog/api/general/types/objects/term)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`UnifiedUser`](/docs/catalog/api/general/types/objects/unified-user)  <Badge class="badge badge--secondary badge--relation" text="object"/>
- [`User`](/docs/catalog/api/general/types/objects/user)  <Badge class="badge badge--secondary badge--relation" text="object"/>

---

## API Responses and Error Codes

Every Catalog Public GraphQL API call returns an HTTP status and a JSON body. A successful read or write includes a `data` object. When something fails, details usually appear in an `errors` array, even when the HTTP status is `200`. This guide explains how to read both layers and how to fix common problems.

Use it alongside the [Catalog GraphQL API][], [Using the GraphQL Reference][], and [Getting Your Catalog API Keys][] when you debug integrations, scripts, or live requests from the GraphQL reference.

## Quick Reference: HTTP Status Codes

Use this table to match an HTTP status to the likely cause and response body shape.

| HTTP status | Typical cause | Body shape |
| ----------- | --------------- | ---------- |
| `200` | Request reached GraphQL. Includes successful calls and many failed calls, such as invalid token, permission denied, or business validation. | `{ "data": ... }` and/or `{ "errors": [...] }` |
| `400` | Malformed JSON, invalid GraphQL document with unknown fields, or empty body with `Content-Type: application/json` | GraphQL `errors` and/or Fastify JSON error object |
| `404` | Wrong URL path, or request blocked before GraphQL, for example when `Authorization` is missing on some deployments | HTML or JSON `Not Found` message |
| `405` | HTTP method not allowed; use `POST` | Often empty or minimal |
| `415` | Missing or unsupported `Content-Type`; send `application/json` | Fastify JSON error object |
| `500` | GraphQL syntax error, missing `query` field, or other failure while building the request context | GraphQL `errors`, often `INTERNAL_SERVER_ERROR` |

## Quick Reference: Error Codes

Look for `extensions.code` on each entry in `errors`:

| Code | Usually means |
| ---- | ------------- |
| `UNAUTHENTICATED` | Missing, invalid, expired, or revoked API token |
| `FORBIDDEN` | Token is valid but not allowed to run this operation, for example a Read Only token on a mutation |
| `ACCESS_DENIED` | Account or feature access blocks the operation |
| `BAD_USER_INPUT` | Variables or mutation input failed validation |
| `NOT_FOUND` | Referenced entity does not exist in your Catalog account |
| `GRAPHQL_VALIDATION_FAILED` | Query text does not match the schema, such as an unknown field or argument |
| `INTERNAL_SERVER_ERROR` | Query syntax error, malformed request body fields, or unexpected server failure |
| `ERROR_RESULT` | Business rule or rate limit; see `extensions.payload` for more detail |

## How to Read a Response

Catalog follows standard GraphQL response shape:

- **`data`** - Fields you asked for in the query selection set. `null` at the operation root usually means the call failed.
- **`errors`** - One or more problems: auth, validation, permissions, or server errors. Check `message` and `extensions.code`.

Treat **HTTP status** and **GraphQL errors** as two layers:

1. **HTTP status** tells you whether the request reached the API and passed basic transport checks: method, JSON body, and content type.
2. **`errors`** in the body tells you whether GraphQL execution succeeded. Auth and permission failures often return **HTTP `200`** with `errors`, not HTTP `401`.

:::tip[Success check]

A call succeeded when HTTP status is `200`, `errors` is absent or empty, and `data` contains the operation you requested, for example `data.getTables`.

:::

### Example: Successful Query

```json
{
  "data": {
    "getTables": {
      "totalCount": 42,
      "data": [{ "id": "abc123", "name": "FCT_ORDERS" }]
    }
  }
}
```

### Example: Invalid API Token

HTTP status is `200`, but the body reports an authentication error:

```json
{
  "errors": [{
    "message": "This API Token is not valid",
    "path": ["getTables"],
    "extensions": {
      "code": "UNAUTHENTICATED"
    }
  }],
  "data": null
}
```

## HTTP Status Details

The following sections expand each HTTP status with common causes and what to check next.

### `200 OK`

`200` means the HTTP request was accepted and processed by the GraphQL endpoint. It does not guarantee success.

| Situation | What to check |
| --------- | ------------- |
| Call worked | `errors` is missing or empty, and `data.<operationName>` has the fields you selected |
| Invalid token | `errors[].extensions.code` is `UNAUTHENTICATED` |
| Wrong token scope | `errors[].extensions.code` is `FORBIDDEN` |
| Bad variables or input | `errors[].extensions.code` is `BAD_USER_INPUT` |

Always inspect `errors` before treating `200` as success.

### `400 Bad Request`

The server rejected the request before or during GraphQL validation.

| Common cause | What to do |
| ------------ | ---------- |
| Malformed JSON in the POST body | Validate JSON. Ensure the body is `{ "query": "...", "variables": { ... } }`. |
| Unknown field or argument in the query | Compare your query to the operation page in the [API reference][]. Remove or rename fields that are not listed under **Response**. |
| Empty body with `Content-Type: application/json` | Send a non-empty JSON object with a `query` string. |

Fastify may return a plain JSON error instead of GraphQL `errors`, for example:

```json
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Body is not valid JSON but content-type is set to 'application/json'"
}
```

### `404 Not Found`

`404` usually means the request never reached the GraphQL handler.

| Common cause | What to do |
| ------------ | ---------- |
| Wrong host or path | Use `https://api.castordoc.com/public/graphql` for EU or `https://api.us.castordoc.com/public/graphql` for US. See [Base URLs][]. |
| Missing `Authorization` header | Send `Authorization: Token <API_TOKEN>` on every request. See [Authentication][]. |
| Wrong `?op=` value | Match `?op=` to the root field name, for example `getTables`. |

### `405 Method Not Allowed`

Send **`POST`** only. GraphQL query strings in a `GET` URL are not supported on the Public API.

### `415 Unsupported Media Type`

Set both headers on every request:

```text
Content-Type: application/json
Accept: application/json
```

### `500 Internal Server Error`

Often appears when the GraphQL query text cannot be parsed, for example an empty `query`, invalid syntax, or a missing `query` key in the JSON body. The `message` may start with `Context creation failed:` followed by a syntax detail.

| What to do |
| ---------- |
| Paste the query from the operation **Example** block and edit incrementally. |
| Confirm the POST body includes `"query": "..."` as a JSON string. |
| See [Introduction to GraphQL][] to validate operation syntax. |

## GraphQL Error Codes

Each entry in `errors` includes `message` and `extensions.code`. The following sections explain the codes you see most often on the Public API.

### `UNAUTHENTICATED`

Catalog could not authenticate your API token for this request.

You may see this message:

- `This API Token is not valid`

Likely causes include:

- Token value is wrong or truncated
- Token was rotated or revoked in **Settings > API**
- Token expired
- `Authorization` header missing the `Token` prefix; use `Token <value>`, not `Bearer`
- Trial account past its end date

To fix the error:

1. Open **Settings > API** and confirm the token is still active and not expired.
2. Copy a fresh token if needed. See [Getting Your Catalog API Keys][] for rotation steps.
3. Set the header exactly as `Authorization: Token <API_TOKEN>`.
4. Confirm your integration uses the correct region, EU versus US. See [Base URLs][].
5. Update every script, importer, and automation after rotation. Creating a new token for a scope revokes the previous one immediately.

:::info[MCP and GraphQL differ]

The [MCP Integration][] can return HTTP `401` for bad tokens. The Public GraphQL API returns HTTP `200` with `UNAUTHENTICATED` in the body instead. Use the same token format, but check `errors` for GraphQL calls.

:::

### `FORBIDDEN`

Your token authenticated, but it does not have permission for this operation.

You may see this message:

- `You do not have the necessary authorization. Min access READ_WRITE`

Likely causes include:

- **Read Only** token used for a **mutation**, such as create, update, or delete
- Operation requires a higher access level than the token provides

To fix the error:

1. Check the operation badge in the reference: green **mut** operations need a **Read & Write** token.
2. Create or switch to a **Read & Write** token in **Settings > API** if your workflow writes metadata.
3. Keep **Read Only** tokens for read-only automation and MCP search use cases.

### `ACCESS_DENIED`

Your account does not have access to the requested feature or resource class.

You may see this message:

- `You do not have the necessary access permission. Please contact your administrators.`

To fix the error:

1. Confirm the capability is enabled for your Catalog account; some write paths require API beta access.
2. Ask your Catalog administrator or [Coalesce Support][] to verify account settings and entitlements.
3. If you recently changed plans or regions, confirm you are calling the correct API host.

### `BAD_USER_INPUT`

The query reached Catalog, but variables or mutation input failed validation.

Likely causes include:

- Required variable omitted for types marked with `!` in the schema
- Wrong JSON type, such as string instead of object, or array shape mismatch
- `nbPerPage` above the maximum of 500
- `page` and `greaterThanId` both set on the same pagination request
- Duplicate IDs in a batch mutation payload
- ID in the payload does not exist in your account

To fix the error:

1. Compare **Variables** JSON to the operation **Arguments** section in the reference.
2. Fix types to match the schema: objects for input types, arrays for `[Type]`.
3. For pagination, use either offset with `page` or cursor with `greaterThanId`, not both. See [Catalog GraphQL API][] for pagination limits.
4. For batch mutations, ensure every referenced `id` exists and keys are unique within the payload.

### `NOT_FOUND`

A specific entity referenced in the request does not exist or is not visible in your Catalog account.

To fix the error:

1. Re-run a getter query such as `getTables` or `getColumns` to confirm the ID or slug.
2. Confirm you are on the correct region and account; EU and US hosts are separate.
3. Check whether the asset was deleted or never ingested.

### `GRAPHQL_VALIDATION_FAILED`

The query text does not match the published schema.

You may see messages such as:

- `Cannot query field "..." on type "Query".`
- `Unknown argument "..." on field "...".`

To fix the error:

1. Open the operation page in the [API reference][] and copy the **Example** query.
2. Request only fields listed under **Response** for that operation.
3. Spell operation and argument names exactly as documented; names are case-sensitive.

This error often returns HTTP `400` instead of `200`.

### `INTERNAL_SERVER_ERROR`

The server failed while handling the request. The `message` field has the actionable detail.

Use this table to match common message patterns to fixes:

| Message pattern | Likely cause | Fix |
| --------------- | ------------ | --- |
| `Context creation failed: Syntax Error: ...` | Invalid GraphQL syntax, empty query, or missing `query` in the POST body | Fix query text; send a valid `query` string in JSON |
| Generic `Internal Server Error` in `extensions.exception` | Unexpected failure | Retry once; if it persists, contact [Coalesce Support][] with the operation name, `?op=` value, and timestamp |

Syntax and context errors may return HTTP `500`. Retry after fixing the query; do not assume the API is down until transport errors persist.

### `ERROR_RESULT`

The operation ran, but a business rule blocked the result. Read `extensions.payload` when present, especially `payload.code`.

| `payload.code` when present | Meaning | What to do |
| --------------------------- | ------- | ---------- |
| `RateLimitedError` | Too many requests in a short window | Back off and retry with exponential delay; reduce call frequency |
| `DuplicatedEmailsError` | Duplicate emails in a team or user payload | Deduplicate the `emails` array |
| `EmailsNotFoundError` | Email addresses are not Catalog users | Verify users exist in the account before adding them to a team |
| `AlreadyExistingUsersError` | Users are already on the team | Skip existing members or use a different mutation |
| `CircularTermHierarchy` | Term parent/child relationship would loop | Fix parent/child IDs in knowledge hierarchy mutations |
| `BulkApplyAllAlreadyRunningError` | Another bulk job is already in progress | Wait for the current job to finish, then retry |

If `payload.code` is missing, use the top-level `message` string and the operation documentation to adjust your input.

## Request Checklist

Before opening a support ticket, verify:

1. **Method:** `POST` to `/public/graphql?op=<operationName>`
2. **Headers:** `Content-Type: application/json`, `Accept: application/json`, `Authorization: Token <API_TOKEN>`
3. **Region:** EU host `api.castordoc.com` or US host `api.us.castordoc.com` matches your Catalog app URL
4. **Body:** Valid JSON with `query` and optional `variables`
5. **Success rule:** HTTP `200` with no `errors` and populated `data.<operationName>`

## What's Next?

- [Catalog GraphQL API][] - Authentication, regional base URLs, and operation index
- [Using the GraphQL Reference][] - Send requests from the reference and read the response panel
- [Getting Your Catalog API Keys][] - Create tokens, scopes, and rotation
- [Introduction to GraphQL][] - Query syntax, variables, and selection sets

[API reference]: /docs/catalog/api
[Authentication]: /docs/catalog/api#authentication
[Base URLs]: /docs/catalog/api#base-urls
[Catalog GraphQL API]: /docs/catalog/api
[Coalesce Support]: mailto:support@coalesce.io
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Introduction to GraphQL]: /docs/catalog/api/graphql
[MCP Integration]: /docs/catalog/developer/mcp-integration
[Using the GraphQL Reference]: /docs/catalog/api/using-the-graphql-reference

---

## Introduction to GraphQL

This guide explains how to read and build Catalog GraphQL operations using the generated API reference. By the end you should be able to:

1. Read a GraphQL query or mutation and its **Variables** block together.
2. Match each `$variable` in an operation to a key in the **Variables** JSON.
3. Tell a query from a mutation, and know when each is required.
4. Build a request that reads data (`getTables`), one that creates data (`createColumns`), and one that removes data (`removeTeamUsers`).
5. Connect a response JSON object back to the fields you selected.
6. Find argument and field details for any operation in the **Types** reference.

## What Is GraphQL?

GraphQL is a text format for asking an API for data, or for asking it to make a change. You write a short operation, list the fields you want back, and the server returns JSON shaped like your request. For background outside Catalog, see the [GraphQL.org introduction][].

## Query Versus Mutation

Every Catalog operation starts with one of two keywords:

| Keyword | Use in Catalog | Example |
|---------|----------------|---------|
| `query` | Read metadata. Nothing changes. | `query GetTables` |
| `mutation` | Create, update, or remove metadata. | `mutation CreateColumns`, `mutation RemoveTeamUsers` |

This guide walks through one of each: `getTables` to read, `createColumns` to create, and `removeTeamUsers` to remove. The three together cover the shapes you'll see across the rest of the reference.

## Reading a Query: Get Tables

The [Get tables][] operation lists tables in Catalog. It's a `query`, so it only reads data, and nothing in your Catalog instance changes when you run it.

### The Operation Signature

Every operation page shows a signature like this one for `getTables`:

```graphql
getTables(
  pagination: PublicPagination
  sorting: [TableSorting!]
  scope: GetTablesScope
): GetTablesOutput!
```

Read this the way you'd read a function definition:

| Piece | What it means |
|-------|----------------|
| `getTables` | The operation name. Spell it exactly as documented. It also identifies the operation when you send the request over HTTP (see [Send the Request with cURL](#send-the-request-with-curl) below). |
| `(pagination: ..., sorting: ..., scope: ...)` | The arguments this operation accepts, with their types. All three are optional here, since none has a trailing `!`. |
| `: GetTablesOutput!` | The shape of the data you get back. The `!` means `getTables` always returns a `GetTablesOutput` object, never `null`. |

### Writing the Query

To actually call `getTables`, wrap it in a named `query` operation and declare the variables you intend to use:

```graphql showLineNumbers
query GetTables($pagination: PublicPagination) {
  getTables(pagination: $pagination) {
    totalCount
    data {
      id
      name
    }
  }
}
```

Breaking down line 1:

| Piece | Name | What it means |
|-------|------|----------------|
| `query` | Operation type | This request only reads data. |
| `GetTables` | Operation name | A label you choose. Naming it after the root field (`getTables`) keeps things easy to trace, but it isn't required to match. |
| `($pagination: PublicPagination)` | Variable declaration | This operation accepts one variable, `$pagination`, typed as `PublicPagination`. The `$` marks it as a placeholder filled in from **Variables** JSON. |

Line 2 is the root field, the actual operation you're calling, matched up with the argument you declared:

| Piece | What it means |
|-------|----------------|
| `getTables` | Must match the operation name in the schema exactly. |
| `(pagination: $pagination)` | Passes your declared variable into the operation's `pagination` argument. The left-hand `pagination` is the argument name from the signature; the right-hand `$pagination` is your variable. |

Everything inside the outer `{ }` is the selection set: the fields you want back.

- `totalCount` returns the total number of matching tables across all pages.
- `data { id name }` returns the table records on this page, with only `id` and `name` included on each row.

GraphQL only returns fields you ask for. Add `description` inside `data { }` and it appears in the response when present; leave it out and it doesn't, even though Catalog stores it. Each row under `data` is a [Table][]. See that type's page for other optional fields such as `slug` or `schema`.

### Returning More Fields

The selection set isn't fixed. It's the main way you control the size and shape of the response. Adding a field to the query adds it to the response; nothing else changes.

Start with the minimal version from above, asking for only `id` and `name`:

```graphql showLineNumbers
query GetTables($pagination: PublicPagination) {
  getTables(pagination: $pagination) {
    data {
      id
      name
    }
  }
}
```

```json
{
  "data": {
    "getTables": {
      "data": [
        { "id": "abc123", "name": "FCT_ORDERS" },
        { "id": "def456", "name": "DIM_CUSTOMERS" }
      ]
    }
  }
}
```

Each table object in the response has exactly two keys, `id` and `name`, because that's all the selection set asked for. `totalCount` isn't in the response either, for the same reason: it was never requested at the top level.

Now add `totalCount` back at the top level, and `description` and `slug` inside `data { }`:

```graphql showLineNumbers
query GetTables($pagination: PublicPagination) {
  getTables(pagination: $pagination) {
    totalCount
    data {
      id
      name
      description
      slug
    }
  }
}
```

```json
{
  "data": {
    "getTables": {
      "totalCount": 42,
      "data": [
        {
          "id": "abc123",
          "name": "FCT_ORDERS",
          "description": "Daily order fact table",
          "slug": "fct-orders"
        },
        {
          "id": "def456",
          "name": "DIM_CUSTOMERS",
          "description": null,
          "slug": "dim-customers"
        }
      ]
    }
  }
}
```

Two things changed in the response, and both trace directly back to the query:

- `totalCount` now appears once, at the same level as `data`, because it was added to the selection set at that level.
- Every object inside `data` now has four keys instead of two. `description` and `slug` were added once in the selection set, and Catalog applies that to every row in the list. You don't repeat field names per row.

`description` came back as `null` for `DIM_CUSTOMERS`. This is normal: you asked for the field, and Catalog returned it, but no description has been set on that table. A field appearing as `null` and a field being absent mean different things. `null` means you asked for the field but nothing is set. Absent means you never asked for it. Required fields marked with `!` on the [Table][] type cannot return `null`; optional fields can.

The same rule applies anywhere you see `{ }` in this guide, including the nested `schema { database { name } }` relation further down. Adding a field anywhere inside a selection set adds it to the response at that exact position, and nowhere else.

### Matching Variables to the Query

The query declares `$pagination`. The **Variables** JSON supplies its value under the matching key, `"pagination"`:

```json showLineNumbers
{
  "pagination": {
    "nbPerPage": 10,
    "page": 0
  }
}
```

| In the query | In Variables JSON |
|--------------|--------------------|
| `$pagination: PublicPagination` | not used as a key. This just declares the variable and its type |
| `getTables(pagination: $pagination)` | `"pagination": { ... }`, the value passed to the argument |
| `PublicPagination`, a schema type | never a JSON key. It describes which keys are allowed inside the value, here `nbPerPage` and `page` |

`PublicPagination` is an input type: it tells you what's allowed inside the object, not where to put the object. `nbPerPage` is capped at 500; `page` is zero-based.

**Variables** JSON is plain JSON, not GraphQL. Don't prefix its keys with `$`. Copy the block from the operation's **Example** as a starting point, then edit the values.

### Reading the Response

```json
{
  "data": {
    "getTables": {
      "totalCount": 42,
      "data": [
        { "id": "abc123", "name": "FCT_ORDERS" },
        { "id": "def456", "name": "DIM_CUSTOMERS" }
      ]
    }
  }
}
```

The response nests under `data.getTables`, the same root field name you called, with `totalCount` and `data` filled in because you asked for them. Nothing else appears.

### Adding Filters, Sorting, and Relations

`getTables` also accepts `sorting` and `scope`, and lets you pull in related objects:

```graphql
query GetTables(
  $pagination: PublicPagination
  $sorting: [TableSorting!]
  $scope: GetTablesScope
) {
  getTables(pagination: $pagination, sorting: $sorting, scope: $scope) {
    totalCount
    data {
      id
      name
      schema {
        name
        database {
          name
        }
      }
    }
  }
}
```

```json
{
  "pagination": { "nbPerPage": 20, "page": 0 },
  "sorting": [{ "sortingKey": "name", "direction": "ASC" }],
  "scope": { "nameContains": "orders" }
}
```

| Argument | Purpose |
|----------|---------|
| `pagination` | Page size (`nbPerPage`, max 500) and zero-based `page`. |
| `sorting` | A list of `{ sortingKey, direction }` objects. Note the `[TableSorting!]` type, meaning a JSON array of sorting objects. |
| `scope` | Filters, combined with AND logic. |

`schema { name database { name } }` is a nested relation. Check the **Response** section on the operation page to see whether paths like `schema.database` are available on that endpoint. Deeper trees mean larger responses, so add nesting only when your integration needs the linked metadata.

## Writing a Mutation: Create Columns

Queries read. Mutations write. [Create columns][] creates one or more columns, so it's a good next example for seeing how a `mutation` differs from a `query`. Since it works on a list of inputs at once, it also shows how list types flow through both the request and the response.

### The Operation Signature

```graphql
createColumns(
  data: [CreateColumnInput!]!
): [Column!]!
```

Compare this to `getTables`. The overall shape is the same: a name, arguments, and a return type. A few details are new:

- The argument is named `data`, and it's a list type: `[CreateColumnInput!]!`. Read this from the inside out. `CreateColumnInput!` means each item in the list is required with no `null` entries. The outer `!` means the list itself is required: you can't omit `data` or pass `null`, though an empty list `[]` is still a list.
- The return type, `[Column!]!`, follows the same pattern: a required list of required `Column` objects, one per column created.
- `createColumns` allows no nested relations in its response. Whatever fields you select on `Column`, you get back flat.

This is a useful contrast with `getTables`, where the list lived inside an output object (`GetTablesOutput.data`). Here, the operation's return type is the list directly.

### Writing the Mutation

```graphql showLineNumbers
mutation CreateColumns($data: [CreateColumnInput!]!) {
  createColumns(data: $data) {
    id
    name
  }
}
```

Reading line 1 against the `getTables` example: the keyword changed from `query` to `mutation`, but the pattern is identical: name, then `($variable: Type)`. The `!` on `[CreateColumnInput!]!` means the **Variables** JSON must include `"data"` as an array. Omitting it, or sending `null`, is a validation error rather than an empty result.

### Variables

```json showLineNumbers
{
  "data": [
    {
      "tableId": "abc123",
      "name": "customer_email",
      "dataType": "STRING",
      "externalId": "customer_email",
      "isNullable": true
    },
    {
      "tableId": "abc123",
      "name": "customer_phone",
      "dataType": "STRING",
      "externalId": "customer_phone",
      "isNullable": true
    }
  ]
}
```

| In the mutation | In Variables JSON |
|------------------|--------------------|
| `$data: [CreateColumnInput!]!` | declares the required variable and its list type |
| `createColumns(data: $data)` | `"data": [ ... ]`, a JSON array matching the `[ ]` in the type |
| `CreateColumnInput` | describes which keys each item in the array may contain. Open the [Types][] reference for the full field list |

The JSON array here is the direct counterpart of the `[ ]` in `[CreateColumnInput!]!`. Whenever you see square brackets in a type, send a JSON array in the matching spot.

### Reading the Response

```json
{
  "data": {
    "createColumns": [
      { "id": "col_001", "name": "customer_email" },
      { "id": "col_002", "name": "customer_phone" }
    ]
  }
}
```

The response nests under `data.createColumns`, same as every other operation. Because the return type is `[Column!]!` rather than a single object, the value is a JSON array with one entry per input item, in the same order you sent them: two columns in, two columns back.

## Writing a Mutation: Remove Team Users

[Remove team users][] removes users from a Team. It's a good second mutation example because its return type is different from both prior examples: it returns a plain `Boolean`, not an object.

### The Operation Signature

```graphql
removeTeamUsers(
  data: TeamUsersInput!
): Boolean!
```

Same `data: SomeInput!` pattern seen on `createColumns`. The difference worth noticing is the return type: `Boolean!` instead of an object or list type. That changes what you're allowed to select in the response.

### Writing the Mutation

```graphql showLineNumbers
mutation RemoveTeamUsers($data: TeamUsersInput!) {
  removeTeamUsers(data: $data)
}
```

Notice there's no `{ }` selection set after `removeTeamUsers(data: $data)`. Selection sets are only valid on fields that return an object or list. Scalars like `Boolean`, `String`, `Int`, and `ID` are leaf values, so you select the field itself and stop. Adding `{ }` here would be a syntax error.

### Variables

```json showLineNumbers
{
  "data": {
    "id": "team_123",
    "emails": ["user456@example.com", "user789@example.com"]
  }
}
```

`emails` as an array matches the `[String!]!` type on [TeamUsersInput][]. Whenever a Variables value is a JSON array, expect a `[Type]` or `[Type!]` in the corresponding input type.

### Reading the Response

```json
{
  "data": {
    "removeTeamUsers": true
  }
}
```

No nested object, no selected fields: just the scalar result. A `true` means the users were removed. This is the simplest response shape in the API, and a useful contrast to the nested `data.getTables` and `data.createColumns` shapes above.

## Comparing All Three

| | `getTables` | `createColumns` | `removeTeamUsers` |
|---|---|---|---|
| Operation type | `query` | `mutation` | `mutation` |
| Arguments | `pagination`, `sorting`, `scope`: all optional | `data: [CreateColumnInput!]!`: required list | `data: TeamUsersInput!`: required |
| Return type | `GetTablesOutput!`, object with `totalCount` and `data` list | `[Column!]!`, list of objects returned directly | `Boolean!`, scalar |
| Selection set required? | Yes, must select at least one field | Yes, must select at least one field on `Column` | No, `Boolean` is a scalar with no `{ }` |
| Changes data? | No | Yes, creates one or more columns | Yes, removes team members |

The pattern that carries across all three: declare your variables on the operation line with `$name: Type`, pass them into the root field's arguments, and shape the **Variables** JSON to match. What differs is whether the argument is optional or required, marked with `!`, whether it's a single value or a list, marked with `[ ]`, and whether the return type needs a selection set. Objects and lists of objects do; scalars don't.

## Types in the Catalog Schema

Every argument and field in the reference has a type. Open the [Types][] section, or click a type name directly from an operation page, when you need a full field list or to check which fields are **Required**.

### Syntax You Will See

| Syntax | Meaning | Example |
|--------|---------|---------|
| `String` | Text | `"revenue"` |
| `Int` | Integer | `10` |
| `Boolean` | `true` or `false` | `true` |
| `ID` | Identifier string | `"abc123"` |
| `SomeInput` | Object shape you send in **Variables** | `{ "nbPerPage": 10, "page": 0 }` |
| `!` | Required | `CreateColumnInput!`, `Boolean!` |
| `[SomeType]` | List, sent or received as a JSON array | `[{ "sortingKey": "name", "direction": "ASC" }]` |
| `[SomeType!]!` | Required list of required items | `["user_456", "user_789"]` |

### Inputs, Objects, and the Rest

| Kind | Role in Catalog | Seen in this guide as |
|------|-----------------|------------------------|
| **Input** | Shape of objects you send in **Variables** | `PublicPagination`, `CreateColumnInput`, `TeamUsersInput` |
| **Object** | Shape of data you can select in `{ }` | `GetTablesOutput`, `Table`, `Column` |
| **Scalar** | A single leaf value with no selection set | `Boolean`, `String`, `ID`, `Int` |
| **Enum** | A fixed set of string values | sort `direction`, for example `ASC` or `DESC` |

The same distinction explains why `getTables { totalCount data { id name } }` needs braces and `removeTeamUsers(data: $data)` doesn't: the first returns an Object, the second a Scalar.

## Using the Reference

Generated pages come from the [public GraphQL schema file][]. When the product schema changes, the reference updates with it. For the three-panel layout, **Try it** form, code samples, and a step-by-step walkthrough of each reference block, see [Using the GraphQL Reference][].

## Send the Request with cURL

Everything above describes the GraphQL text and the **Variables** JSON. To call the API from a terminal or script, send both inside an HTTP `POST` body. See [Catalog GraphQL API][] for the `?op=` query parameter and regional hosts.

```bash
curl -X POST 'https://api.castordoc.com/public/graphql?op=getTables' \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'Authorization: Token YOUR_CATALOG_API_TOKEN' \
  -d '{
    "query": "query GetTables($pagination: PublicPagination) { getTables(pagination: $pagination) { totalCount data { id name } } }",
    "variables": {
      "pagination": { "nbPerPage": 10, "page": 0 }
    }
  }'
```

The same pattern applies to mutations. Swap the `?op=` value, the `query` string, and the `variables` object:

```bash
curl -X POST 'https://api.castordoc.com/public/graphql?op=removeTeamUsers' \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'Authorization: Token YOUR_CATALOG_API_TOKEN' \
  -d '{
    "query": "mutation RemoveTeamUsers($data: TeamUsersInput!) { removeTeamUsers(data: $data) }",
    "variables": {
      "data": { "id": "team_123", "emails": ["user456@example.com", "user789@example.com"] }
    }
  }'
```

Use a token from [Getting Your Catalog API Keys][]. For US-hosted Catalog instances, replace the host with `api.us.castordoc.com`.

## What's Next?

- [Catalog GraphQL API][]
- [Using the GraphQL Reference][]
- [Get tables][]
- [Create columns][]
- [Remove team users][]
- [Types][]

[Catalog GraphQL API]: /docs/catalog/api
[Using the GraphQL Reference]: /docs/catalog/api/using-the-graphql-reference
[Get tables]: /docs/catalog/api/general/operations/queries/get-tables
[Create columns]: /docs/catalog/api/general/operations/mutations/create-columns
[Remove team users]: /docs/catalog/api/general/operations/mutations/remove-team-users
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Types]: /docs/catalog/api
[Table]: /docs/catalog/api/general/types/objects/table
[TeamUsersInput]: /docs/catalog/api/general/types/inputs/team-users-input
[public GraphQL schema file]: https://github.com/Coalesce-Software-Inc/public-product-documentation/blob/develop/static/graphql/public-schema.graphql
[GraphQL.org introduction]: https://graphql.org/learn/introduction/

---

## Catalog GraphQL API

The Catalog Public GraphQL API is the machine-readable reference for Catalog metadata operations. For workflow context, the in-app playground, and token management, see [Catalog Public API][].

:::tip[New to GraphQL?]

Start with [Using the GraphQL Reference][] for a walkthrough of this site, or [Introduction to GraphQL][] for query syntax and variables.

:::

## Authentication

Include a Catalog API token on every request:

```json
{"Authorization": "Token <API_TOKEN>"}
```

Catalog administrators create tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for scopes, rotation, and revocation.

## Base URLs

Send all requests with a JSON body. Append `?op=<operationName>` using the root field you call.

| Region | URL |
|--------|-----|
| EU | `https://api.castordoc.com/public/graphql?op=<operationName>` |
| US | `https://api.us.castordoc.com/public/graphql?op=<operationName>` |

## What's Next?

- [API Responses and Error Codes][]
- [Using the GraphQL Reference][]
- [Introduction to GraphQL][]
- [Catalog Public API][]
- [AI Assistant API][]

[API Responses and Error Codes]: /docs/catalog/api/api-responses
[Using the GraphQL Reference]: /docs/catalog/api/using-the-graphql-reference
[Introduction to GraphQL]: /docs/catalog/api/graphql
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[AI Assistant API]: /docs/catalog/developer/ai-assistant-api

---

## Using the GraphQL Reference

This guide walks through the Catalog GraphQL API reference. Each section explains one block of the reference UI: what it shows, how the pieces connect, and where to edit values before you send a request. For GraphQL syntax and how to read a query line by line, see [Introduction to GraphQL][].

## Open the Reference

Start at [Catalog GraphQL API][]. That page covers authentication, regional base URLs, and links to the rest of the reference.

The left sidebar is your map:

| Section | What it contains |
|---------|------------------|
| **Overview** | Authentication headers, EU and US base URLs |
| **Introduction to GraphQL** | How to read query text and **Variables** JSON |
| **Using the GraphQL Reference** | This walkthrough of the reference UI |
| **Operations** | Every **query** (read) and **mutation** (write) you can call |
| **Types** | Input shapes, response objects, enums, and scalars |

Under **Operations**, blue **query** badges mark read-only operations. Green **mut** badges mark mutations that create or update metadata.

## The Three-Panel Layout

Open any operation, such as [Get tables][], to see the full layout. On a wide screen the page splits into three areas:

1. **Left (navigation)** - Jump between operations and types.
2. **Center (documentation)** - What the operation does, which arguments it accepts, example query text, and the response shape.
3. **Right (Try it)** - Live code samples and a form to send the request from your browser.

On smaller screens the Try it panel stacks under the documentation. Widen your browser window if you do not see the side-by-side layout.

## Read the Center Panel

Each operation page follows the same structure top to bottom. The subsections that follow walk through each block in order.

### Header and Description

At the top you see a badge (**query** or **mutation**) and the operation name, for example **getTables**. The short description tells you what the call returns or changes.

| Block | What it tells you |
|-------|-------------------|
| **Badge** | **query** means read-only; **mutation** means create, update, or delete |
| **Operation name** | The root field name to use in GraphQL text, for example `getTables` |
| **Description** | Plain-language summary of what the call does |

The operation name matches the root field in the **Example** query and the `?op=` value in HTTP URLs on [Catalog GraphQL API][].

### Arguments

The **Arguments** section lists every input the operation accepts. Each row shows:

| Part | Meaning |
|------|---------|
| **Name** | Argument name used in the query, for example `pagination` |
| **Type** | Schema type; click the link to open the type page |
| **Required** or **Optional** | Whether you must supply a value |

Nested fields under an argument describe the keys allowed inside a **Variables** object. For example, when `pagination` has type [PublicPagination][], the nested rows show `nbPerPage` and `page` as the keys you can put inside `"pagination": { ... }` in **Variables** JSON.

Read this block top to bottom:

1. **Top-level rows** - Arguments you pass in parentheses on the root field, for example `getTables(pagination: $pagination)`.
2. **Nested rows** - Keys inside the **Variables** object for that argument.
3. **Type links** - Open the type page when you need the full field list or enum values.

### Example: Query Block

The **Example** section is the fastest way to copy a working request. It has two blocks. The **Query** block is GraphQL operation text.

```graphql showLineNumbers
query GetTables($pagination: PublicPagination) {
  getTables(pagination: $pagination) {
    totalCount
    data {
      id
      name
    }
  }
}
```

Walk through each line:

| Line | Piece | What it means on this page |
|------|-------|----------------------------|
| 1 | `query` | Read-only operation. Mutations start with `mutation`. |
| 1 | `GetTables` | Operation label for your logs or client. It does not change the API call. |
| 1 | `($pagination: PublicPagination)` | Declares one variable. `$pagination` is filled from **Variables** JSON. `PublicPagination` links to the type page. |
| 2 | `getTables(pagination: $pagination)` | Root field. Matches the operation name in the page header and the **Arguments** section. |
| 3 | `totalCount` | Field to return: total rows across all pages. |
| 4–7 | `data { id name }` | Paginated rows and the fields to include on each row. |

Copy the whole **Query** block into the Try it **Query** field or into your client.

Some operations have no variables. Those pages show only the **Query** block.

### Example: Variables Block

The **Variables** block is plain JSON that supplies values for each `$variable` in the query:

```json showLineNumbers
{
  "pagination": {
    "nbPerPage": 10,
    "page": 0
  }
}
```

Walk through each part:

| Line | Key or value | What it means on this page |
|------|--------------|----------------------------|
| 2 | `"pagination"` | Matches `$pagination` from the query. No `$` on JSON keys. |
| 3 | `"nbPerPage": 10` | Page size. Capped at 500 for [PublicPagination][]. |
| 4 | `"page": 0` | Zero-based page index. |

The type name `PublicPagination` is schema documentation only. It never appears as a JSON key. Operations with multiple arguments add more top-level keys. Each key matches a `$variable` declared on line 1 of the query.

### Response

The **Response** section lists **only the fields you can request on this operation**—not every field that exists on the underlying GraphQL type.

| Block | What it shows |
|-------|----------------|
| **Root type** | Return type for the operation, for example `[Column!]!` or `GetTablesOutput` |
| **Scalar fields** | Simple values you can always ask for on this endpoint (`id`, `name`, `tableId`, …) |
| **Nested rows** | Nested objects you can request on this operation, when supported |

Nested paths use dot notation in your query—for example `schema { database { name } }` on table rows. The **Response** section lists only fields you can request on **this** endpoint. If you need related data not shown there (for example `table` after `createColumns`), run a follow-up read query such as `getColumns` or `getTables`.

If nested fields are not listed under **Response**, stick to simple fields only.

## Use the Try It Panel

The right column is an interactive explorer. You do not need a separate GraphQL client to test a call.

### Code Samples

At the top, tabs switch between **curl**, **Python**, **PowerShell**, **PHP**, and **NodeJS**. Each sample is a complete HTTP request built from the current query and variables. To change the values, update the query and variables block.

A curl sample breaks down like this:

| Block | What it contains |
|-------|------------------|
| **URL** | Regional host plus `?op=getTables` (operation name from the page header) |
| **Headers** | `Content-Type`, `Accept`, and `Authorization: Token ...` |
| **Body `"query"`** | The GraphQL text from the **Query** field, often on one line |
| **Body `"variables"`** | The JSON object from **Variables (JSON)** |

When you change the **Query** or **Variables (JSON)** fields, every language tab updates immediately.

For example, changing `"page": 0` to `"page": 1` and `"nbPerPage": 5` to `"nbPerPage": 25` updates the `"variables"` object in every language tab:

### Request Form

Changing values here, will update the values in the code samples block. Below the samples, the **Request** form has three collapsible sections:

#### Server Block

| Control | Effect |
|---------|--------|
| **EU** | Sets the host to `api.castordoc.com` |
| **US** | Sets the host to `api.us.castordoc.com` |

#### Auth Block

| Control | Effect |
|---------|--------|
| Token field | Value sent as the `Authorization` header. Include the `Token` prefix. |

#### GraphQL Block

| Control | Effect |
|---------|--------|
| **Query** textarea | Same GraphQL text as the **Example > Query** block. Edit fields inside `{ }` to change the response shape. |
| **+ Show variables** | Opens the **Variables (JSON)** textarea when it is hidden |
| **Variables (JSON)** textarea | Same JSON as the **Example > Variables** block. Invalid JSON is rejected when you send the request. |

## Send a Request From the Browser

To run a live call against your Catalog environment:

1. Open an operation page, for example [Get tables][].
2. In **Auth**, paste a token with the `Token` prefix.
3. Confirm **Server** matches your Catalog region (EU or US).
4. Review or edit **Query** and **Variables (JSON)**. Start from the **Example** section if you are unsure.
5. Click **Send API Request**.
6. Read the **Response** panel. HTTP `200` means the request reached the API; check `data` and `errors` to see whether the call succeeded. See [API Responses and Error Codes][] for status codes and troubleshooting.

### Response Panel

After you click **Send API Request**, the **Response** panel shows the HTTP result:

| Block | What to look for |
|-------|------------------|
| **Status code** | `200` means the HTTP request reached the API (auth failures can still return `200`) |
| **`data`** | Fields you asked for when the call succeeded |
| **`errors`** | GraphQL or validation messages when something failed; see `extensions.code` |

Compare the `data` object to the fields in your **Query** block. Nested keys should match the selection set you built in the center panel **Example** or in the Try it **Query** field.

## Query and Variables: Quick Reference

Use this table when you need a side-by-side reminder of how **Query** text and **Variables** JSON fit together:

| | **Query** (GraphQL text) | **Variables** (JSON) |
|---|--------------------------|----------------------|
| **Format** | GraphQL syntax | Plain JSON object |
| **Declares** | Operation name, `$variables`, fields to return | Actual values for each variable |
| **Key rule** | `$pagination` in the query | `"pagination": { ... }` in JSON. No `$` on keys. |
| **Where to edit in the reference** | **Example > Query** and Try it **Query** field | **Example > Variables** and Try it **Variables (JSON)** field |
| **Sent to the API as** | `"query": "..."` in the POST body | `"variables": { ... }` in the POST body |

The HTTP POST body always combines both:

```json showLineNumbers
{
  "query": "query GetTables($pagination: PublicPagination) { ... }",
  "variables": {
    "pagination": { "nbPerPage": 10, "page": 0 }
  }
}
```

| Line | Key | What it is |
|------|-----|------------|
| 2 | `"query"` | Full GraphQL operation text as a JSON string |
| 3 | `"variables"` | JSON object whose keys match `$variables` in the query |
| 4 | `"pagination"` | One variable value. Nested keys come from the **Arguments** section and type pages. |

For a deeper explanation of each line in the query, see [Introduction to GraphQL][].

## Type Pages

When an argument or field links to a type name, open that type under **Types** in the sidebar. Each type page follows the same block pattern as operation **Arguments** and **Response** sections:

| Type kind | What the page describes |
|-----------|-------------------------|
| **Input** | Keys allowed in **Variables** JSON |
| **Object** | Fields you can select in `{ }` |
| **Enum** | Allowed string values such as sort direction |
| **Scalar** | Single values such as `String`, `Int`, or `ID` |

Use type pages when the **Arguments** nested rows or **Response** tree do not show every field you need.

## What's Next?

- [API Responses and Error Codes][] - HTTP status codes and GraphQL error troubleshooting
- [Catalog GraphQL API][] - Authentication and base URLs
- [Introduction to GraphQL][] - Read query syntax and variables line by line
- [Get tables][] - Worked example operation
- [Getting Your Catalog API Keys][] - Create and manage tokens

[API Responses and Error Codes]: /docs/catalog/api/api-responses
[Catalog GraphQL API]: /docs/catalog/api
[Get tables]: /docs/catalog/api/general/operations/queries/get-tables
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Introduction to GraphQL]: /docs/catalog/api/graphql
[PublicPagination]: /docs/catalog/api/general/types/inputs/public-pagination

---

## Dashboards

<!-- vale Coalesce.FirstPerson = NO -->

The Catalog surfaces assets from your BI and visualization tools alongside warehouse data. Open **Dashboards** in the left navigation, or go to [the Dashboards hub in Catalog][], to browse, document, and trace lineage for every synced visualization asset.

## Open the Dashboards Hub

Click **Dashboards** in the left navigation. Catalog expands the section and shows two areas:

1. **Left** - Every connected visualization integration, for example ThoughtSpot, Domo, Power BI, Tableau, Sigma, or Omni. Click **Dashboards** again if the list is collapsed.
2. **Right** - A searchable list of visualization assets from all integrations.

### List Columns

Every row in the list uses the same three columns:

| Column | What it shows |
| ------ | ------------- |
| **Name** | Asset title, an icon for the asset type, and a path showing where the asset lives in the source tool, for example `Tableau > default > 2025-Marketing-Metrics`. A green checkmark on the name means the asset is verified. |
| **Description** | A snippet from the Catalog **Read Me** tab, often labeled **Context**, plus any tags applied in Catalog. |
| **Popularity** | A star rating based on view activity in the source tool when that data is available. |

Click any row to open the asset. Use the external-link icon beside the description to open the asset in the native BI tool.

## Filter by Integration

Click an integration name in the left navigation, for example **Tableau**, to filter the list to that connection only.

Under the integration you typically see:

* A **Data Sources** entry. The label varies by platform; see the walkthrough below.
* One or more **project or folder** entries that mirror how content is organized in the source tool, for example **default** or **Personal project**.

The right-hand list updates to show only assets from that integration. Breadcrumbs above the list reflect your path, for example `Dashboards > Tableau`.

Folder and data-source names depend on the platform and how your team organizes content. For connection setup and naming by tool, see [Data visualization integrations][].

## Walkthrough: Tableau Example

The steps below use Tableau as an example. The same navigation pattern applies to other integrations; only folder names and asset labels change.

### Browse Data Sources

Under **Tableau**, click **Data Sources**, then a subfolder such as **default**.

The list uses the same **Name**, **Description**, and **Popularity** columns. Rows show semantic or modeling assets from Tableau, such as published data sources. The path under each name often includes **Models**, for example `Tableau > Data Sources > default`.

### Data Source Detail Page

Click a data source, for example `Application_Users`, to open its page.

#### Tabs

| Tab | Purpose |
| --- | ------- |
| **Home** | Fields synced from the source tool, grouped by table or model. Expand a group to see field names, types, and descriptions. Use **Filter** and **Search label** to narrow the list. |
| **Read Me** | Catalog documentation: edit manually, use **Describe with AI**, or request documentation from owners. |
| **Lineage** | Upstream [warehouse tables][] and downstream dashboards that use this data source. |
| **Comments** | Questions and collaboration on this asset. |
| **History** | Changes to descriptions, owners, tags, and other metadata. |

#### Details Panel

The **Details** panel on the right side of every tab shows verified and deprecated status, favorites, **Link to source**, owners, popularity, data product toggle, domain, tags, frequent users, **Mentioned in** links to [Knowledge][] pages and other assets, and the source connection. **Link to source** opens the asset in Tableau.

Field-level lineage on data sources is available for supported integrations. See [Data visualization integrations][] for your tool.

### Browse Dashboards in a Project Folder

Under **Tableau**, click a project folder such as **default**, not **Data Sources**.

The list shows dashboards and related content for that project. Icons beside each name indicate the asset type in Catalog:

| Icon | Meaning |
| ---- | ------- |
| Grid, four squares | Workbook or top-level dashboard container. |
| Line chart | Individual view, sheet, or chart inside a workbook. |
| Bar chart, horizontal bars | Another dashboard or sheet type, depending on how Tableau exported the asset. |

### Open a Workbook Folder

Click a workbook folder in the left navigation, for example **2025-Marketing-Metrics** under **default**.

The list usually contains two kinds of rows:

1. The workbook itself, **2025-Marketing-Metrics**, with a grid icon.
2. Views inside that workbook, for example **Dashboard 1**, with a line-chart icon.

Both rows use the same **Name**, **Description**, and **Popularity** columns.

### Workbook Detail Page

Click the workbook row, **2025-Marketing-Metrics**.

Workbook pages include **Read Me**, **Lineage**, **Comments**, and **History**. They do not include a **Home** tab; field lists live on data sources.

On the **Read Me** tab, document purpose, audience, and metrics. Pin related [tables][], data sources, other dashboards, or [Knowledge][] pages so readers see context without leaving the page. See [Pinned assets][].

On the **Lineage** tab, see which [warehouse tables][] and data sources feed this workbook and which downstream assets depend on it.

The **Details** panel can include **Mentioned in** links to Knowledge pages and other Catalog assets, plus owners, domain, tags, and a **Tableau** button to open the live workbook.

### View Detail Page

Click a view inside the workbook, for example **Dashboard 1**.

The page matches the workbook layout: **Read Me**, **Lineage**, **Comments**, **History**, and the same **Details** fields. Breadcrumbs include the parent workbook, for example `Dashboards > Tableau > default > 2025-Marketing-Metrics > Dashboard 1`.

Use the **Lineage** tab to see tables and data sources behind this specific view. Use the **Read Me** and **Comments** tabs the same way as on the workbook.

## How Dashboards Connect to the Rest of Catalog

| Catalog area | Connection to dashboards |
| ------------ | ------------------------ |
| [Tables][] | Lineage links dashboards and data sources to warehouse tables and columns. |
| [Knowledge][] | Document definitions and KPIs; link them from **Mentioned in** or pin them on the **Read Me** tab. |
| [Reports][] | Export dashboard metadata, unused-dashboard reports, and governance analytics from **Governance**. |
| [Data visualization integrations][] | Connect BI tools and control what Catalog syncs. |

## Collaboration and Metadata

These capabilities apply on dashboard, view, and data source pages:

* **Comments** - Discuss the asset on the **Comments** tab.
* **History** - Review metadata changes on the **History** tab. See [History][] for logged change types and governance workflows.
* **Governance** - Admins can mark assets verified or deprecated from the **Details** panel. Export metadata through [Reports][].

---

[the Dashboards hub in Catalog]: https://app.castordoc.com/viz
[Data visualization integrations]: /docs/catalog/integrations/data-viz
[Tables]: /docs/catalog/assets/tables
[Knowledge]: /docs/catalog/assets/knowledge
[Reports]: /docs/catalog/assets/reports
[Pinned assets]: /docs/catalog/document-your-data/pinned-assets
[History]: /docs/catalog/collaborate/history
[warehouse tables]: /docs/catalog/assets/tables

---

## Knowledge

One of the Catalog's goals is to centralize all information around your data. Some knowledge goes beyond tables and dashboards, such as KPIs and frequently used terms, and can be stored in the Knowledge section of the Catalog App.

Each entry is composed of a [Home][], [Comments][], and [History][] tab. You can nest Knowledge pages in one another.

## Home

The Home tab is a Rich Text section allowing you to describe and explain any concept, KPI, or definition.

## Details

On the right-hand side of your screen:

* **Verified:** admins can mark an entry as verified to indicate documentation has been reviewed
* **Favorites:** users can mark entries as favorites
* **Owners:** a user can be assigned as the owner of an entry
* **Description:** contains an extract of the description in the [Home][] tab. By clicking on it, you will return to the [Home][] tab.
* **Mentioned in:** lists the assets which reference the entry
* **Tags:** tags can be used to describe assets. They are common for all users of a company.
* **Delete definition:** button to delete definitions. Either an admin or the definition owner can do so

## Pinned Assets

To improve your knowledge content, you can directly pin relevant assets to it.

<figure><figcaption></figcaption></figure>

How does this work? See the [pinned assets page][] for details.

## Link a Tag

Determine which tag relates to the definition you're writing to make all tagged assets appear in the page directly and dynamically.

1. Select the tag that is linked to your definition:

    

2. Once the tag is linked, all assets with that tag will appear for easier discovery

    <figure><figcaption></figcaption></figure>

## Comments

To easily collaborate and communicate between users, each dashboard has a Comments section:

<figure><figcaption></figcaption></figure>

## History

Catalog records metadata changes on the **History** tab. See [History in Catalog][] for what is logged and how stewards use the timeline for governance reviews.

<figure><figcaption></figcaption></figure>

[Home]: /catalog/assets/knowledge.md#home
[Comments]: /catalog/assets/knowledge.md#comments
[History]: /catalog/assets/knowledge.md#history
[History in Catalog]: /docs/catalog/collaborate/history
[pinned assets page]: /catalog/document-your-data/pinned-assets.md

---

## Reports

<!-- vale off -->

Exportable metadata and analytics reports are available to Admins through the Governance section in Catalog as CSV or XLSX downloads.

<figure><figcaption></figcaption></figure>

## When To Use Reports

Here are some examples of scenarios when you might use reports:

* **Understanding and navigating your data health**
  * How well are your tables, dashboards, or knowledge pages documented?
  * Which table should have the highest SLA?
  * Where is the PII data located?
* **Cost Reduction**
  * Who is running the most queries? Which user runs the most expensive queries?
  * Which tables are queried the most? Which dashboards are the most expensive to compute?
* **Noise reduction**
  * Which dashboards are never used?
  * Which tables can be dropped?

## Unused Assets Reports

Generated when Catalog ingests either the warehouse or the visualization tool.

Unused asset reports analyze how assets are used in your organization. An asset is flagged as unused if both itself and all its downstream tables and dashboards have not been used by a human or BI tool user (classification of users is performed by Catalog and can be edited on demand).

The analysis is based on the past 180 days of data.

### Unused Tables

* last\_usage: date where Catalog has last seen a "useful" query or dashboard view for the table and its downstream tables/dashboards. Field will be empty if no activity has been seen by Catalog in earliest of past 180 days or date of Catalog connection to your warehouse
* table\_path: path of table in format `{database_name}.{schema_name}.{table_name}`
* table\_type: `VIEW` or `TABLE`
* creation\_date: date where table was first seen by Catalog
* table\_size\_mb: storage size in MB of table (NB: tables of size 0.4 MB or below will have size 0 displayed).
* refresh\_computation\_duration: sum of query duration required to refresh table data over past 180 days
* lineage\_type:
  * `isolated` : table has neither parent or child tables/dashboards
  * `root` : table has no parent tables but has children tables/dashboards
  * `branch` : table has both parent and children tables/dashboards
  * `leaf`: table has parent tables but no children tables/dashboards
* table\_url: Catalog URL of table (points to columns tab)

### Unused Dashboards

* last\_usage: date where Catalog has last seen a dashboard view of dashboard. Field will be empty if no activity has been seen by Catalog in earliest of past 180 days or date of Catalog connection to your BI tool
* dashboard\_path: dashboard path
* dashboard\_name: dashboard name
* creation\_date: date where table was first seen by Catalog
* total\_views: number of views since dashboard creation
* dashboard\_url: Catalog URL of dashboard (points to Read Me tab)
* dashboard\_type: `DASHBOARD`, `TILE`, `EXPLORE` (Looker), `APP` (Qlik), `DATA_SOURCE` (Tableau), `DATASET` (PowerBI & Sigma)

## Metadata Reports

Generated when the download is requested.

### Dashboard Metadata

* dashboard\_path: dashboard path and name
* dashboard\_url: Catalog URL of dashboard (points to Read Me tab)
* description: the description that is in the Read Me
* popularity: popularity of dashboard
* individual\_owners: list of owner emails
* team\_owners: list of team owner names
* editors: editor emails
* user\_tags: list of tags added by you
* external\_tags: list of tags added from the source
* internal\_tags: list of tags added by Catalog
* is\_certified: has the dashboard been certified by a user
* is\_described: does the dashboard have a non-empty Read Me

### Knowledge Metadata

* knowledge\_path: path of knowledge page
* knowledge\_name: name of knowledge page
* knowledge\_url: Catalog URL of knowledge page
* description: the description that is in the Read Me
* individual\_owners: list of owner emails
* team\_owners: list of team owner names
* tags: list of tags
* is\_certified: is the knowledge page certified
* is\_described: does the knowledge page have a non-empty Read Me
* num\_pins: how many assets are pinned in this knowledge page
* num\_references: how many times this knowledge page is pinned

### PII Columns

* column\_key: key of column in format `{database_name}.{schema_name}.{table_name}.{column_name}`
* column\_url: Catalog URL of column
* column\_type: data type of column
* column\_editable\_description:  in-app user description
* column\_external\_description: description fetched from external source (warehouse, BI tool, or dbt)
* column\_propagated\_description: column description propagated from another column using column lineage

### Table Metadata

* table\_key: key of table in format `{database_name}.{schema_name}.{table_name}`
* table\_url: Catalog URL of table (points to columns tab)
* description: the description related to that table
* popularity: popularity of table
* has\_pii: does the table contain any columns flagged as PII
* individual\_owners: list of owner emails
* team\_owners: list of team owner names
* user\_tags: list of tags generated by a Catalog user
* external\_tags: list of tags imported from the external source
* internal\_tags: (or Catalog tags) list of tags added by Catalog when parsing the queries
* is\_certified: has the table been certified by a user
* is\_deprecated: has the table been deprecated by a user
* is\_described: does the table have a non-empty Read Me
* num\_columns: number of columns in table
* num\_columns\_described\_by\_client: number of columns with descriptions, excluding descriptions from Catalog [Auto doc][]
* num\_columns\_described\_by\_castor: number of columns with descriptions performed by Catalog [Auto doc][]

### Tag Metadata

* tag\_name: name of the tag
* tag\_source: source of the tag
* asset\_type: type of the asset
* asset\_path: path of the asset `{database_name}.{schema_name}.{table_name}`

## Analytics Reports

Generated when Catalog ingests either the warehouse or the visualization tool.

### Dashboard Analytics

* dashboard\_path: dashboard path and name
* dashboard\_url: Catalog URL of dashboard (points to Read Me tab)
* num\_views: number of views of dashboards over past 180 days
* num\_parent\_tables: number of direct parent tables
* num\_ascendant\_tables: total number of upstream tables
* num\_children\_dashboards: number of direct children dashboards
* num\_descendant\_dashboards: total number of downstream dashboards
* num\_parent\_dashboards:  number of direct parent dashboards
* num\_ascendant\_dashboards: total number of upstream dashboards

### Table Analytics

* table\_name: key of table in format `{database_name}.{schema_name}.{table_name}`
* num\_read\_queries: total number of read queries over past 180 days
* num\_write\_queries: total number of write queries over past 180 days
* num\_queries: number of read + write queries over past 180 days (aka the sum of the 2 fields above)
* popularity: popularity (1-1 000 000 score)
* num\_parent\_tables: number of direct parent tables
* num\_ascendant\_tables: total number of upstream tables
* num\_children\_dashboards: number of direct children dashboards
* num\_descendant\_dashboards: total number of downstream dashboards
* num\_children\_tables: number of direct children tables
* num\_descendant\_tables: total number of downstream tables
* table\_url: Catalog URL of table (points to columns tab)

### User Activities in the Warehouse

* user\_name: user email address (if available)
* user\_type: user categorisation by Catalog: `Human`, `BI tool`, or `Other`
* num\_queries: total number of queries performed by user over past 180 days
* num\_read\_queries: total number of read queries performed by user over past 180 days
* num\_write\_queries: total number of write queries performed by user over past 180 days
* total\_read\_queries\_duration: sum of read query duration times (for queries performed after November 13, 2022)
* total\_write\_queries\_duration: sum of write query duration times (for queries performed after November 13, 2022)
* num\_tables\_read: total number of tables read by user over past 180 days
* num\_tables\_write: total number of tables written by user over past 180 days

:::warning[Query Duration Data Availability]

For query durations, data is only available for queries after **November 13, 2022.**

:::

### User Activities in the Warehouse per Table

* user\_name: user email address (if available)
* table\_path: path of table in format `{database_name}.{schema_name}.{table_name}`
* creation\_date: date where table was first seen by Catalog
* num\_queries: total number of queries performed by user over past 180 days
* num\_read\_queries: number of read queries performed by user on this table over past 180 days
* num\_write\_queries: number of write queries performed by user on this table over past 180 days
* total\_read\_queries\_duration: sum of read query duration times (for queries performed after November 13, 2022)
* total\_write\_queries\_duration: sum of write query duration times (for queries performed after November 13, 2022)
* num\_days\_w\_queries: number of days where user has a ran a read or write query on table out of the past 180 days

:::warning[Query Duration Data Availability]

For query durations, data is only available for queries after **November 13, 2022.** Null values mean that queries were performed before then.

:::

### Ingestion report

You want to know what was loaded by Catalog when it last extracted metadata from your warehouses?

* source\_id: UUID of the source
* name: name of the source
* technology: technology of the source
* ingestion\_start: start date and time of the ingestion
* ingestion\_end: end date and time of the ingestion, that is, when the report was generated
* day\_of\_extracted\_queries: date of the queries retrieved by this ingestion
* database\_allowed: list of allowed databases
* database\_blocked: list of blocked databases
* schema\_allowed: list of allowed schemas
* schema\_blocked: list of blocked schemas
* users\_blocked\_for\_query\_display: list of users whose queries are not shown in the app
* users\_blocked\_for\_popularity\_compute: list of users whose activity is ignored when calculating the popularity
* num\_databases: total number of databases in the catalog
* num\_schemas: total number of schemas in the catalog
* num\_tables: total number of tables in the catalog
* num\_columns: total number of columns in the catalog
* num\_table\_descriptions: total number of tables with an external description (including propagated descriptions)
* num\_column\_descriptions: total number of columns with an external description (including propagated descriptions)
* num\_table\_tags: total number of external table tags
* num\_new\_databases: number of databases added during the ingestion
* num\_new\_schemas: number of schemas added during the ingestion
* num\_new\_tables: number of tables added during the ingestion
* num\_new\_columns: number of columns added during the ingestion
* num\_deleted\_databases: number of databases removed during the ingestion
* num\_deleted\_schemas: number of schemas removed during the ingestion
* num\_deleted\_tables: number of tables removed during the ingestion
* num\_deleted\_columns: number of columns removed during the ingestion
* num\_queries\_on\_day: total number of queries performed on the extraction date, including duplicate queries
* num\_queries\_on\_day\_deduplicated: total number of unique queries performed on the extraction date
* num\_read\_queries\_on\_day: total number of read queries performed on the extraction date
* num\_write\_queries\_on\_day: total number of write queries performed on the extraction date
* num\_query\_authors\_on\_day: number of unique query authors on the extraction date
* num\_table\_to\_table\_lineage\_links: total number of table lineage links
* num\_column\_to\_column\_lineage\_links: total number of column lineage links
* num\_new\_table\_to\_table\_lineage\_links: number of table lineage added during the ingestion
* num\_new\_column\_to\_column\_lineage\_links: number of column lineage added during the ingestion

More reports are coming soon.

For feedback or requests concerning reports, reach out to the [Reports feedback and requests][] team.

[Auto doc]: https://docs.coalesce.io/docs/catalog/document-your-data/catalog-scribe/castor-auto-doc
[Reports feedback and requests]: mailto:feedbacks-aaaadhjesweipwdab7yypni4ay@castorglobal.slack.com?subject=Reports%20feedbacks%20and%20requests

---

## Tables

<!-- vale off -->

The first step to get visibility on your data is to look into your tables. Tables are organized with the following hierarchy in the left menu:

1. Warehouse technology
2. Database
3. Schema

Each table is documented by a Detail Panel and 7 tabs:

* Tab Columns
* Tab Lineage
* Tab Read Me
* Tab Queries
* Tab Comments
* Tab History
* Tab Data Quality

## Right Panel: Details

<figure><figcaption></figcaption></figure>

At the right, the Details menu summarizes key attributes of the selected table. It is visible from all tabs.

* **Certified:** Admins can mark a table as certified to indicate documentation has been reviewed :white_check_mark:
* **Deprecated:** Admins can mark a table as deprecated to prevent users from using it :warning:
* **PII:** Specifies if the table contains a column with Personal Identifiable Information
* **Favorites:** you can mark tables as favorites, it will appear on the homepage
* **Owners:** a user can be assigned as the owner of a table
* **Popularity:** computed based on the number of table read (select) queries in the last 30 days (not writes). Non human accounts are excluded of the count
* **Latest data update:** when turned on, Catalog displays when the table data was last updated
* **Description:** contains an extract of the description in the [Read Me tab][]. Click it to return to the Read Me tab.
* **Completion level:** computes how well the table has been documented
* **Tags:** tags can be used to describe tables. They are common for all users of a company.
* **Frequent users:** lists the power users of the table
* **Table type:** can either be a table, view, or an external table
* **SQL source:** shows the queries which create and update the selected table

> Completion score is calculated in the following way:
>
> * 20% of the score is attributed if a table owner is assigned
> * 10% of the score is attributed if table Read Me is complete
> * 70% of the score is attributed if column description are complete

### Links

<figure><figcaption>From a table in Catalog, jump back to either Snowflake, Airflow, or the URL of your choice</figcaption></figure>

#### Source Link

A Table has an URL of its own, to discover it in its own warehouse (**Snowflake** & **BigQuery**).

#### External Links

Associate different URLs with a Table, from the following applications:

* GitHub
* GitLab
* Airflow
* Other

These links can be pushed with [**Catalog API**](https://apidocs.castordoc.com/#55f09066-8757-47e0-afe2-6a0f2311870f).

There can currently exist one link of each type for one given Table.

## Tab Columns

The Home tab lists all the [columns](/catalog/assets/tables.md#columns) present in the selected table and tags the [related assets](/catalog/assets/tables.md#related-assets)

### Columns

Each column is described by 5 fields.

<figure><figcaption></figcaption></figure>

<Tabs>
  <TabItem value="type" label="Type" default>
    The column type is specified by Catalog.

    For example: Timestamp, integer, float, etc.
    
    <figure><figcaption></figcaption></figure>
  </TabItem>
  <TabItem value="name" label="Name">
    Column name
  </TabItem>
  <TabItem value="attribute" label="Attribute">
    A column can be tagged as:

    * containing Personal Identifiable Information (**PII**)
    * being a primary key (**PK**)
    
    <figure><figcaption></figcaption></figure>
  </TabItem>
  <TabItem value="description" label="Description">
    There are several options to fill the description of each column:

    * You can manually input a description
      * Auto complete: if you have already filled out the description of the same column in another table, Catalog will suggest it to you
    * Pre-filed documentation: Catalogs will centralise all the documentation you have already completed, in your warehouse or dbt as in the example below. The imported description will be tagged with the logo of its source.
    
    <figure><figcaption></figcaption></figure>
  </TabItem>
</Tabs>

### Related Assets

By clicking on the Related assets button at the bottom of the page, a menu will appear to give you an overview of how the table is created and used.

<figure><figcaption></figcaption></figure>

<Tabs>
  <TabItem value="mostly-joined-with-tables" label="Mostly joined with Tables" default>
    Lists the tables with which are most often joined with the selected table

    Links to the [Queries](/catalog/assets/tables.md#tab-queries) tab
  </TabItem>
  <TabItem value="created-from-tables" label="Created from Tables">
    Lists the tables used to create the selected table

    Links to the [Lineage](/catalog/assets/tables.md#tab-lineage) tab
  </TabItem>
  <TabItem value="used-in-dashboards" label="Used in Dashboards">
    Lists the [Dashboards](/docs/catalog/assets/dashboards) which use the selected table

    Links to the [Lineage](/catalog/assets/tables.md#tab-lineage) tab
  </TabItem>
</Tabs>

## Tab Lineage

<figure><figcaption></figcaption></figure>

You have three ways to view lineage

<Tabs>
  <TabItem value="list-view" label="List View" default>
    In this view you can see:

    * Source Tables: the most upstream tables Catalog could find
    * Parents: the direct upstream tables of your table
    * Children: the direct downstream tables and dashboards

    <figure><figcaption></figcaption></figure>
  </TabItem>
  <TabItem value="table-lineage" label="Table Lineage">
    You can see asset-level lineage.

    * You can filter out in or out dashboards.
    * Also, since lineage is computed based on the observed queries from your warehouse, you can decide to only consider the queries from the last N days.

    <figure><figcaption></figcaption></figure>
  </TabItem>
  <TabItem value="column-lineage" label="Column Lineage">
    After picking the column you want, you can see its specific lineage.

    <figure><figcaption></figcaption></figure>
  </TabItem>
</Tabs>

:::info

Want to know all downstream, on all depth, of a table?

Click on the "**Export all downstream nodes**" button

:::

<figure><figcaption></figcaption></figure>

## Tab Read Me

The third tab of a table page contains the table Read Me.

<figure><figcaption></figcaption></figure>

The Read Me contains the selected table description.

<Tabs>
  <TabItem value="user-inputed-description" label="User-entered description" default>
    * The first section of the Read Me tab gives space for you to input all relevant table descriptions
    * This section is a Rich Text editor, allowing for maximum flexibility and formatting
    * To view Rich Text options, highlight text portions

    
  </TabItem>
  <TabItem value="external-description" label="External description">
    * Catalog centralizes all the documentation you have already completed. The imported descriptions are tagged with the logo of their source and are immutable. Classic sources are dbt or column comments in your warehouse
  </TabItem>
</Tabs>

### Pinned Assets

To improve your table Read Me, you can directly pin relevant assets to it

<figure><figcaption></figcaption></figure>

[Check out how pinned assets work][]

## Tab Queries

The Queries tabs lists queries performed on the selected table

<figure><figcaption></figcaption></figure>

To find a query of interest, you can filter the queries by:

* Author: who wrote the query
* Joined tables: tables joined

To view the full query, click it and a modal will appear:

To easily copy a query click either:

* Normal view: the top right icon of the query block
* Single query view: the icon by the author name

## No Queries

If there is no query, it might be due to one of the following reasons:

* **Query Age:** The queries in question are older than 30 days.
* **Excluded Sources:** These are read queries generated by BI tools or other service accounts that have been intentionally excluded.
* **Relevance:** Some queries have been flagged as irrelevant, either because they are too simple or exceptionally large in scope.

## Comments

To easily collaborate and communicate between users, each table has a Comments section.

<figure><figcaption></figcaption></figure>

## History

Catalog records metadata changes on the **History** tab. See [History][] for what is logged and how stewards use the timeline for governance reviews.

<figure><figcaption></figcaption></figure>

## Data Quality

See data quality when browsing tables, along with a detailed breakdown of the tests and statuses.

<figure><figcaption></figcaption></figure>

[Read Me tab]: https://docs.coalesce.io/docs/catalog/assets/tables#tab-read-me
[Check out how pinned assets work]: https://docs.coalesce.io/docs/catalog/document-your-data/pinned-assets
[History]: /docs/catalog/collaborate/history

---

## Catalog Onboarding Guide

This guide walks you through onboarding to Coalesce Catalog, from account creation through organization-wide adoption. Whether you're an admin setting up integrations or an end user exploring your data, you'll find the steps you need here.

## Who This Guide Is For

Catalog onboarding involves two main roles:

* **Admins and setup owners:** Request accounts, connect warehouses and BI tools, configure integrations, and manage users. See [Phase 1: Initial Setup](#phase-1-initial-setup).
* **End users:** Sign in, explore the interface, and use Catalog to find and understand data. See [Sign In and Explore](#sign-in-and-explore).

## Phase 1: Initial Setup

### Request a Catalog Account

New Catalog accounts are created by the Coalesce team and are not self-service. Contact [support@coalesce.io](mailto:support@coalesce.io) or your Coalesce representative to request a new Catalog account.

Include in your request:

* Company or organization name
* Key user email addresses
* Integrations needed (for example, Snowflake, Sigma, Tableau, Coalesce Transform)
* Timeline and any organizational context (for example, merger, hierarchy change)

Your Catalog admin or Coalesce contact will confirm when your space is ready. For details on user and team provisioning, including SCIM sync, see [User and Team Provisioning][].

### Access and Authentication

* **Basic login:** Start with the credentials provided in your onboarding email. Sign up using the [Catalog sign-up link][] after your space is ready.
* **SSO configuration:** Work with your IT team to set up Single Sign-On for broader access. See [Authentication][] for Okta, Google SSO with SAML, and Azure AD with SAML.
* **SCIM setup:** For automated user provisioning (recommended for larger teams), see [SCIM setup for Microsoft Entra ID][] and [SCIM setup for Okta][].

### Connect Your Data Sources

Once your Catalog space is ready, admins connect data sources so users can discover and understand assets. The core setup has three steps: sign up, connect your warehouse, and connect your BI tool.

#### Step 1: Sign Up

After your Catalog space is set up, sign up using the [Catalog sign-up link][].

#### Step 2: Connect Your Warehouse

Catalog connects to your warehouse to extract metadata (table and column names, types, lineage, and so on). The dedicated Catalog user has very limited access: it can read metadata only, not your data.

1. Go to the [Integrations][integrations] page, find your warehouse technology, and follow the setup instructions. Each integration requires a dedicated user or service account with specific permissions.
2. In Catalog, go to **Settings** > **Integrations** and add your warehouse using the credentials from step 1.

Catalog tests the connection, applies settings, and runs the first sync (usually a couple of hours). You'll be notified when the first sync finishes. After that, Catalog syncs once per day with your warehouse.

#### Step 3: Connect Your BI Tool

Catalog ingests metadata from your BI tools (dashboards, reports, data sources) to power discovery and lineage.

BI tool onboarding follows one of two paths:

* **Catalog managed:** You share admin credentials; Catalog handles extraction and daily syncs. No further action on your part.
* **Client managed:** You run the `castor-extractor` package locally to extract metadata and send the output files to Catalog. Catalog never accesses your BI tool directly.

For full details and supported technologies, see [Warehouses][] and [BI Tools][].

#### Step 4: Connect Other Integrations

Catalog supports transformation tools (Coalesce, dbt), quality tools, communication (Slack, Microsoft Teams), knowledge bases, and more. See the [Integrations][integrations] page for the full list.

### User Management

Assign roles and invite your core team before broader rollout.

* **Admin setup:** Designate 2–3 catalog administrators. See [User roles in Catalog][] for role capabilities.
* **Initial users:** Invite your core data team. See [User and Team Provisioning][] and [Teams in Catalog][].
* **Role assignment:** Set up viewer, contributor, steward, and admin roles. See [User roles in Catalog][] for the full role matrix.

## Phase 2: Foundation Building

### Domain Structure

Create your organizational domains in the [Knowledge section][]. See [Step 1: Create your domains][] for a guided approach.

* **Business domains:** Finance, Marketing, Operations, and so on
* **Technical domains:** Data Engineering, Analytics, Governance
* **Hierarchical structure:** Use parent-child relationships where appropriate. See [Step 1: Create your domains][] for structuring guidance.

### Governance Framework

Define how you tag, own, and document assets consistently.

* **Tagging strategy:** Define consistent tags for PII, data quality, certification status. See [Step 2: Tag your assets][] and [Tags][].
* **Ownership assignment:** Use the [Metadata Editor][] for bulk ownership assignment. If admins enable **custom ownership labels** under **Account Configuration > Ownership Labels**, follow [Assign Custom Ownership Labels in Bulk][] to update existing rows so owners do not stay on **No label**. See [Step 4: Assign ownership][] for governance concepts.
* **Templates:** Create documentation templates for consistency. See [Templates][].
* **Metadata audit trail:** Use the [History][] tab on tables, dashboards, and knowledge pages to review who changed owners, tags, descriptions, and related metadata during rollout.

### Initial Documentation

Start with your most critical assets. See [How to approach a documentation initiative][] for the full 5-step process.

* **Key tables:** Document your 10–15 most important data tables. Use [Describe with AI][] or [Upload existing descriptions][] if you have docs elsewhere.
* **Critical dashboards:** Add descriptions to your most-used reports. See [Dashboards][].
* **Glossary terms:** Define 5–10 essential business terms. See [Step 3: Create key metrics and glossary terms][].
* **Metrics:** Document key KPIs and their calculations. See [Step 3: Create key metrics and glossary terms][].

## Phase 3: Rollout Strategy

### Phased User Rollout

Follow this recommended sequence:

#### Technical Teams

* Data Engineers
* Data Analysts
* BI Developers

#### Data Stewards

* Domain experts
* Business analysts
* Governance team members

#### Business Users

* Report consumers
* Self-service analytics users
* Executive stakeholders

### Docs by Role

What each role needs to read to use Catalog:

**Viewer (read-only):** Search, view assets, add comments, request descriptions.

* [5-Minute Quick Start Guide][]
* [Search][]
* [Advanced Search][]
* [AI Assistant introduction][]
* [Tables][]
* [Lineage][]

**Contributor:** Everything Viewer can do, plus edit descriptions, create knowledge pages, edit tags, certify owned assets, publish to Marketplace.

* All Viewer docs above
* [Describe with AI][]
* [Templates][] (apply when documenting)
* [Knowledge][]
* [Catalog and Marketplace Interfaces][]
* [Source tags][]

**Steward:** Everything Contributor can do, plus certify any asset, manage tags and templates, bulk edit, analytics.

* All Contributor docs above
* [Metadata Editor][]
* [Tag Manager][]
* [Data Documentation Coverage][]
* [Users Activities][]

**Admin:** Everything Steward can do, plus manage users, content access, data sources, account settings.

* All Steward docs above
* [Quick Set Up][]
* [Integrations][integrations]
* [User and Team Provisioning][]
* [Assets Access Control][]
* For hierarchy planning and visibility triage, see [Choose where to restrict access in Catalog][choose restrict access] and [Troubleshooting visibility for asset access][asset access troubleshooting] on the [Assets Access Control][] page.
* [User roles in Catalog][]
* [Tag Settings][]
* [AI Administration][]

## Phase 4: Adoption and Growth

### AI Features Activation

AI features help users find data faster, document at scale, and lower the barrier for business users who prefer natural language over filters and keywords. AI Search lets people ask questions like "What is ARR?" or "Do we have a dashboard about churn?" Describe with AI generates table and column descriptions from metadata so contributors can document assets quickly without writing everything manually.

* **Catalog AI:** See [AI Administration][].
* **AI Assistant:** Set up the AI chatbot for [Slack][] or [Microsoft Teams][].
* **Auto-documentation:** Use [Describe with AI][] to generate initial table and column descriptions.

### Advanced Features

* **Data quality integration:** Catalog integrates with Anomalo, Coalesce Quality, dbt Tests, Great Expectations, Monte Carlo, Sifflet, and Soda. For custom tools, use the [Quality API][]. See [Quality integrations][].
* **Lineage enhancement:** Ensure end-to-end lineage from source to BI. See [Lineage][].
* **Usage analytics:** Monitor catalog adoption and asset popularity. See [Analytics][].

### Continuous Improvement

* **Documentation targets:** Set goals for description coverage (aim for 80%+). See [Step 5: Set targets and execute][].
* **Regular reviews:** Monthly check-ins to assess progress. Use [Data Documentation Coverage][] and [Users Activities][].
* **Feedback collection:** Gather user feedback and iterate.

## Sign In and Explore

Once your space is set up and data sources are connected, end users can sign in and start exploring.

### Log In

1. Go to [https://app.castordoc.com/](https://app.castordoc.com/)
2. Sign in using your SSO or directly using your app credentials
3. If you're having trouble, contact your Catalog admin or [support team](#get-help). For network or login issues, see [Browser login troubleshooting][]

### Explore the Interface

Catalog has two main interfaces: the **Catalog** (all assets and documentation) and the **Data Marketplace** (curated, consumer-friendly view of production-ready data products). Most use cases refer to the Catalog interface.

Key areas to explore:

* **Left panel:** Browse your data ecosystem by Warehouse, BI tool, Business Knowledge, and People. See [Navigation menu][] for details.
* **Search bar:** Use [Advanced Search][] and filters to find assets. Try terms like `revenue`, `orders`, or `customer`, and use filters (type, tags) to narrow results.
* **Asset zoom:** Click into a table or dashboard to see metadata such as lineage, descriptions, and popularity.
* **AI Assistant:** Ask questions in natural language, for example: "Do we have a dashboard about churn?" See [AI Assistant introduction][] for capabilities.

For a visual walkthrough, see the [5-Minute Quick Start Guide][].

## Catalog Versus Marketplace

The **Catalog** is the primary interface where you find all assets from your data stack and documentation. The **Marketplace** is a curated, consumer-friendly interface that showcases only production-ready data products organized by domain.

If your account has Marketplace access, see [Catalog and Marketplace Interfaces][] for setup and publishing guidelines.

## Best Practices

### Documentation Strategy

Focus on high-impact assets first and build from there.

1. **Start small:** Begin with one domain or use case. See [Step 1: Create your domains][].
2. **Use templates:** Create consistent documentation formats. See [Templates][].
3. **Leverage AI:** Use [Describe with AI][] as starting points.
4. **Link assets:** Pin related dashboards, tables, and metrics together. See [Pinned assets][].

### Change Management

Build momentum and adoption across your organization.

1. **Champion network:** Identify power users in each domain.
2. **Success stories:** Share early wins to build momentum.
3. **Regular communication:** Keep stakeholders informed of progress.
4. **Training materials:** Develop internal documentation and videos. Reference [5-Minute Quick Start Guide][] and [How to approach a documentation initiative][].

### Technical Considerations

Plan for security, performance, and reliability.

1. **Network policies:** Allowlist Catalog IPs if needed. See [Snowflake setup][] for IP addresses.
2. **Security reviews:** Work with InfoSec on data classification.
3. **Performance monitoring:** Track integration sync times and success rates in [Integration settings][].
4. **Backup plans:** Document rollback procedures for integrations.

## Success Metrics

Track these KPIs to measure onboarding success:

* **User adoption:** Number of active users per month. See [Users Activities][].
* **Content coverage:** Percentage of assets with descriptions. See [Data Documentation Coverage][].
* **Search usage:** Frequency of catalog searches.
* **Documentation quality:** User ratings of asset descriptions.
* **Time to value:** How quickly new users find relevant data.

## Get Help

### Support Channels

Reach out through these channels depending on your need.

* **Catalog support:** [support@coalesce.io](mailto:support@coalesce.io)
* **In-app support:** Use the Intercom chat on your Catalog instance. See [How to reach us][].
* **Customer Success:** [support@coalesce.io](mailto:support@coalesce.io)
* **General Coalesce support:** [support@coalesce.io](mailto:support@coalesce.io) or the [Support Portal][]

### Common Issues and Solutions

If you run into problems, these resources can help.

* **SSO problems:** Check with IT team on SAML configuration. See [Authentication][].
* **Integration failures:** Verify network connectivity and credentials. See [Warehouses][] and [BI Tools][].
* **Slow adoption:** Focus on high-value use cases first. See [How to approach a documentation initiative][].
* **Documentation gaps:** Use [Describe with AI][] and [Templates][].

## What's Next?

* [Upload existing descriptions][] if you have table or column docs elsewhere
* [Explore Catalog][] to search, discover lineage, and document your data
* [AI Assistant introduction][] to use AI Search, SQL Copilot, and Dashboard Q&A
* [User and Team Provisioning][] for SCIM, Okta, and Microsoft Entra ID setup
* [Step 5: Set targets and execute][] to define and track documentation goals

---

[Catalog sign-up link]: https://app.castordoc.com/auth/sign-up
[User and Team Provisioning]: /docs/catalog/administration/user-and-team-provisioning/
[Authentication]: /docs/catalog/administration/authentication/
[SCIM setup for Microsoft Entra ID]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-microsoft-entra-id
[SCIM setup for Okta]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-okta
[integrations]: /docs/catalog/integrations
[Search]: /docs/catalog/navigate/search/
[Tables]: /docs/catalog/assets/tables
[Knowledge]: /docs/catalog/assets/knowledge
[Source tags]: /docs/catalog/document-your-data/tags/source-tags
[Tag Manager]: /docs/catalog/document-your-data/tags/tag-manager
[Quick Set Up]: /docs/catalog/quick-set-up
[Assets Access Control]: /docs/catalog/administration/assets-access-control
[choose restrict access]: /docs/catalog/administration/assets-access-control#choose-where-to-restrict-access-in-catalog
[asset access troubleshooting]: /docs/catalog/administration/assets-access-control#troubleshooting
[Tag Settings]: /docs/catalog/administration/tag-settings
[Warehouses]: /docs/catalog/integrations/data-warehouses/
[Snowflake setup]: /docs/catalog/integrations/data-warehouses/snowflake
[BI Tools]: /docs/catalog/integrations/data-viz/
[Slack]: /docs/catalog/integrations/communication/slack
[Microsoft Teams]: /docs/catalog/integrations/communication/microsoft-teams
[User roles in Catalog]: /docs/catalog/administration/user-roles-in-catalog
[Teams in Catalog]: /docs/catalog/administration/teams-in-catalog
[Knowledge section]: /docs/catalog/use-cases/discover/knowledge-section
[Step 1: Create your domains]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-1-create-your-domains
[Step 2: Tag your assets]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-2-tag-your-assets
[Step 3: Create key metrics and glossary terms]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-3-create-key-metrics-and-glossary-terms
[Step 4: Assign ownership]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-4-assign-ownership
[Step 5: Set targets and execute]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-5-set-targets-and-execute
[Tags]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/tags
[Metadata Editor]: /docs/catalog/document-your-data/metadata-editor
[History]: /docs/catalog/collaborate/history
[Assign Custom Ownership Labels in Bulk]: /docs/catalog/document-your-data/metadata-editor#assign-custom-ownership-labels-in-bulk
[Templates]: /docs/catalog/document-your-data/templates
[How to approach a documentation initiative]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/
[Describe with AI]: /docs/catalog/document-your-data/catalog-scribe/describe-with-ai
[Upload existing descriptions]: /docs/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions
[Dashboards]: /docs/catalog/assets/dashboards
[AI Administration]: /docs/catalog/administration/ai-administration
[Quality integrations]: /docs/catalog/integrations/quality/
[Quality API]: https://apidocs.castordoc.com/#e4604160-95c5-4f6d-88ce-7908f5e13686
[Lineage]: /docs/catalog/use-cases/discover/lineage
[Analytics]: /docs/catalog/administration/analytics/
[Data Documentation Coverage]: /docs/catalog/administration/analytics/data-documentation-coverage
[Users Activities]: /docs/catalog/administration/analytics/users-activities
[Browser login troubleshooting]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Navigation menu]: /docs/catalog/use-cases/discover/navigation-menu
[Advanced Search]: /docs/catalog/use-cases/discover/advanced-search
[AI Assistant introduction]: /docs/catalog/ai-assistant/introduction
[5-Minute Quick Start Guide]: /docs/catalog/use-cases/five-minute-quick-start-guide
[Catalog and Marketplace Interfaces]: /docs/catalog/navigate/catalog-and-marketplace-interfaces
[Pinned assets]: /docs/catalog/document-your-data/pinned-assets
[Integration settings]: /docs/catalog/administration/integration-settings
[How to reach us]: /docs/catalog/support/how-to-reach-us
[Support Portal]: https://support.coalesce.io
[Explore Catalog]: /docs/catalog/use-cases/five-minute-quick-start-guide

---

## Changelog

## **07-08-2026**

### **Automated Snowflake Sync-Back**

Based on your feedback, we've improved Snowflake sync-back to make documentation maintenance even easier.

This update brings:

- Catalog descriptions are now synchronized to Snowflake automatically on a daily basis, without manual sync requests.
- Sync-back now applies to all cataloged Snowflake assets: tables, views, dynamic tables, and all their columns.

Write documentation once in Catalog and keep it available in Snowflake automatically.

Check out the [documentation](https://docs.coalesce.io/docs/catalog/integrations/sync-back/sync-back-to-snowflake) to set it up.

## **05-27-2026**

### **Cross-warehouse indirect lineage**

When pipelines span multiple warehouses through an intermediate storage layer, Catalog can now stitch their lineage into one end-to-end path instead of isolated segments per system.

Coalesce configures downstream warehouses to inherit upstream lineage for your environment. Multi-warehouse teams get full visibility where the graph previously broke at each handoff.

## **05-18-2026**

### **Snowflake: Dynamic Tables lineage now supported**

You can now keep full lineage visibility and SQL source queries for Snowflake Dynamic Tables.

To enable it, make sure the required Snowflake permissions are configured in your environment. See [Grant Monitor on Dynamic Tables](/docs/catalog/integrations/data-warehouses/snowflake#26-grant-monitor-on-dynamic-tables).

## **05-15-2026**

### **Catalog MCP: New lineage tools**

The Coalesce Catalog MCP server now includes a `get_field_lineage` and `get_asset_lineage`tool, bringing column-level and asset-level lineage traversal directly into your AI assistant.

## **05-07-2026**

### **New native integration: Omni connector now available**

We're excited to announce the release of our Omni connector, the newest addition to our suite of BI tool integrations.

The connector covers the core capabilities Omni users need to govern Workbooks, Queries, and Dashboards in Coalesce Catalog.

## **03-30-2026**

### **Power BI: Lineage now supports Dataflows**

Using Power BI [Dataflows](https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-introduction-self-service) could break lineage, as intermediate layers were not processed.

We now support **asset-level and column-level lineage through Dataflows.**

## **03-13-2026**

### **Power BI: Improved lineage for renamed fields**

Previously, renaming columns in Power BI could break lineage, as transformed field names no longer matched their original source columns.

We've improved Power BI lineage to handle renamed fields, preserving column-level lineage between Power BI and upstream data sources.

Supported rename methods include:

- **`Table.RenameColumns` with a direct mapping** — for explicit column renames defined in Power Query M

- **`Table.RenameColumns` with multiple rename pairs** — for transformation steps that rename several columns at once

### **Domo connector: Cards now available in Coalesce Catalog**

We've enhanced the Domo connector to catalog Domo Cards, making them fully governable and accessible via the AI Assistant.

## **03-09-2026**

### **API: Data Products & Pinned Assets Endpoints**

We've expanded the capabilities of our **Public Catalog API** with **four new endpoints**.

**New endpoints:**

- **[Get Data Products](https://apidocs.castordoc.com/#8917c306-c3c8-4169-9a6d-be8ac01f4af3)** — Retrieve all assets marked as Data Products in the Coalesce Catalog

- **[Get](https://apidocs.castordoc.com/#9b8ce3ca-abd2-4ccb-8961-de3c55886da6) | [Upsert](https://apidocs.castordoc.com/#9fca9370-1623-41f0-9cc5-6cde99506cf0) | [Remove](https://apidocs.castordoc.com/#40dccd05-f1b8-4d24-8595-7cea64b2f87f) Pinned Assets** — Fully manage Pinned Assets through the API

**Reminder:** You can test Catalog API requests directly in the [**in-app Catalog API Playground**](https://app.castordoc.com/settings/public-api-settings), which lets you run live queries against your environment and explore available endpoints before building integrations.

## **03-06-2026**

### 🚀 **New: Self-service API token management**

Admins can now **generate and manage Catalog API tokens directly from the application**, without needing to contact Support.

You'll find a new **API tab in Settings**, where you can:

- Generate a new API token
- Rotate your token when needed
- Revoke access instantly

<figure><figcaption></figcaption></figure>

## **02-27-2026**

### **🚀 Create custom Ownership Labels in the Catalog**

Ownership is a core concept: every asset must have a clearly identified accountable owner (documentation, technical expertise, business context, etc.).

Previously, ownership was **flat** (all owners shared the same label), which made responsibilities ambiguous and impacted governance.

We now introduce **Custom Ownership Labels**, so each organization can define label ownership (e.g. *Technical*, *Business*, *Domain Owner*) and make responsibilities explicit and operational.

<figure><figcaption></figcaption></figure>

<details>

<summary>How to get started?</summary>

**#1 Create labels (Admin)**

- Go to Account Configuration → Ownership Labels
- Create the labels you want (e.g. *Technical*, *Business*, *Domain*, etc.)

**#2 Assign labels to existing owners (Bulk update)**

- Open Metadata Editor
- Filter by Owner
- Select the assets for a given owner
- Use Replace owner → choose the same owner but with the right label
- Repeat for each owner you want to label

**Without step #2, existing owners will appear in the application as "No label" owners**

**#3 Verify**

- Anywhere owners appear (Details panel, AI Assistant, …), owners will now be **grouped by label**
- In the future, each time an owner is added to an asset, a label must be assigned.

<VideoPlayer url="/img/catalog/assets/label-setup.mov" subtitle="" />

</details>

## **02-12-2026**

### **Apply changes to all matching assets in the Metadata Editor**

You can now apply metadata updates to all assets matching your filters, not just the first 500 loaded results.

Keep metadata consistent at scale and eliminate repetitive bulk updates.

**How does it work?**

- When filtering and results exceed what's loaded, "Select all matching assets" triggers Apply All (runs in the background)
- Progress/status is visible in the Metadata Editor header

<VideoPlayer url="/img/catalog/assets/apply-all.mov" subtitle="" />

## **12-08-2025**

### 🚀 **New: MCP Server for Coalesce Catalog**

You can now connect your favorite AI tools directly to Coalesce Catalog using the Model Context Protocol (MCP).

**What it does**

The MCP server exposes 8 tools that let external AI assistants search and retrieve information from your catalog:

- 🔍 **Search tools:** Find tables, dashboards, columns, and glossary terms using the same search engine as our AI Assistant

- 📚 **Catalog tools:** Access company context, tags, users, and teams

**Why it matters**

Build your own data agents with your preferred AI tools (Claude, Cursor, Dust, and other MCP-compatible clients). Combine Coalesce Catalog with other MCP servers like dbt, Looker, or Snowflake to create powerful, customized workflows.

**⚡ How to get started**

1. Generate an API token in your account settings

2. Configure your MCP client with the server URL: `https://api.castordoc.com/mcp/server`

3. Start querying your catalog through natural conversation

📖 See the [full setup documentation](https://docs.coalesce.io/docs/catalog/developer/mcp-integration) for detailed instructions.

## **12-04-2025**

### 🕓 **Full Metadata History**

The History tab now logs all metadata changes on assets.

**Why you'll care?**

- **Traceability for governance:** Get a complete audit trail of who changed what and when on tables, dashboards, and knowledge pages.

- **Smoother collaboration:** Everyone can quickly verify updates in one place, aligned with what notifications show. No more confusion.

<figure><figcaption></figcaption></figure>

## **11-28-2025**

### 🖼️ **Data Product Images in Catalog Marketplace**

You can now add an image to every Data Product in the Catalog Marketplace — a feature many of you have been asking for since the Marketplace launch to make it nicer to scan.

Images can be added by uploading your own or generating one with AI.

<figure><figcaption></figcaption></figure>

### 🧪 **In-app Catalog API Playground**

Make it easier and faster for teams to start using the Catalog API directly from the product.

🎯 **What it does:**

- **New API playground in the app** where users can experiment with the Catalog API in a safe, guided way
- Users simply **paste the API token** provided by the catalog support team and can immediately run real requests
- A **pre-configured example query** is available out of the box—click "Run" to see live results right away
- The **query editor supports autocomplete** (for example, when typing `folders` and related paths), helping users discover the right endpoints and parameters much faster

💡 **Why it matters:**

- **Reduces friction** for developers and data teams to start integrating with the Catalog API
- **Accelerates API-based initiatives** by shortening the time from idea to first successful request

🚧 **Coming soon**

- Admin users will be able to **generate API tokens directly in the app**, then immediately test them in the playground with pre-written queries
- This will provide a **fully self-service API onboarding flow**: generate token → test requests live → integrate into your own workflows

![Catalog API Playground](/img/catalog/assets/api-playground.png)

## **11-07-2025**

### 🔍 **Search Assistant - Natural Language Filtering**

Filtering capabilities are now released in the Search assistant!

🎯 **What's new:**

Users can now filter search results directly through natural language queries, without needing to switch to manual filters. The assistant automatically understands and applies filters based on:

* **OWNER** – Filter by business owner (partial or full match)
* **TAG** – Filter by asset tags
* **NAME** – Filter by asset name (partial or full match)

💡 **Examples:**

* "Which tables are owned by Alice?"
* "Show me dashboards tagged GDPR"
* "Find assets named customer"

This makes the Catalog fully explorable through conversation and significantly speeds up asset discovery! 🚀

![Search Assistant Filtering](/img/catalog/assets/filtersonNLP.png)

## **10-27-2025**

### 🔍 **Metadata Editor - Filter by Upstream Table**

Enhanced the metadata editor with powerful lineage-based filtering capabilities to streamline bulk metadata operations.

🎯 **What it does:**

* Added "Ancestors" filter in the metadata editor's table tab
* Filter tables by their ancestors using lineage relationships
* Supports case-insensitive search with flexible naming formats (`database.schema.table` or `database schema table`)
* Considers all lineage types: `AUTOMATIC`, `MANUAL_CUSTOMER`, `MANUAL_OPS`, and `OTHER_TECHNOS`

💡 **Why it matters:**

* Enables data governance managers to efficiently propagate ownership and tags downstream
* Eliminates manual navigation through lineage tabs for each table
* Leverages existing lineage relationships for smarter bulk metadata operations
* Streamlines governance workflows by focusing on dependent assets

### 🗃️ **Metadata Editor - Hierarchical Source/Database/Schema Selection**

Improved the metadata editor with intelligent cascading filters for better navigation and selection accuracy.

🎯 **What it does:**

* Database options now depend on selected sources
* Schema options now depend on selected databases
* Creates a logical hierarchy: Source → Database → Schema
* Prevents invalid combinations and reduces selection errors

💡 **Why it matters:**

* Provides more intuitive and logical navigation through data assets
* Reduces user errors by showing only valid options at each level
* Improves efficiency when working with large, multi-source environments
* Creates a more structured approach to metadata management
## **10-20-2025**

### 🔍 Unified Search Experience: ASK + SEARCH in One Place

We've merged AI-powered and keyword-based search into a single, intelligent search bar on the homepage — making it easier than ever to find what you need, however you prefer to search.

![Unified Search](/img/catalog/assets/unifiedsearch.png)

#### 🎯 What it does:

- **Single entry point**: Access both AI Assistant (ASK) and keyword search (SEARCH) from one unified search bar on the homepage
- **Smart persistence**: Your last search mode choice is automatically remembered across sessions
- **Seamless switching**: Toggle between search modes without losing context

#### 💡 Why it matters:

- **Cleaner interface**: No more hunting between two separate search locations — everything you need is in one place
- **Better AI adoption**: Making AI Assistant more discoverable increases usage and helps users get faster, smarter answers
- **Personalized experience**: The system remembers how you like to search, adapting to your workflow

## **10-18-2025**

## 📊 New: Data Quality Dashboard

Monitor your data quality health across Transform and Catalog integrations in one unified view.

### 🎯 What it does:

The Data Quality Dashboard aggregates quality tests from Transform and external tools (Monte Carlo, dbt test, Soda, Sifflet) into a single dashboard with two views:

**Overview Tab:**
- Overall test pass rate for all tables
- Tables grouped by status (Alert, Warning, Passed)
- Test coverage metrics showing which tables have tests vs. no tests
- Test distribution by technology (Coalesce, Monte Carlo, dbt, Soda, etc.)

**Result Details Tab:**
- Detailed table list with test results (Failing, Warning, Passing)
- Table popularity metrics
- Direct links to investigate specific assets

**Filtering:**
Filter by Warehouse, Database, Schema, Table Tag, Owner, and Result Status to focus on specific areas.

### 💡 Why it matters:

Get a complete picture of data quality across all your tools without jumping between Transform, Monte Carlo, and dbt dashboards. Quickly identify coverage gaps and quality issues to report to leadership.

![Data Quality Dashboard](/img/catalog/assets/DQnew.png)

## **10-17-2025**

### 🎉 **New native integration: Count connector now available**

We're thrilled to announce the release of our **Count connector**, the newest addition to our suite of BI tool integrations.

## **10-15-2025**

### 🔧 **API Public Types Alignment**

We've aligned our public API types for better consistency and accuracy.

🎯 **What changed:**

* Changed `nbPerPage` parameter from Float to Int type
* Changed `page` parameter from Float to Int type

💡 **Why it matters:**

* Ensures type consistency across our API endpoints
* Provides better validation for pagination parameters
* Improves developer experience with more accurate type definitions

## **10-13-2025**

### 🚀 **New: Sigma Data Models are now supported in our Sigma connector!**

Our Sigma connector just got an upgrade: it now supports Data Models, Sigma's asset type introducing a semantic layer for organizing and reusing data.

## **10-09-2025**

### 🎛️ **Admin Widget Settings**

Admins now have full control over widget visibility on the homepage, allowing for a more customized user experience.

🎯 **What it does:**

* Admins can activate or deactivate widgets for all users from the admin settings
* When widgets are deactivated, users get a sleek, focused experience centered on search and AI assistant
* When enabled, users can choose which widgets they want to see on their homepage
* Provides flexibility to tailor the interface based on organizational preferences

💡 **Why it matters:**

* Gives admins control over the user interface and experience
* Allows organizations to create a more focused, distraction-free environment when needed
* Maintains user choice and customization when widgets are enabled

<Loom id="eaf7a2dc187d4010a61a1fa77e190800" />

## **10-03-2025**

### 📝 **Company Context for AI Descriptions**

Make AI-generated knowledge descriptions more accurate and relevant by adding business context.

🎯 **What it does:**

* Admins can now set a short company & industry description in **Settings → AI System**
* This context is automatically included (by default) when generating new knowledge descriptions
* Users can choose to include or exclude the company context per description

💡 **Why it matters:**

* AI-generated content is now better tailored to your business and industry
* Admins gain more control over how the AI understands their environment

<details>

<summary>Examples</summary>

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

</details>

## **09-04-2025**

### **📝 Bulk Edit Column Descriptions**

Quickly apply the same description to multiple columns in Catalog.

* Search, filter, and select similar columns
* Use smart suggestions or write your own
* Save time and ensure consistency

Watch the [demo](https://www.loom.com/share/51e002591585417fa5e6b99c3a101d5b?sid=703a870a-6748-46a9-a478-25e2ca7823e5)

**Live now** for all Catalog customers

<Loom id="51e002591585417fa5e6b99c3a101d5b" />

### New: Data Quality Dashboard 📊

**You have Data Quality integrations?** If yes,

Discover a unified dashboard that brings ALL your data quality metrics into one place - no more jumping between Transform, Monte Carlo, dbt, and other tools to get the full picture of your data health.

:dart: **What it does:** This new dashboard aggregates quality signals from multiple sources into a single, executive-friendly view. You'll see:

* Overall data quality health across your entire data ecosystem
* Test coverage and pass rates from all your quality tools (Transform, Monte Carlo, dbt test, Soda, etc.)
* Assets grouped by health status (failing, warning, passing, no tests)
* Powerful filtering by warehouse, database, schema, tags, and owner
* Quick drill-down to investigate specific issues

<figure><figcaption></figcaption></figure>

## **07-31-2025**

### New: Data Marketplace is Live! 🛍️

Turn your Catalog content into a **business-friendly experience** for data consumers.

The **Data Marketplace** allows you to showcase **consumption-ready data assets (Data Products)** in a clear, curated view — organized by domain and easy to explore.

**Built for data consumers** to discover valuable resources and for **data builders** to boost the visibility and adoption of their work.

What to do next:

**✔️** Set clear publishing guidelines for your team

**✔️** Select & document the right assets for the Marketplace

**✔️** Start sharing your Marketplace with business users!

<figure><figcaption></figcaption></figure>

_The marketplace feature isn't currently available to all accounts. Please reach out if you'd like to activate it_

## **07-29-2025**

### **Bulk Tag Columns in Metadata Editor**

You can now tag multiple columns at once directly from the metadata editor—making it faster to manage sensitive data and scale your governance workflows.

#### **🔍 Use Cases**

* Tag all columns matching a pattern (e.g. `phone`) as `PII`, `sensitive`, and more
* Use filters to target columns by tag, data type, documentation status, primary key presence, and more
* Apply tags across entire domains in one click for streamlined governance

### **New Assistant Intelligence: Disambiguate User Questions**

Catalog Assistant now helps users clarify ambiguous questions by proactively offering follow-up suggestions.

#### **🔍 What’s New**

When users ask vague queries like “How are we doing this month?”, the Assistant now responds with clarifying options (e.g., “Are you asking about sales, engagement, operations?”).

#### **🤔 Why it matters**

* Many user questions are underspecified and led to irrelevant or partial answers
* Disambiguation improves the quality of answers and builds trust in the Assistant
* Makes the Assistant feel more conversational and helpful—just like a real analyst would

## 07-24-2025

### Knowledge Duplicate Warner

Avoid duplicate knowledge pages by flagging existing similar content _as you're creating a new one_.

Use Case

* Say you're about to create a knowledge page on Revenue. You type in the title…
* A warning appears: _“There are 5 existing pages with similar names.”_
* You can then: Review existing pages, Edit one of them instead, Avoid adding redundant content

This helps to

* Keep knowledge clean and well-organized
* Prevent confusion and rework
* Minimize governance overhead for data teams

<figure><figcaption></figcaption></figure>

## 07-15-2025

### :mag_right: New Filters in advanced search

You can now choose AND/OR behavior in the Advance Search Filters for Owners and Tags

### 📊 Analytics: **Column Monitoring**

**Find it** [**here**](https://app.castordoc.com/governance/analytics/data?column_name=\&column_tag=\&database=\&owner=\&schema=\&tab=47-column-monitoring\&table_tag=\&warehouse=)

Gain visibility over your column metadata at scale:

* Total number of columns
* Description coverage by percentage
* Count of distinct column names
* Distribution of column name occurrences (e.g., "C Phone" appears in 2,500+ tables)

Clicking a column name redirects you to its detailed view.

### 🔍 Analytics: **Column Details**

**Find it** [**here**](https://app.castordoc.com/governance/analytics/data?column_name=\&column_tag=\&database=\&owner=\&schema=\&tab=48-column-details\&table_tag=\&warehouse=)

Dive deeper into individual columns:

* See which tables use a specific column
* View column descriptions directly
* Prepare for upcoming support for **inline column metadata editing**

### 🏷️ Analytics: **Column Tag Filtering**

**Find it** [**here**](https://app.castordoc.com/governance/analytics/data?column_name=\&column_tag=\&database=\&owner=\&schema=\&tab=47-column-monitoring\&table_tag=\&warehouse=)

You can now **filter columns by tags**—e.g., see all columns tagged as `sensitive:email`.
Quick stats are available:

* Total tagged columns
* Distinct column names
* Distribution breakdown

## 06-19-2025

### **🎉 New native integration: Strategy connector now available**

We're excited to announce the launch of our Strategy connector, the latest addition to our suite of BI tool integrations.

With this new Strategy connector, you can:

* Seamlessly connect your Cubes, Reports, Documents and Dashboards to our platform
* Gain visibility into column-level lineage, providing a clear understanding of your data's journey
* Access all relevant metadata and get instant answers to your questions directly within Strategy, thanks to our  Chrome Extension that appears on each asset

Check out our [documentation](/docs/catalog/integrations/data-viz/strategy) for setup instructions!

## 04-17-2025

### Introducing **Catalog’s AI Assistant in Microsoft Teams**

**Get instant data insights right from your everyday communication tool!**

* **How it works:** Chat 1-on-1 or mention `@Coalesce` in a group conversation to instantly get trusted answers to questions like:
  • What is ARR?
  • What dashboards track user activity?
  • How is customer margin calculated?
* **Stay focused:** No need to switch tools, get the insights you need without ever leaving Microsoft Teams
* **Extra support, anytime:** The data team is always here if you need a hand

<figure><figcaption></figcaption></figure>

## 04-16-2025

#### Mentions users

You can mention users using the "@" handle. They will receive a notification of mention.

<figure><figcaption></figcaption></figure>

#### New Homepage

The Homepage now allow:

* Each user can pick the relevant widget they want to see
* The Admins can add company news

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

#### Dashboard & Table Description AI Generator Upgrade

AI generated description now leverage all available metadata for Tables and Dashboards, now 100% of them can be described with AI, it used to be 15%

## 03-26-2025

#### New Role Data Steward

How to use it? A steward can do everything an admin can, except user and integration management.

<figure><figcaption></figcaption></figure>

<ButtonLink to="/docs/catalog/administration/user-roles-in-catalog">
  User Roles in Catalog
</ButtonLink>

## 02-28-2025

### Show Quality issues on Dashboards

If a dashboard is using tables with failing quality tests, Catalog will show it in the app and the chrome extension.

<figure><figcaption></figcaption></figure>

### Export all downstream from a given column

We've offered for a few years the ability to download table downstream nodes list, now you can do with any given column. You want to know the impact of changing one single column? Simply click the button and you'll get the list of downstream columns and dashboards using it.

<figure><figcaption></figcaption></figure>

### API for SQL Assistant

We enabled you to to build your own SQL assistant using our building blocks. How to use it? [Here is the public code repository](https://github.com/castordoc/api-playbook) that can be shared with customers. It relies on existing and new API endpoints available here [https://apidocs.castordoc.com/](https://apidocs.castordoc.com/)

### Pinned Assets: Suggestions by AI

How to use it? Simply open the Pin Asset, you'll get suggestion of assets to pin based on title and nam. [Updated documentation for asset pinning is here](/docs/catalog/document-your-data/pinned-assets)

<figure><figcaption></figcaption></figure>

### Looker Studio Native Integration

The Looker Studio connector is now live in Catalog! This is a big deal because Catalog is the only one on the market to offer this integration.

[Here](/docs/catalog/integrations/data-viz/looker-studio) is how to set it up

### UI Improvements

* Editing Readme now sits behind a dedicated button to avoid misclicks
* The menu spacing has changed to allow for more space
* We’ve also updated the text block color and background in the rich text (Readme).

## 01-23-2025

### **ThoughtSpot connector: govern your Answers with** Catalog

We have enhanced our ThoughtSpot connector to enable you to **govern Answers** directly within Catalog.

You can now:

* Track the **popularity** of each Answer
* View **descriptions** and **tags**, automatically retrieved from ThoughtSpot
* Visualize the **complete lineage** between Datasource ↔ Answer

## 01-17-2025

### Knowledge Pages: simplified creation and more intuitive navigation

We’ve worked to make your _Knowledge Pages_ even easier to manage and explore.
Discover the new features:

1️⃣ **New view for sub-pages**

The sub-pages of a given page are now displayed in the **“Sub-pages & Map”** tab. This provides a clear and organized view of your documentation structure

2️⃣ **Simplified sub-page creation**

Creating sub-pages has never been faster and more accessible:

* From the left menu, by hovering over your assets
* Directly from the **“Sub-pages”** tab of a _Knowledge Page_

<figure><figcaption></figcaption></figure>

## 12-02-2024

### Enhance your "ReadMe" Sections with table elements

You can now enhance the “ReadMe” sections (of all your technical assets and documentation) by adding **tables elements.**

**Why use it?**

* **Boost readability**: organize your content into a structured format
* **Simplify understanding**: present your information concisely for improved comprehension

<figure><figcaption></figcaption></figure>

## 11-25-2024

### Sync Your Notion and Confluence Content with Catalog!

What’s New?
We’ve made it easier than ever to maintain a complete and up-to-date knowledge base by integrating **`Notion`** and **`Confluence`** with Catalog.

Here's what you gain:

* Enhanced AI Assistant: The synced content feeds your AI Assistant, allowing it to provide smarter, more accurate answers to your questions
* Always Up-to-Date Content: Automatic synchronization ensures that your documentation stays accurate, eliminating manual copy-pasting and reducing maintenance efforts

How It Works
Find step-by-step instructions to connect your workspaces here:

* [Confluence Integration Guide](/docs/catalog/integrations/knowledge-bases/confluence)
* [Notion Integration Guide](/docs/catalog/integrations/knowledge-bases/notion)

## 11-25-2024

### Something new in AI-generated descriptions for your assets

Your AI assistant for automatically generating descriptions just got even more powerful!
**What’s changing:** generated descriptions now align with your **templates**
**The result:** your documentation is both:

* **Fully compliant with your internal standards,** thanks to template integration
* **Accurate** (leveraging SQL sources for technical assets and document type for knowledge assets)

<figure><figcaption></figcaption></figure>

## 10-22-2024

### AI Assistant: Improvement of result relevance through the popularity of assets :star:

The search results for the Assistant are evolving!

We are now taking `popularity of assets` into account.

When you ask a question to the AI Assistant, the most popular assets are highlighted more prominently. This helps improve relevance.

What does that mean?

* Popular assets are boosted :up:
* Less popular assets are less visible :arrow_down_small:
* Assets without popularity (such as Knowledge assets) are neither boosted nor diminished in visibility, which means they are treated as equivalent to assets with a popularity rating of 2.5/5.

## 08-31-2024

### Never Lose Track: Unified History of conversation

* Unified Conversation History: Easily revisit past interactions with the Assistant in **both the Chrome Extension and In-App**.
* Seamless Continuity: Access your latest conversations across Dashboard Q\&A, SQL CoPilot, and AI Search, even after navigating away or refreshing the page.

<figure><figcaption></figcaption></figure>

## 07-18-2024

### Real-Time Response Streaming

* Instant Feedback: Watch responses build in real-time as you ask your questions.
* Reduced Wait Times: No more long waits for complete answers—see results immediately.

:point_right: Available on both the Catalog app and Chrome extension.

<figure><figcaption></figcaption></figure>

## 07-12-2024

### Optimize Chrome Extension Usability: Sleek & Mobile in Collapsed Mode!

* Compact Design: The extension now takes up less space in collapsed mode.
* Enhanced Mobility: Easily move the extension around your screen to fit your workflow.

<figure><figcaption></figcaption></figure>

## 06-14-2024

### 🌟 **Introducing SQL Copilot Integration in** Catalog **App!**

#### **What's New**

We are excited to introduce the SQL Copilot feature directly within the Catalog application. While the Chrome extension version remains available, you can now enjoy the convenience of accessing SQL tools without leaving the app!

#### **How it Works**

* **Integrated Access:** SQL Co-Pilot is now available directly in the Catalog app, located next to the AI search tab.
* **Easy Switching:** Seamlessly switch between AI search and SQL Co-Pilot within the same interface.
* **Enhanced Practicality:** Utilize advanced SQL tools directly within Catalog, streamlining your workflow and enhancing productivity.

Experience the enhanced practicality of having SQL Co-Pilot at your fingertips within the Catalog app. Try it out today and see how it simplifies your data analysis tasks!

<figure><figcaption></figcaption></figure>

## 06-12-2024

### Introducing: Dashboards Access Control

Admins can now manage who can access which dashboard folders/subfolders in Catalog!

[More Documentation here](/docs/catalog/administration/assets-access-control#dashboards-access-control)

<figure><figcaption></figcaption></figure>

## 05-21-2024

### Enhanced AI Search for Data Analysts: `Table Search`

#### Previously, our AI could search for `knowledge` assets and `dashboards`. Now, it has improved to support natural language queries specifically about `data tables`.

#### How it Works :arrow_down_small: 1. Natural Language Processing for Table Queries

* Our AI can now understand and interpret specific questions about data tables.
* Simply ask the AI about the table you need, and it will recognize your request.

#### 2. Direct Access to Data Tables

* When you query the AI, it will confirm the existence of the table.
* You'll receive direct links or shortcuts to access the tables quickly _(only in App for the moment)_, eliminating the need to manually search through databases or consult colleagues.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

## 05-06-2024

### Catalo&#x67;**'s AI Assistant now within Slack!**

<figure><figcaption></figcaption></figure>

**What's New:**

* **Effortless Interactions:** Just "ping" the assistant in your #ask-data channel. When you're in a channel, simply type in `@CastorDoc` and ask your question, the Assistant will answer you!
* **Instant Insights:** Access instant, reliable data insights without leaving Slack.
* **Enhanced Support:** The data team is on standby for additional help if needed.

## 03-22-2024

### Introducing templates for assets documentation

Full documentation here: [Templates](/docs/catalog/document-your-data/templates.md)

Define your documentation good practices and must-have. Guide your contributors and improve your overall documentation quality by sharing the same formatting throughout your company.

<figure><figcaption>Create your templates under the Governance page</figcaption></figure>

<figure><figcaption>Then your contributors apply templates to the asset's read-me sections</figcaption></figure>

## 02-20-2024

### Introducing AI-Powered Dashboard Insights in Chrome Extension

Dive deep into the 'hows' and 'whys' behind your data dashboards. Understand not just what your data shows, but how it’s structured and why it matters.

You can ask questions related to those attributes: `Dashboard Name`, `Source Name`, `Technology`, `Type`, `isVerified`, `Folder Path`, `isDeprecated`, `Description User`, `Description External`, `Pinned Assets`and `Mentioned in`

<figure><figcaption></figcaption></figure>

## 02-13-2024

* Delete a team - You can now fully manage teams: create, edit team information, edit members and delete the team

## 01-30-2024

### Introducing Advanced Filtering: Search by Source and Database

Filter your searches more precisely by selecting specific sources and databases.

<figure><figcaption></figcaption></figure>

## 01-17-2024

* UI update on the "Mentioned In" section
* Notification for teams - teams as owner can now get Slack notifications. [Set it up on your account ](/docs/catalog/collaborate/notifications/notifications-for-teams)

Admin features

* Assets access control at Schema granularity
* Tag edition: Color Picker - [more info](/docs/catalog/document-your-data/tags/tag-manager#edit-tags)
*   Integration settings page

    <figure><figcaption></figcaption></figure>

## 12-27-2023

Enriched tooltip for teams as owner

Admin features

* Tag renaming from the Tag Manager page - [More info here](/docs/catalog/document-your-data/tags/tag-manager#edit-tags)
* Reportings: tag column has been split in three columns for each type of tags: User tags, Catalog tags, External tags
* API new endpoints for [user](https://apidocs.castordoc.com/#b3e86b91-5326-4319-8096-19569dfde571) and [team](https://apidocs.castordoc.com/#f4b93b99-48ed-47f8-ad5c-93973cf77d41) management

## 12-13-2023

### Fivetran Integration

Catalog

* represents objects extracted and loaded by each Fivetran connector
* computes lineage between Fivetran assets and the warehouses where tables are loaded

<figure><figcaption></figcaption></figure>

## 12-7-2023

### External Lineage

Your data is sometimes used somewhere else than warehouses or dashboards: ML algorithms, data apps or third party services. Catalog materializes these usages in catalog and lineage.

<figure><figcaption>Representing data apps and reverse ETL reading into a Snowflake table</figcaption></figure>

<ButtonLink to="/docs/catalog/document-your-data/lineage-to-service-accounts">
  Lineage to service accounts
</ButtonLink>

## 12-1-2023

### Table Links

<figure><figcaption>From a table in Catalog, jump back to either Snowflake, Airflow or the URL of your choice 🔗</figcaption></figure>

#### Source Link

BigQuery and Snowflake Tables now have their link to redirect to respectively the BigQuery console or the Snowsight interface.

#### External Links

Tables can be associated with links (Airflow, GitLab, GitHub, other), via the Catalog API.

<ButtonLink to="/docs/catalog/assets/tables">
  Tables
</ButtonLink>

### dbt test managed by Catalog

Catalog now manages the dbt test integration, no more files to push and schedule. All Tables present in Catalog and that dbt tests evaluate will display these tests in their Data Quality tab.

<ButtonLink to="/docs/catalog/integrations/quality/dbt-tests">
  dbt tests
</ButtonLink>

## 11-30-2023

AI Search over your knowledge content. Like a Q\&A, your users get their questions directly answered if the answer is within one or several knowledge pages.

## 11-20-2023

### Asset Access Control

Admins now have a way to manage who can access which database in Catalog!

<ButtonLink to="/docs/catalog/administration/assets-access-control">
  Assets Access Control
</ButtonLink>

Next iteration will introduce assets management at schema granularity!

## 10-18-2023

### InApp Selection of Database and Schemas

It is quite common to have to finetune content extracted from a warehouse. You may want to exclude that particular schema or later add back that other database. You can now do that autonomously in the app.

<ButtonLink to="/docs/catalog/administration/integration-settings">
  Integration settings
</ButtonLink>

### Notifications via Emails

Catalog, while pushing for Slacks notifications, understands that some customers prefer having their notifications via email, it is now available

### Native Column Tags Extraction

* Snowflake and BigQuery column tags are now extracted natively

### Chrome Extension Improvements

* New technologies covered now: Domo, Sigma and Qlik. Refer to the page to get an overview of the capabilities
* You can now see "Mentioned Assets" on the snippet"

### Salesforce Native Integration

Search, leverage, and govern all your Salesforce Data Assets from day 2 with Catalog. See here how to connect your Salesforce Instance to Catalog. This connector is also available for Client-Managed configuration.

Catalog extracts Object metadata as well as reports and dashboards.

<ButtonLink to="/docs/catalog/integrations/data-warehouses/salesforce">
  Salesforce
</ButtonLink>

### Holistics Native Integration

Catalog now supports Holistics BI Tool

### Domo Native integration

Domo can be used as a BI tool but as well to store data itself. Catalog now fully supports all Domo implementations, federated or not.

<ButtonLink to="/docs/catalog/integrations/data-viz/domo">
  Domo
</ButtonLink>

## 9-14-2023

### Ingestion Report

You want to know what was loaded by Catalog when we last extracted metadata from your warehouse? This new report is made for you

<ButtonLink to="/docs/catalog/assets/reports#ingestion-report">
  View Ingestion report details
</ButtonLink>

## 9-07-2023

### API Warehouse Management Endpoints

You can now fully manage a warehouse using our API. This means you can add, update or delete tables, columns, or lineage to a warehouse. Please note that this warehouse must be entirely managed via API, to avoid conflicts with the catalog-run extraction

<ButtonLink to="https://apidocs.castordoc.com/#30df1dc3-5af4-4592-9963-9679c57291d3">
  Create Source endpoint
</ButtonLink>
Create Source endpoint

### API Data Quality Endpoint

After releasing integrations to MonteCarlo and Soda, we now enable our customers to load observability and quality tests from any tools, SaaS or home made

View our Data Quality Endpoint documentation here

<ButtonLink to="https://apidocs.castordoc.com/#e4604160-95c5-4f6d-88ce-7908f5e13686">
  Upsert data quality test endpoint
</ButtonLink>
Upsert data quality test endpoint

### Teams as owners

We all agree that having a team owning a dataset is ten times better than having an individual user. This is now possible in Catalog.

Team ownership will also appear on teams profile page.

### Lineage from S3 to Snowflake, out of the box

Just by looking at your Snowflake queries Catalog can now the origin of data loaded from S3. We can identify buckets, paths, and files used to load data into your snowflake tables.

From that we create an S3 source, add all these buckets, paths and files and finally automatically create lineage to your Snowflake tables.

## 8-31-2023

### Columns Tagging

You can now tag columns in Catalog, these tags can be added by a user or come from an external source such as dbt or snowflake

### New: dbt Owners

Catalog analyses the contributions to each of your dbt models to determine the top contributors and display them in the app as **Technical Owners**.

<figure><figcaption></figcaption></figure>

## 8-15-2023

### New: Chrome Extension

Catalog can now be leveraged directly from your BI tool or SQL editor. There is a lot more coming from the team but lot of value to get from already

See how to install the chrome extension here

<figure><figcaption></figcaption></figure>

### Improvement: "Struct" Column Lineage

The Catalog SQL Parser can now retrieve column for STRUCT elements. This increases heavily our coverage for customers relying on that pattern.

## 8-8-2023

### New: AI Knowledge draft generation

Do you think the definition of ARR, Customer Success, or Churn may be somewhat similar across different companies? We do too!

That is why you can now generate a draft for your Knowledge Metric/Domains/... pages, leveraging OpenAI capabilities.

<figure><figcaption></figcaption></figure>

See the full details here:

### New: Source Tags Automated

For several years, Catalog has been able to identify data extracted from popular SAAS tools. Think about SAP, Salesforce, Google Ads, and 53 more.

Now Catalog will automatically tag any tables or dashboard leveraging data from one of these SAAS tools

So if one of your Dashboards uses Salesforce data, it will be tagged with "source:salesforce" as on the example below

<figure><figcaption></figcaption></figure>

### Improvement: dashboard popularity for more BI tools

Catalog now automatically computes popularity for

* PowerBI
* Sigma Computing

## 8-1-2023

### New: Curated Content

For business users, information about schemas, tables, and columns can be overwhelming. This is why Catalog has released a Business User view, simplifying the UI by removing technical data elements. It also changes the defaults search filters to exclude searching within technical data assets.

<figure><figcaption></figcaption></figure>

See more here

### Improvement: Looker Lineage upgrade

Your Lookml is full of User Attributes and Constants? Catalog now resolves these with any issue

## 7-25-2023

### New: Catalog Scribe

Writting description for tables and dashboards is bothersome and at least a part of the description can be inferred from their underlying SQL queries and metadata.

This is exactly what Catalog Scribe does. Ask AI to write a description for you.

<figure><figcaption></figcaption></figure>

### New: Teams Editable

You can now create and edit teams directly within Catalog.

<figure><figcaption></figcaption></figure>

Find more information here

<ButtonLink to="/docs/catalog/administration/teams-in-catalog">
  Teams in Catalog
</ButtonLink>

### Improvement: lineage partial matching

Sometimes, due to missing context, our parser in only able to retrieve a table name but without its schema or database.

From now on, if that table name is unique in your catalog, the lineage link will be created. Huge improvement coming, especially for users of Postgres and Redshift.

## 7-18-2023

### Dashboard Frequent Users

Who uses which dashboards? Answering that question can bring a lot of trust in a given dashboard! Catalog now enables it on Looker

<figure><figcaption></figcaption></figure>

This comes out of the box for any customer with a live looker integration. If you are using the Client-Side set up, make sure to update your castor-extractor package.

## 7-11-2023

### New: Integration to Domo

Catalog now connects to Domo dashboards and datasets, including lineage.

## 6-27-2023

### New: Unused Assets Report

You can now generate an unused assets report to clean up your unused data assets and build an efficient data stack. You can download the unused assets report directly from the Governance > Reports section of Catalog. You can [read more here](https://www.castordoc.com/blog/now-live-unused-asset-reports).

<figure><figcaption></figcaption></figure>

## 6-23-2023

### New: Rich Text Images

Our rich text fields now support images! Now you can supplement you documentation with image assets, with more rich text improvements on the way.

### New: Sigma Integrations - Datasets Support

Catalog's Sigma Integration now supports Sigma Datasets. You can read more about integrating with Sigma [here](/docs/catalog/integrations/data-viz/sigma).

### New: Sync a Tag to a Knowledge Page

You can now sync a tag to a specific knowledge page. This means you can easily link all assets with a tag to a specific knowledge page.

### Improvement: Certified and Deprecated Details

You can now see more details on who certified or deprecated an asset by hovering on the respective icon.

### Improvement: Looker Column Field Lineage

We have improved the lineage for Looker, so you can now see the lineage between a field of an Explore and its related columns.

## 6-13-2023

### New: User Teams

[Assign users to teams](https://www.castordoc.com/blog/user-teams), so you can build context around teams and access associated knowledge pages that document the context of their domains.

## 6-7-2023

### Early Access: Assistant AI for asset descriptions and knowledge pages

Assistant AI now supports generating column, table and dashboard descriptions and drafting knowledge pages, so you never have to start with a blank slate.

Read more about Assistant AI [here](https://www.castordoc.com/blog/introducing-castor-ai-2-0).

<figure><figcaption></figcaption></figure>

## 6-6-2023

### New: Knowledge Map

We are excited to introduce [Knowledge Map](https://www.castordoc.com/blog/introducing-knowledge-map), a full visual overview of your organization’s knowledge pages. With Knowledge Map, you can now see the complete hierarchy of how your organization’s data spans domains, metrics, and KPIs you’ve documented.

<figure><figcaption></figcaption></figure>

## 5-5-2023

### New: Catalog API

Catalog now has a public API available with new endpoints for integrations. You can read more [here](https://apidocs.castordoc.com/). 

### **New: PowerBI Datasets Integration**

Catalog now integrates with PowerBI datasets. You can set up your PowerBI integration directly from the Integrations tab in the Settings. You can read more on how to integrate [here](/docs/catalog/integrations/data-viz/power-bi/powerbi).

<figure><figcaption></figcaption></figure>

## 5-2-2023

### Early Access: Sync Back to Snowflake, BigQuery, Tableau & Looker

We are excited to announce Sync Back early access with more tools, now supporting Snowflake, BigQuery, Tableau, and Looker in addition to dbt. So you can now push your data documentation out across your entire data stack.

These new Sync Backs are available by request for Early Access, and current customers can reach out now to get started. If you’re not yet a customer and would like to learn more, [get in touch today](https://www.castordoc.com/try-castor) to discuss how Sync Back improves your data experience.

Read the [full announcement here](http://www.castordoc.com/blog/sync-back-everywhere).

<figure><figcaption></figcaption></figure>

## 4-25-2023

### New: Surface Data Quality with dbt Tests & Soda

You can now see data quality tests from [dbt Tests](/docs/catalog/integrations/quality/dbt-tests) and [Soda](/docs/catalog/integrations/quality/soda) directly alongside your assets in Catalog. Easily stay up to date on your data’s health at a glance and let everyone in your organization know they can trust your data.

Read more about using data quality integrations [here](https://www.castordoc.com/blog/now-live-surface-data-quality).

<figure><figcaption></figcaption></figure>

## 4-21-2023

### New: Source Tagging

**Catalog now supports source tagging on tables and will be rolled out to customers starting April 25th.** These source tags will appear alongside your existing tags, supporting tools like Intercom, Stripe, Mixpanel, Salesforce and more, so you can see exactly which tool your data originates from.

Read more about source tagging [here](/docs/catalog/document-your-data/tags/source-tags).

### New: Tableau Data Source Field Lineage

Catalog now supports field lineage for published Tableau Data Sources. Lineage will be available for table columns and dashboards connected to Data Sources, from Columns to Field to Workbooks.

## 4-14-2023

### New: Databricks Integration

Catalog now integrates with Databricks. You can set up your Databricks integration directly from the Integrations tab in the Settings. You can read more on how to integrate [here](/docs/catalog/integrations/data-warehouses/databricks).

<figure><figcaption></figcaption></figure>

### New: Tag Styles

We made tags prettier with new styling options. Tags can now be colored, support emoji use and have new formatting options to support subcategories like Priority: Low, Priority: High.

<figure><figcaption></figcaption></figure>

## 4-12-2023

### New: Catalog AI Query Explainer

Catalog AI is here with the first AI powered feature, Query Explainer. This translates queries into natural language and lets anyone in your organization easily understand the logic behind queries with a single click.

Read more on using Catalog AI's Query Explainer [here](/docs/catalog/document-your-data/query-explainer).

<figure><figcaption></figcaption></figure>

## 3-31-2023

### New: Tableau Data Sources

Catalog now supports Tableau Data Sources as assets, similar to Looker Explore. These will now appear in the side navigation under Dashboards, as the related visualization models as well as in lineage.

Read more on using Tableau DataSources in [here](/docs/catalog/assets/dashboards#browse-data-sources).

<figure><figcaption></figcaption></figure>

## 3-03-2023

### New: Export Reports for Excel

You can export Reports in an Excel format in addition to the previously available CSV format. You can read more about exporting reports [here](/docs/catalog/assets/reports).

<figure><figcaption></figcaption></figure>

### New: dbt Cloud Integration

We now integrate with dbt Cloud, you can set up your dbt Cloud integration directly from the Integrations tab in the Settings. You can read more on how to integrate [here](/docs/catalog/integrations/transformation/dbt/dbt-cloud).

<figure><figcaption></figcaption></figure>

### Improvement: Column Lineage

These improvements surface column lineage directly in the Lineage tab. So you can now select Lineage and choose between dashboard lineage or selecting a specific column to view column lineage. This update also makes additional UI improvements to the Snowflake column lineage experience.

### Improvement: Looker Explore UI Updates

These improvements update the experience for Looker Explores by adding view labels, filters for view labels, displaying field usage, as well as ordering by field usage and filtering on field type.

## 2-24-2023

### New: Report an Issue

Let your whole team help keep an eye on your data’s health. You can now report an issue with a table from within Catalog. When the issue is reported, a comment will added to the asset in Catalog and the owner will be notified.

<figure><figcaption></figcaption></figure>

Read more on using Report an Issue[ here](/docs/catalog/collaborate/report-an-issue).

## 2023-02-06

### New: dbt Sync Back

You can now push table and column descriptions back to your dbt GitHub repo. To start using, grant contributor access to our technical GitHub user `techcastor` and trigger a sync using the “Sync back to dbt” button in the integrations section.

<figure><figcaption></figcaption></figure>

Read more on how to use [dbt Sync Back here](/docs/catalog/integrations/transformation/dbt/sync-back-to-dbt).

## 2023-02-04

### New: Manual Lineage

Incorporate your full lineage in Catalog with manual lineage. You can now link directly to the backend and delete links via Ops notebooks for tables and columns for sources where Catalog cannot retrieve data.

## 2023-02-02

### New Integration: Amazon Athena

We added a new integration with Amazon Athena. You can read more on setting this integration up [here](/docs/catalog/integrations/data-warehouses/amazon-athena-and-glue).

## 2023-01-31

### New: Request Data Access

You can now request access to a table directly in Catalog without needing to hunt down an admin to grant permissions. 

<figure><figcaption>Request access while viewing a table.</figcaption></figure>

<figure><figcaption>The access request will be logged in the asset comments.</figcaption></figure>

## 2023-01-27

### New Integration: Glue

We added a new integration with AWS Glue. You can read more on setting this integration up [here](/docs/catalog/integrations/data-warehouses/amazon-athena-and-glue).

## 2023-01-25

### Coming Soon: Data Quality

Soon you will be able to easily keep an eye on your data’s health, with Data Quality on tables. You will soon be able to see data quality at a glance when searching for tables, along with a detailed breakdown of the tests and statuses. Expect the first test integration in the coming weeks.

<figure><figcaption>See results of your latest data quality check.</figcaption></figure>

<figure><figcaption>Data quality status for tables at a glance.</figcaption></figure>

## 2023-01-23

### New: Request Documentation

You can now request documentation for a table or column. This will trigger a slack or email notification to the table owner to update the documentation for the related asset. If no owner is specified, admins are notified of the request. Read more [here](/docs/catalog/collaborate/request-documentation).

<figure><figcaption></figcaption></figure>

### New Reports Available: Deeper insights on user activity and data usage.

Admin can now access metadata on user activities in their warehouse, table analytics and dashboard analytics. These new reports can be used for insights to identify tables and dashboards with high compute costs, see who is running the most queries, or identify gaps in documentation. 

<figure><figcaption></figcaption></figure>

The following tables were added:

* PII Columns: Includes metadata on columns like Catalog URL and data type of a column.
* Knowledge Metadata: like path and tags for a knowledge page.
* User activities in the warehouse: Includes user activities in your warehouse for the last 30 days.
* User activities in the warehouse per table: Includes user activities by table in your warehouse for the last 30 days.
* Table analytics: Includes tables usage data like number of queries and number of parent tables.
* Dashboard analytics: Includes dashboard usage data like, url path, and number of parent tables.

Read more on what is included in each table [here](/docs/catalog/assets/reports).

## 2023-01-20

* Increased coverage for large queries
* LookML lineage improvements

## 2023-01-13

* Updates user mentions with link formatting
* Updates UI for invite modal
* Updates to column lineage for Tableau

## 2023-01-06

* Reports to include Tag metadata
* Updates UI of Lineage tab to provide clearer view of lineage

## 2023-12-30

* Adds Knowledge Metadata report
* Add PII Column report
* Enables dbt self onboarding in integration setup

## 2022-12-23

* Reports update to include dashboard analytics report

## 2022-12-16

* Added loader for pinned assets so users know they are loading
* Metadata Bulk Edit - Add tags to tables
* Bug fixes:
  * Catalog icon size issue
  * Make horizontal scrolling possible when there are long menu items

## 2022-12-09

* New feature: Metadata bulk Edit
  * Bulk add owners to tables

## 2022-12-02

* Renamed tags to attributes on table’s columns page
* Bug fixes
  * Selecting an asset in the pinned asset modal causes the search to reset
  * Modal overflow
* Column lineage coverage improvements
* Tableau tags are not dropped anymore when name’s too long

## 2022-11-18

* Added clickable breadcrumb paths on dashboards
* Bug fixes:
  * Toast error message
  * Resolve incorrect user count in People page

## 2022-11-11

* Support table of type `CLONE` for BigQuery
* Allow self onboarding for all warehouses and BI tools in Catalog managed mode

## 2022-11-04

* Direct lineage from dashboard to table for Metabase

## 2022-10-21

* Hide Google service accounts from People page
* Allow self onboarding for all warehouses and BI tools in Client managed mode
* Download table and dashboard metadata

## 2022-10-14

* Create your [teams in Catalog](/docs/catalog/administration/teams-in-catalog.md)
* Ability to assign[ teams as table/dashboards owner](/docs/catalog/collaborate/interaction-between-teams.md)

## 2022-10-06

* Search improvements: more context on assets location, search by database
* Fix on dashboard popularities
* BigQuery extractor safe mode

## 2022-09-23

* Download all downstream nodes as CSV file
* Hide “Sources” in lineage when asset has no parent
* Order History in reverse chronology
* Contribution links in home page redirects to user contribution tab
* dbt manifest parsing upgrade
* Table lineage: handle `INSERT_INTO` patterns

## 2022-09-16

* More context in search: table and dashboard descriptions are systematically displayed in search
* More assets in profile pages: tables, where user is a technical owner, and dashboards, where user is an editor
* Added a toggle/input on the lineage graph to recompute it on less than 30 days
* Qlik integration is live

## 2022-08-26

* New native integration: Qlik in Catalog
* Export lineage children as CSV file
* Column lineage: official release and UI improvements
* Complete search for very large Looker explores

## 2022-08-12

* Lineage Graph: Ability to decrease the 30 days default period in order to visualize recent changes
* Slack Notification on user signup to Catalog

## 2022-07-29

* Lineage: added sources (first ancestral tables) of element & improved perf issue on fetching it

## 2022-07-22

* Profile tabs performance improvements

## 2022-07-15

* Column lineage developments

## 2022-07-01

* Quotes added to BQ Table descriptions
* PyPi package - Looker: Add timeout as parameter and allow using `all_looks` end point to fetch Looks
* Lineage revamp: group tables by schemas and group BI assets by type and technology

## 2022-06-24

* Support `MERGE` statement in column lineage
* Tableau syncs can be Catalog managed

## 2022-06-17

* Dashboard Fields can be pinned
* Slack avatars are synced
* DBT owners automatically added
* Support SAML SSO loggin
* Support line breaks in Looker fields descriptions
* Revamp of lineage (step 1)

## 2022-06-10

* Larger assets nodes in graph lineage
* Upgrade from linked to Pinned assets

## 2022-06-03

* Fix DBT last refresh
* Clean blank Looker avatar pictures
* Exclude users only present on Slack from people page
* Dependency resolution for PyPi package

## 2022-05-27

* Handle null characters in Looker Dashboard titles
* Handle Looks with no directory
* Remove deprecated assets from home page
* Show last Catalog sync for each integration

## 2022-05-20

* Fix Okta sign-in bug
* Add "source queries" for Generic viz

## 2022-05-13

* Add Dashboards from Metabase "Our Analytics"
* Admins can now deprecate dashboards and knowledge pages

## 2022-05-06

* Handle generic warehouse ingestions
* Improve data catalog sync performance

## 2022-04-29

* Display by default the most 5 popular linked assets in lineage - click on "+" to display all links
* Support additional SQL statement for column lineage
* Owners get Slack notification when a comment is added on their assets

## 2022-04-22

* Handle markdown table rendering and h3-6 headings
* Deal with Snowflake latency issues: avoid empty tables

## 2022-04-15

* Fix: can use spaces in "refers to" search bar
* Display all "+" and "-" on lineage graph
* Handle more thoroughly sync with warehouse descriptions

## 2022-04-08

* Possibility to "deprecate" table assets
* Update of Column lineage target metric
* Ability to batch upload "Knowledge pages"
* Handle of "Update" queries for table lineage

## 2022-04-01

* Support multiple domains for independent user creations
* Trial count downs now present in app
* Knowledge "Drag and drop" (easily move knowledge subpages) is now live
* Read only role now live

## 2022-03-25

* Prep of external users as asset owners: allow users without Catalog account to be set as an asset owner
* Prep of knowledge "Drag & Drop"

## 2022-03-11

* Ingestion of multiple DBT manifests now possible
* Slack unfurls Dashboard and Knowledge links
* Hide Dashboards in lineage graph

## 2022-03-04

* Top Dashboards appear on homepage
* Dashboard source queries added
* Added additional filters in advanced search: owner, PII and verified
* Deleted transient filter for Snowflake accounts: all transient objects on imported into Catalog

## 2022-02-25

* By default, advanced search does not contain columns and fields
* Added external description to column description suggestion (AutoDoc) and column describer
* Improved column description suggestion performance

## 2022-02-18

* Fixed broken favorite links: tables appear in user pages
* Added new warehouse technology icons
* Fixed query mismatch in user pages

## 2022-02-11

* Added "Tables", "Dashboards", and "Knowledge" tabs to user pages
* Filtering by schema and tags now possible in advance search

## 2022-02-04

* All BI tools extraction agents merged into 1 package
* Scheduling now possible for Mode

## 2022-01-28

* External descriptions can now be rendered as markdown
* Advanced search applies filters based on quick search

## 2022-01-21

* External description for tables used as fallback on Table Explorer
* Dashboard editors appear in addition to dashboard owners

## 2022-01-14

* On people pages, view queries performed by user
* Display "is Certified" in search results

## 2022-01-07

* Save your favorite queries with "Pin queries"

## 2021-12-31

* Hide links in Graph lineage
* Mode package release

## 2021-12-17

* Display labels of Looker Explore fields
* Dbt Tags Automated Sync
* Client-side Extractors for Redshift, BigQuey and Snowflake
* Extended search filtering design preparation

## 2021-12-10

* Lineage graph with expansion and details
* Nested Knowledge pages
* View all active members on your Data Stack in People page
* View on which technology each member is active
* Invite to Catalog from People page
* Catalog Slack App can be installed from the Admin's integration page

## 2021-12-03

* Lineage graph first version: table parents and children
* Sharded tables Exploration for BigQuery

## 2021-11-26

* Slack: show message snipped in Catalog
* Redshift Spectrum is supported by Catalog
* Datastudio is compatible with Generic Dashboard

## 2021-11-19

* Slack App allow to trace back Slack message to Catalog
* Users can link columns in any text area of Catalog
* BigQuery nested fields are handled

## 2021-11-12

* Catalog integrated natively with Tableau
* Table freshness information is refreshed hourly
* Users can describe schemas and databases

## 2021-10-22

* Admin can edit Warehouse credentials
* Improved python module to push to GCP bucket

## 2021-10-15

* BigQuery Views now have popularity and lineage
* Catalog URLs are formatted prettily in table and column descriptions
* Looker Lineage Dashboard ↔ Explores
* Richtext Toolbar now always appears in descriptions

## 2021-10-08

* Catalog Auto Doc is live. See the dedicated page [here](/docs/catalog/document-your-data/catalog-scribe/castor-auto-doc)
* Admins can change user roles and deactivate/reactivate them
* Source queries for dbt tables are now available

## 2021-10-01

* Users can use table schemas to find tables
* Admin can see their company's employees on Catalog
* Users can see SQL queries that created/edited the tables
* Catalog sync your tags from BigQuery

## 2021-09-24

* Users can link data assets together, with backlinking activated
* URLs in Catalog are nicely formatted everywhere in the app
* Catalog offers a python package to push content to a GCP Bucket
* Lineage Parent and Child are redesigned

---

## History

The **History** tab records metadata changes on Catalog assets so you can see who updated what and when. Use it for governance reviews, to confirm bulk edits from the [Metadata Editor][], and to compare the full timeline on an asset with [notifications][] you receive.

<figure><figcaption></figcaption></figure>

## Where You Find History

Open any table, dashboard, or knowledge page and select the **History** tab. The same tab appears on workbook and view pages in supported BI integrations. Column-level metadata changes that Catalog tracks on a table, such as column tags or PII flags, appear in that table's **History** timeline.

## What History Records

History logs in-Catalog metadata edits, not warehouse DDL or changes made only in external BI tools unless Catalog surfaces them as metadata events.

The timeline can include:

- **Descriptions** - Manual edits and AI-generated description updates
- **Owners** - Owner or team added or removed, including technical owners sourced from integrations
- **Tags** - Tag added or removed on the asset
- **Certification** - Asset certified or uncertified
- **Deprecation** - Asset deprecated or undeprecated
- **PII** - Column marked or unmarked as PII on a table
- **Pinned assets** - Related table, dashboard, knowledge page, or field pinned or unpinned on the asset documentation tab
- **Data products** - Asset added to or removed from a data product

## How the Timeline Is Organized

Events are grouped into **recent** for the last 24 hours, **past month**, and **older** sections so you can scan activity quickly.

Consecutive description edits by the same person or automation may appear as one grouped entry. Other change types, such as tags or owners, show as individual events.

Each entry shows who made the change. AI-assisted description updates are labeled separately from manual edits. Some automated changes record a service name instead of a user when Catalog applies the update.

## Use History for Governance

Stewards and admins use the **History** tab as the audit trail after tagging, ownership, or certification work.

Typical workflows include:

1. Open **History** on a table, dashboard, or knowledge page before you certify or verify the asset. Review recent description, owner, or tag updates and confirm they match what you expect.
2. After a [Metadata Editor][] bulk run, spot-check **History** on a sample of updated assets to confirm owners, tags, or certification status.
3. When you receive a notification about a tag or owner change, open **History** on the same asset to see the full sequence of updates.

For how tags, ownership, and related concepts fit together, see [Governance concepts][].

## What's Next?

- [Metadata Editor][] - Apply bulk metadata changes across many assets
- [Governance concepts][] - Tags, ownership, and pinned assets
- [Notifications][] - Configure Slack, Microsoft Teams, or email alerts

[Metadata Editor]: /docs/catalog/document-your-data/metadata-editor
[Governance concepts]: /docs/catalog/use-cases/document/content-and-context/governance-concepts
[Notifications]: /docs/catalog/collaborate/notifications

---

## Interaction Between Teams

Reach consensus on data documentation by using Catalog's collaboration features: ownership, comments, and Slack integration.

## Ownership

Now that you have set your teams in the Catalog, you can start using it. [_Wait, you haven't done that yet_](/catalog/administration/teams-in-catalog.md).

Go to your favorite table or dashboard and assign the responsible team as owner.

<figure><figcaption></figcaption></figure>

## Comments

Every asset page in the Catalog contains a Comments section.

It's a great place to ask questions if existing definitions do not cover everything.

For example:

* Who is the owner of this asset?
* Can a definition be more explicit (include units)?
* Will the asset become deprecated?

Whenever you have a question about an asset, it's a good place to check if someone has already answered it.

Keep in mind that you can mention users using the "@" handle.

## Slack

Slack is now a go-to for collaboration in all companies. That's why we created our Slack app to seamlessly integrate within your communication process.

### Preview Asset Metadata in Slack

When pasting a Catalog link in Slack, details of the related asset will unfurl in Slack.

### Keep a Trace of Your More Insightful Exchanges

You don't want to answer the same question repeatedly.

Click on `Save thread link in Catalog` and the Slack thread link and first message will appear on the corresponding asset's comment section. It will build a log for users to see the FAQ.

:::warning[Public Channels Only]

Only available in public channels.

:::

### Good Practice Advice

To get your people to use the Catalog more and become more independent with data:

:::info[Include Catalog Links]

All questions in Slack regarding a data asset should include a Catalog link.

:::

:::info[Channel Headers]

A Catalog link should appear in the header of your dedicated data support Slack channels.

:::

---

## Notifications

A notification system has been implemented using [Slack][], [Microsoft Teams][], and **email**. Notifications will be sent to relevant team members when certain events occur, such as comment, request, or description changes. This system is designed to ensure that all team members are kept informed and up-to-date on important developments.

If there is an owner, then the notification is sent to them. If there is no owner, the system sends the notification to the admin.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

## Two Types of Notifications

- **Direct Notifications:** These notifications are triggered by a user's action and are sent instantly.
- **Digest Notifications:** Notifications are grouped to avoid disturbing the user. We check for any activity in the last hour, and if there is none, we send the notification group. However, if it has been more than 12 hours since the last notification, we send the group immediately.

## Edit Notification

- Someone commented on your asset
- Someone added you as owner
- Someone added a description on your asset
- Someone added a column description on your asset
- Someone added a tag on your asset
- Someone marked as deprecated your asset
- Someone marked as certified your asset
- Someone marked a column as PII

## Request Notification

- Someone requested a description
  - [Request Documentation][] feature
- Someone requested data access
  - [Request Data Access][] feature
- Someone reported an issue
  - [Report an Issue][] feature

[Slack]: https://slack.com/
[Microsoft Teams]: https://www.microsoft.com/en-us/microsoft-teams/group-chat-software
[Request Documentation]: https://docs.coalesce.io/docs/catalog/collaborate/request-documentation
[Request Data Access]: https://docs.coalesce.io/docs/catalog/collaborate/request-data-access
[Report an Issue]: https://docs.coalesce.io/docs/catalog/collaborate/report-an-issue

---

## Notifications for Teams

As with your individual contributors, the teams in Catalog can also get notified through Slack.

## How To Set It Up

1. [Create the relevant teams in Catalog][]
1. Fill the dedicated notification channel for each of them

        <figure><figcaption></figcaption></figure>

1. Invite the Catalog Slack bot into each channel and you are all set

        <figure><figcaption></figcaption></figure>

## How Does It Work

Once the previous steps are done, the teams will be able to receive notifications on their Slack channel.

Assign your teams as asset owners so they can start receiving them.

<figure><figcaption></figcaption></figure>

:::info[Account Settings]

Make sure your notifications are activated in your [account settings][].

:::

[Create the relevant teams in Catalog]: /docs/catalog/administration/teams-in-catalog
[account settings]: /docs/catalog/collaborate/notifications

---

## Report an Issue

When viewing a table in Catalog, you can report an issue if needed. From the top right menu, you can select **Report an Issue** and add details about the issue. This will prompt the table owner or admin to investigate.

Reporting an issue notifies relevant users by:

- Adding a comment to the asset in Catalog
- Triggering a Slack notification to the table owner; if Slack is not enabled for that owner, an email notification will be sent
- Otherwise, if that table has no owner, the admin will be notified in the same manner

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

---

## Request Data Access

If you find a table you don't have access to in the backend, you can request access from the table owner directly in Catalog.

<figure><figcaption></figcaption></figure>

Once you have requested access, your request will be documented in the table comments.

<figure><figcaption></figcaption></figure>

---

## Request Documentation

When viewing your organization's data in Catalog, you can request documentation for a table or a column if needed. This will prompt the table owner or admin to update the related documentation.

Requesting documentation notifies relevant users by:

- Triggering a Slack notification to the table owner; if Slack is not enabled for that owner, an email notification will be sent
- Otherwise, if that table has no owner, the admin will be notified in the same manner

<figure><figcaption></figcaption></figure>

---

## AI Assistant API

Use the Coalesce Catalog public API to call the AI Assistant from your own agent, MCP server, or custom tool chain. You submit a question with `addAiAssistantJob`, then poll `getAiAssistantJobResult` until the job completes.

## Summary

You can use the AI Assistant API with:

- an MCP server
- your agent's `tools`

## Resources

- [Coalesce Catalog public API][]
- [addAiAssistantJob endpoint][]
- [getAiAssistantJobResult endpoint][]

## Prerequisites

You need a Catalog API token with **Read & Write** scope. Catalog administrators create tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for generation and rotation steps.

## How the API Works

The Coalesce Catalog public API is a [GraphQL API][]. To interact with the AI Assistant, implement a polling flow with these two queries:

- [addAiAssistantJob][]: Creates an asynchronous AI Assistant job and returns a `jobId`.
- [getAiAssistantJobResult][]: Uses the `jobId` to poll for the job result.

## Add AI Assistant Job

Use the `addAiAssistantJob` query to create an asynchronous AI Assistant job.

### Add Job Request

Headers:

```json
{"Authorization": "Token <API_TOKEN>"}
```

Body:

```graphql
query {
  addAiAssistantJob (
    data: {
      question: "<user_question>"
      email: "<user_email>"
      externalConversationId: "<unique_conversation_id>"
    }
  ){
    data {
      jobId
    }
  }
}
```

### Add Job Response

```graphql
{
  "data": {
    "addAiAssistantJob": {
      "data": {
        "jobId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      }
    }
  }
}
```

## Poll for Job Result

Use the `getAiAssistantJobResult` query to poll for the job result.

### Poll Request

Headers:

```json
{"Authorization": "Token <API_TOKEN>"}
```

Body:

```graphql
query {
  getAiAssistantJobResult (
    data: {
      id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      delaySeconds: 5
    }
  ){
    data {
      status
      answer
      assets {
        id
        internalLink
        name
        url
      }
    }
  }
}
```

### Poll Response

```graphql
{
  "data": {
    "getAiAssistantJobResult": {
      "data": {
        "status": "completed",
        "answer": "<AI_Assistant_answer>",
        "assets": [
          {
            "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "internalLink": "https://castordoc.com",
            "name": "",
            "url": "https://example.com"
          }
        ]
      }
    }
  }
}
```

First, call `addAiAssistantJob` to receive the `jobId`. Then use that `jobId` to poll for the result with `getAiAssistantJobResult`, which includes a built-in polling delay controlled by the `delaySeconds` parameter.

[Coalesce Catalog public API]: /docs/catalog/api
[GraphQL API]: /docs/catalog/api
[addAiAssistantJob endpoint]: /docs/catalog/api/general/operations/queries/add-ai-assistant-job
[getAiAssistantJobResult endpoint]: /docs/catalog/api/general/operations/queries/get-ai-assistant-job-result
[addAiAssistantJob]: /docs/catalog/api/general/operations/queries/add-ai-assistant-job
[getAiAssistantJobResult]: /docs/catalog/api/general/operations/queries/get-ai-assistant-job-result
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys

---

## BI Tools

These pages document **`castor-extract-*`** commands for visualization sources. Install the package and workflow steps on [Castor Extractor][], then follow the guide for your BI tool.

<DocCardList />

---

[Castor Extractor]: /docs/catalog/developer/castor-extractor

---

## Looker Studio

Extract Looker Studio assets and sync them to the Catalog using castor-extractor.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before proceeding.

:::

* You will need a Service Account with permissions to access Looker Studio. The instructions can be found in the [Looker Studio integration docs][].
* You will also need to provide the email of a Google Workspace user with admin access.

### Optional: BigQuery Access

If your Looker Studio data sources are connected to BigQuery:

* Make sure to install the BigQuery dependencies as well:

```bash
pip install castor-extractor[lookerstudio,bigquery]
```

* You will require a Service Account with BigQuery access. Consider using the Service Account linked to your BigQuery integration. Note that this can be the same Service Account as for Looker Studio, or a separate one.

### Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-looker-studio [arguments]
```

The script will run and display logs similar to the following:

```bash
INFO - Loading credentials from /looker/studio/credentials.json
INFO - Loading credentials from /bigquery/credentials.json
INFO - Extracting ASSETS from API
INFO - Refreshing authentication token...
INFO - Refreshing authentication token...

...

INFO - Wrote output file: /tmp/ls/1742219344-summary.json
```

### Command Arguments

#### Required Arguments

**For a full Looker Studio extraction:**

* `-c`, `--credentials`: File path to Service Account credentials with Looker Studio access
* `-a`, `--admin-email`: Email of a Google Workspace user with admin access

**For source-queries-only mode:**

* `-b`, `--bigquery-credentials`: File path to Service Account credentials with BigQuery access

#### Optional Arguments

* `-o`, `--output`: Directory to write the extracted files to
* `--source-queries-only`: If selected, only extracts BigQuery source queries (bypasses Looker Studio extraction)
* `--skip-view-activity-logs`: Skip extraction of activity logs (use if credentials lack required scopes)
* `--users-file-path`: Optional path to a JSON file with user email addresses as a list of strings (for example, `["foo@bar.com", "fee@bar.com"]`). If provided, only extracts assets owned by the specified users.
* `--db-allowed`: Optional list of GCP projects to allow for source queries extraction
* `--db-blocked`: Optional list of GCP projects to block from source queries extraction

You can always get help with the argument `--help`.

### Use ENV Variables

You can set the following ENV variables in your `.bashrc` to avoid having to pass them as arguments:

```text
CASTOR_LOOKER_STUDIO_ADMIN_EMAIL=admin@your-domain.com
CASTOR_OUTPUT_DIRECTORY=/tmp/catalog

GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials/service-account.json
```

Then the script can be executed without any arguments:

```bash
castor-extract-looker-studio
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-looker-studio --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[Looker Studio integration docs]: /docs/catalog/integrations/data-viz/looker-studio

---

## Looker

Extract Looker metadata (users, folders, dashboards, and more) into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

* Get credentials with permissions to call the [Looker API](https://docs.looker.com/reference/api-and-integration/api-auth)

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-looker [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting users from Looker API
INFO - POST(https://cloud.looker.com/api/4.0/login)
INFO - GET(https://cloud.looker.com/api/4.0/users/search)
INFO - Fetched page 1 / 7 results
INFO - GET(https://catalog.cloud.looker.com/api/4.0/users/search)
INFO - Fetched page 2 / 0 results

...

INFO - Wrote output file: /tmp/catalog/1649079699-projects.json
INFO - Wrote output file: /tmp/catalog/1649079699-summary.json
```

### Credentials

* `-c`, `--client-id`: Looker API Client ID
* `-s`, `--client-secret`: Looker API Client Secret

### Other Arguments

* `-b`, `--base-url`: Looker base URL
* `-o`, `--output`: Target folder to store the extracted files
* `-t`, `--timeout`: Timeout (in seconds) parameter for Looker API
* `--log-to-stdout`: Will write all log outputs to `stdout` instead of `stderr`

### Specific Export Methods for Looks and Dashboards

* `--search-per-folder`: Will export looks and dashboards per folder using parallel threads (see argument below)
* `--thread-pool-size`: Number of parallel threads, defaults to 20

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```text
CASTOR_LOOKER_BASE_URL
CASTOR_LOOKER_CLIENT_ID
CASTOR_LOOKER_CLIENT_SECRET

CASTOR_OUTPUT_DIRECTORY
CASTOR_LOOKER_TIMEOUT_SECOND

# To log outputs to `stdout` instead of `stderr`
CASTOR_LOOKER_LOG_TO_STDOUT=TRUE

# To use search_per_folder multithreading
CASTOR_LOOKER_SEARCH_PER_FOLDER=TRUE
CASTOR_LOOKER_THREAD_POOL_SIZE=20
```

Then the script can be executed without any arguments:

```bash
castor-extract-looker
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-looker --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/

---

## Metabase

Extract Metabase metadata into Catalog using the castor-extractor package. The package supports Postgres and API connectors.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

The package currently supports two connectors:

### Postgres

The data is extracted by running SQL queries on the Postgres database used by your Metabase instance:

* Directly on your Metabase DB
* On a replica or views with similar structure

You will need Postgres credentials with read access on Metabase tables (or views with similar columns)

### Metabase API

The data is extracted by calling the [Metabase API](https://www.metabase.com/docs/latest/api-documentation.html).

Prerequisites:

* Metabase credentials with role `superuser`

:::danger[Popularity Not Available]

This connector does not allow computing `popularity`, since the [activity endpoint](https://www.metabase.com/docs/latest/api-documentation.html#get-apiactivity) brings partial results.

:::

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-metabase-db [arguments]

# or, if you use the API connector:
castor-extract-metabase-api [arguments]
```

The script will run and display logs as following:

```bash
INFO - Getting session_id: {'id': '****'}
INFO - Fetching USER (15 results)
INFO - Wrote output file: /tmp/catalog/1649081473-user.json
INFO - Fetching COLLECTION (41 results)

...

INFO - Wrote output file: /tmp/catalog/1649081473-dashboard_cards.json
INFO - Wrote output file: /tmp/catalog/1649081473-summary.json

```

### Credentials

You can sign-in using one of the following methods:

* with _Postgres connector_
  * `-H`, `--host`: Host name where the server is running
  * `-P`, `--port`: TCP/IP port number
  * `-d`, `--database`: Database name
  * `-s`, `--schema`: Schema name where the views or tables are located
  * `-u`, `--user`: User
  * `-p`, `--password`: Password
  * `-o`, `--output`: Directory to write to
* with _API connector_
  * `-b`, `--base-url`: Metabase base URL, such as `http://company.cloud.metabase.com`
  * `-u`, `--user`: Metabase user
  * `-p`, `--password`: Metabase password

### Other Arguments

* `-o`, `--output`: Target folder to store the extracted files

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```sql
# with DB connector #
CASTOR_METABASE_DB_HOST
CASTOR_METABASE_DB_PORT
CASTOR_METABASE_DB_DATABASE
CASTOR_METABASE_DB_SCHEMA
CASTOR_METABASE_DB_USERNAME
CASTOR_METABASE_DB_PASSWORD

# with API connector #

CASTOR_METABASE_API_BASE_URL
CASTOR_METABASE_API_USERNAME
CASTOR_METABASE_API_PASSWORD

# optional, valid with both connectors #

CASTOR_METABASE_ENCRYPTION_SECRET_KEY

CASTOR_OUTPUT_DIRECTORY
```

Then the script can be executed without any arguments:

```bash
castor-extract-metabase-db
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-metabase-db --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/

---

## Mode Analytics

Extract Mode Analytics metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

* Get `admin` credentials on your Mode Analytics instance.
* Generate an API token by following the [Mode API token instructions](https://mode.com/help/articles/api-reference/#generating-api-tokens)

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-mode [arguments]
```

The script will run and display logs as following:

```bash
INFO - Authentication succeeded.
INFO - Starting extraction for DATASOURCE...
INFO - Calling https://modeanalytics.com/api/...
INFO - 1 rows extracted
INFO - Wrote output file: /tmp/catalog/1649080890-datasource.json
INFO - Starting extraction for COLLECTION...

...

INFO - Wrote output file: /tmp/catalog/1649080890-member.json
INFO - Wrote output file: /tmp/catalog/1649080890-summary.json
```

### Credentials

* `-H`, `--host`: The name or IP of the host where your Mode Analytics instance is installed.

:::info[Managed Instance]

If you are using a managed instance, the host is `https://modeanalytics.com`.

:::

* `-w`, `--workspace`: The name of your workspace. See [Mode organizations documentation](https://mode.com/help/articles/organizations/) for more information.
* `-t`, `--token`: The `Token` value from the API token created in Mode.
* `-s`, `--secret`: The `Password` value from the API token created in Mode.

### Other Arguments

* `-o`, `--output`: Target folder to store the extracted files

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```text
CASTOR_MODE_ANALYTICS_HOST
CASTOR_MODE_ANALYTICS_WORKSPACE
CASTOR_MODE_ANALYTICS_TOKEN
CASTOR_MODE_ANALYTICS_SECRET

CASTOR_OUTPUT_DIRECTORY
```

Then the script can be executed without any arguments:

```bash
castor-extract-mode
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-mode --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/

---

## Tableau

Extract Tableau metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

* Get Tableau credentials with role `superuser`

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-tableau [arguments]
```

The script will run and display logs as following:

```bash
INFO - Logging in using user and password authentication
INFO - Signed into https://eu-west-1a.online.tableau.com as user with id ****
INFO - Extracting USER from Tableau API
INFO - Fetching USER
INFO - Querying all users on site

...

INFO - Wrote output file: /tmp/catalog/1649078755-custom_sql_queries.json
INFO - Wrote output file: /tmp/catalog/1649078755-summary.json
```

### Credentials

If sign-in fails with `401001: Signin Error`, `FailedSignInError`, or similar Tableau REST sign-in errors in extractor logs, follow [Sign-In or Authentication Errors][] in the Tableau integration documentation. That section covers PAT or password rotation, `serverUrl`, `siteId`, and network allowlisting.

You can sign in using one of the following methods:

* _user and password_
  * `-u`, `--username`: Tableau username
  * `-p`, `--password`: Tableau password
* _Tableau personal access token (PAT)_
  * `-n`, `--token-name`: Tableau token name
  * `-t`, `--token`: Tableau token

### Other Arguments

* `-b`, `--server-url`: Tableau base URL (your API endpoint, usually your Tableau homepage; for example, `https://eu-west-1a.online.tableau.com`)
* `-i`, `--site-id`: Tableau Site ID, is empty if your site is the default one
* `-o`, `--output`: Target folder to store the extracted files

### Optional Arguments

* `--with-pulse`: Option to extract Tableau Pulse assets: Metrics and Subscriptions
* `--page-size`: Option to override the page size used in the pagination, to be used it extraction hits the node limit for a request in Tableau
* `--skip-columns`: Option to avoid extracting Tableau columns, default to False

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```text
CASTOR_TABLEAU_USER
CASTOR_TABLEAU_PASSWORD
CASTOR_TABLEAU_TOKEN_NAME
CASTOR_TABLEAU_TOKEN
CASTOR_TABLEAU_SERVER_URL
CASTOR_TABLEAU_SITE_ID

CASTOR_OUTPUT_DIRECTORY
```

Then the script can be executed without any arguments:

```bash
castor-extract-tableau
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-tableau --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[Sign-In or Authentication Errors]: /docs/catalog/integrations/data-viz/tableau#sign-in-or-authentication-errors

---

## Castor Extractor Reference

Use this page for per-command flags and environment variables. For installation, workflow, and troubleshooting, see [Castor Extractor][].

## Global Variables

These variables apply across all commands:

| Variable | Purpose |
| --- | --- |
| `CASTOR_OUTPUT_DIRECTORY` | Default output directory for all extractors. |
| `GOOGLE_APPLICATION_CREDENTIALS` | Default GCP credentials file for BigQuery and Looker Studio. |

## Zone Selection

- Use `US` if your instance is on `app.us.castordoc.com`.
- Use `EU` if your instance is on `app.castordoc.com`.

## Upload and Validate

### `castor-file-check`

Validate generic warehouse CSV files before upload.

- `-d, --directory`: directory containing generic warehouse CSV files
- `--verbose`: show detailed validation logs

### `castor-upload`

Push extracted files to Catalog-managed GCS.

- `-k, --token`: API token from Catalog
- `-s, --source_id`: source ID from Catalog
- `-t, --file_type`: file type (`WAREHOUSE`, `VIZ`, `DBT`, `QUALITY`)
  - `WAREHOUSE` extractors
  - `VIZ` Visualization extractors
  - Knowledge bases (Confluence and Notion) use `VIZ`
  - `QUALITY` - Used for external data quality tools along with generic CSV files.
- `-z, --zone`: upload zone (`US` or `EU`, default `EU`)
- `-f, --file_path`: upload one file
- `-d, --directory_path`: upload all files in a directory

You can only use `--file_path` or `--directory_path`, not both.

:::info[CLI help]

Use `--help` to get the most up-to-date flags. For example, `castor-extract-sqlserver --help`.

:::

## Warehouse Extractors

These use upload file type `WAREHOUSE`

### `castor-extract-bigquery`

| Flag | Description |
| --- | --- |
| `-c, --credentials` | Path to Google credentials file. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |
| `--db-allowed <list>` | Allowed GCP projects. |
| `--db-blocked <list>` | Blocked GCP projects. |
| `-s, --safe-mode` | Safe mode. |

These environment variables are supported:

- `GOOGLE_APPLICATION_CREDENTIALS`

### `castor-extract-databricks`

| Flag | Description |
| --- | --- |
| `-H, --host` | Databricks host. |
| `-t, --token` | Access token. |
| `-p, --http-path` | HTTP path. |
| `-o, --output` | Output directory. |
| `--catalog-allowed <list>` | Allowed catalogs. |
| `--catalog-blocked <list>` | Blocked catalogs. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_DATABRICKS_HOST`
- `CASTOR_DATABRICKS_HTTP_PATH`
- `CASTOR_DATABRICKS_TOKEN`

### `castor-extract-glue-athena`

Requires `pip install castor-extractor[glue-athena]`.

| Flag | Description |
| --- | --- |
| `--access-key-id` | AWS access key ID. |
| `--access-key-secret` | AWS access key secret. |
| `--aws-region` | AWS region. |
| `--aws-account-id` | AWS account ID. |
| `--schema-allowed <list>` | Glue schemas to include. |
| `--schema-blocked <list>` | Glue schemas to exclude. |
| `-s, --skip-queries` | Skip SQL query and view DDL extraction. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_GLUE_ACCESS_KEY_ID`
- `CASTOR_GLUE_ACCESS_KEY_SECRET`
- `CASTOR_GLUE_AWS_REGION`
- `CASTOR_GLUE_AWS_ACCOUNT_ID`

### `castor-extract-mysql`

| Flag | Description |
| --- | --- |
| `-H, --host` | MySQL host. |
| `-P, --port` | MySQL port. |
| `-u, --user` | MySQL user. |
| `-p, --password` | MySQL password. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_MYSQL_USER`
- `CASTOR_MYSQL_PASSWORD`
- `CASTOR_MYSQL_HOST`
- `CASTOR_MYSQL_PORT` (optional)

### `castor-extract-postgres`

| Flag | Description |
| --- | --- |
| `-H, --host` | Postgres host. |
| `-P, --port` | Postgres port. |
| `-d, --database` | Postgres database. |
| `-u, --user` | Postgres user. |
| `-p, --password` | Postgres password. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_POSTGRES_USER`
- `CASTOR_POSTGRES_PASSWORD`
- `CASTOR_POSTGRES_HOST`
- `CASTOR_POSTGRES_PORT`
- `CASTOR_POSTGRES_DATABASE`

### `castor-extract-redshift`

| Flag | Description |
| --- | --- |
| `-H, --host` | Redshift host. |
| `-P, --port` | Redshift port. |
| `-d, --database` | Redshift database. |
| `-u, --user` | Redshift user. |
| `-p, --password` | Redshift password. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |
| `--serverless` | Extract from Redshift Serverless. |

These environment variables are supported:

- `CASTOR_REDSHIFT_USER`
- `CASTOR_REDSHIFT_PASSWORD`
- `CASTOR_REDSHIFT_HOST`
- `CASTOR_REDSHIFT_PORT`
- `CASTOR_REDSHIFT_DATABASE`
- `CASTOR_REDSHIFT_SERVERLESS` (optional; `true`/`false`)

### `castor-extract-snowflake`

| Flag | Description |
| --- | --- |
| `-a, --account` | Snowflake account. |
| `-u, --user` | Snowflake user. |
| `-p, --password` | Password. Mutually exclusive with `--private-key`. |
| `-pk, --private-key` | Private key. Mutually exclusive with `--password`. |
| `--warehouse` | Warehouse override. |
| `--role` | Role override. |
| `--db-allowed <list>` | Allowed databases. |
| `--db-blocked <list>` | Blocked databases. |
| `--query-blocked <list>` | Blocked query patterns. Supports `%` and `_` wildcards. |
| `--fetch-transient` | Include transient tables. |
| `--insecure-mode` | Disable OCSP checking. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_SNOWFLAKE_ACCOUNT`
- `CASTOR_SNOWFLAKE_USER`
- `CASTOR_SNOWFLAKE_PASSWORD` (optional if using private key)
- `CASTOR_SNOWFLAKE_PRIVATE_KEY` (optional if using password)
- `CASTOR_SNOWFLAKE_INSECURE_MODE` (optional)

### `castor-extract-sqlserver`

| Flag | Description |
| --- | --- |
| `-H, --host` | MSSQL host. |
| `-P, --port` | MSSQL port. |
| `-u, --user` | MSSQL user. |
| `-p, --password` | MSSQL password. |
| `-s, --skip-queries` | Skip SQL query extraction. |
| `--db-allowed <list>` | Allowed databases. |
| `--db-blocked <list>` | Blocked databases. |
| `--default-db` | Fallback database for login issues. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_MSSQL_USER`
- `CASTOR_MSSQL_PASSWORD`
- `CASTOR_MSSQL_HOST`
- `CASTOR_MSSQL_PORT`
- `CASTOR_MSSQL_DEFAULT_DB` (optional)

## Visualization Extractors

These use file type `VIZ`.

### `castor-extract-count`

| Flag | Description |
| --- | --- |
| `-c, --credentials` | GCP credentials as string. |
| `-d, --dataset_id` | Data set ID storing Count data. |
| `-o, --output` | Output directory. |

### `castor-extract-domo`

| Flag | Description |
| --- | --- |
| `-b, --base-url` | Domo host. |
| `-a, --api-token` | API token. |
| `-d, --developer-token` | Developer token. |
| `-c, --client-id` | Client ID. |
| `-C, --cloud-id` | External warehouse ID. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_DOMO_API_TOKEN`
- `CASTOR_DOMO_BASE_URL`
- `CASTOR_DOMO_CLIENT_ID`
- `CASTOR_DOMO_DEVELOPER_TOKEN`
- `CASTOR_DOMO_CLOUD_ID`
- `CLOUD_ID`

### `castor-extract-looker`

| Flag | Description |
| --- | --- |
| `-b, --base-url` | Looker base URL. |
| `-c, --client-id` | Client ID. |
| `-s, --client-secret` | Client secret. |
| `-t, --timeout` | Timeout in seconds. |
| `--thread-pool-size` | Thread pool size. |
| `-S, --safe-mode` | Safe mode. |
| `--log-to-stdout` | Log to `stdout`. |
| `--search-per-folder` | Fetch looks and dashboards per folder. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_LOOKER_BASE_URL`
- `CASTOR_LOOKER_CLIENT_ID`
- `CASTOR_LOOKER_CLIENT_SECRET`
- `CASTOR_LOOKER_TIMEOUT_SECOND` (optional override)
- `CASTOR_LOOKER_PAGE_SIZE` (optional override)
- `CASTOR_LOOKER_THREAD_POOL_SIZE` (optional override)
- `CASTOR_LOOKER_IS_SAFE_MODE` (optional; `true`/`false`)
- `CASTOR_LOOKER_LOG_TO_STDOUT` (optional; `true`/`false`)
- `CASTOR_LOOKER_SEARCH_PER_FOLDER` (optional; `true`/`false`)

### `castor-extract-looker-studio`

| Flag | Description |
| --- | --- |
| `-o, --output` | Output directory. |
| `--source-queries-only` | Only extract BigQuery source queries. |
| `--skip-view-activity-logs` | Skip activity log extraction. |
| `-c, --credentials` | Service account credentials file. |
| `-a, --admin-email` | Google Workspace admin email. |
| `--users-file-path` | Path to JSON array of user emails. |
| `-b, --bigquery-credentials` | BigQuery service account credentials file. |
| `--db-allowed <list>` | Allowed GCP projects for source queries. |
| `--db-blocked <list>` | Blocked GCP projects for source queries. |

These environment variables are supported:

- `GOOGLE_APPLICATION_CREDENTIALS`: path to the Looker Studio service account JSON file (when using `-c` / `--credentials`)
- `CASTOR_LOOKER_STUDIO_ADMIN_EMAIL`: Google Workspace admin email (when using `-a` / `--admin-email`)
- `CASTOR_OUTPUT_DIRECTORY`: default output directory

### `castor-extract-metabase-api`

| Flag | Description |
| --- | --- |
| `-b, --base-url` | Metabase base URL. |
| `-u, --user` | Username. |
| `-p, --password` | Password. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_METABASE_API_BASE_URL`
- `CASTOR_METABASE_API_USERNAME`
- `CASTOR_METABASE_API_USER`
- `CASTOR_METABASE_API_PASSWORD`

### `castor-extract-metabase-db`

| Flag | Description |
| --- | --- |
| `-H, --host` | Host. |
| `-P, --port` | Port. |
| `-d, --database` | Database. |
| `-s, --schema` | Schema. |
| `-u, --user` | Username. |
| `-p, --password` | Password. |
| `-k, --encryption_secret_key` | Encryption key. |
| `--require_ssl` | Require SSL. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_METABASE_DB_HOST`
- `CASTOR_METABASE_DB_PORT`
- `CASTOR_METABASE_DB_DATABASE`
- `CASTOR_METABASE_DB_SCHEMA`
- `CASTOR_METABASE_DB_USERNAME`
- `CASTOR_METABASE_DB_PASSWORD`
- `CASTOR_METABASE_DB_ENCRYPTION_SECRET_KEY` (optional)
- `CASTOR_METABASE_DB_REQUIRE_SSL_KEY` (optional)

### `castor-extract-mode`

| Flag | Description |
| --- | --- |
| `-H, --host` | Mode host. |
| `-w, --workspace` | Workspace. |
| `-t, --token` | API token. |
| `-s, --secret` | API token password. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_MODE_ANALYTICS_HOST`
- `CASTOR_MODE_ANALYTICS_SECRET`
- `CASTOR_MODE_ANALYTICS_TOKEN`
- `CASTOR_MODE_ANALYTICS_WORKSPACE`

### `castor-extract-omni`

Requires `pip install castor-extractor[omni]`.

| Flag | Description |
| --- | --- |
| `-u, --base-url` | Omni instance root URL (the hostname you use to sign in). |
| `-t, --token` | Omni organization API key (Bearer token). |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_OMNI_BASE_URL`
- `CASTOR_OMNI_TOKEN`

### `castor-extract-powerbi`

| Flag | Description |
| --- | --- |
| `-t, --tenant_id` | Tenant ID. |
| `-c, --client_id` | Client ID. |
| `-s, --secret` | Client secret. Mutually exclusive with `--certificate`. |
| `-cert, --certificate` | Certificate file. Mutually exclusive with `--secret`. |
| `-sc, --scopes <list>` | API scopes. Optional. |
| `-l, --login_url` | Login URL. Optional. |
| `-a, --api_base` | Power BI REST API base. Optional. |
| `-g, --graph_api_base` | Microsoft Graph API base. Optional. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_POWERBI_CLIENT_ID`
- `CASTOR_POWERBI_TENANT_ID`
- `CASTOR_POWERBI_SECRET` (optional if using certificate)
- `CASTOR_POWERBI_CERTIFICATE` (optional if using secret)
- `CASTOR_POWERBI_API_BASE` (optional)
- `CASTOR_POWERBI_GRAPH_API_BASE` (optional)
- `CASTOR_POWERBI_LOGIN_URL` (optional)
- `CASTOR_POWERBI_SCOPES` (optional)

### `castor-extract-qlik`

| Flag | Description |
| --- | --- |
| `-b, --base-url` | Qlik base URL. |
| `-a, --api-key` | API key. |
| `-e, --except-http-error-statuses <list>` | HTTP status codes to ignore as warnings. |
| `-s, --include-sheets` | Include sheets extraction. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_QLIK_API_KEY`
- `CASTOR_QLIK_BASE_URL`

### `castor-extract-salesforce`

| Flag | Description |
| --- | --- |
| `-u, --username` | Salesforce username. |
| `-p, --password` | Password. |
| `-c, --client-id` | Client ID. |
| `-s, --client-secret` | Client secret. |
| `-t, --security-token` | Security token. |
| `-b, --base-url` | Instance URL. |
| `-o, --output` | Output directory. |
| `--skip-existing` | Keep previously extracted files. |

These environment variables are supported:

- `CASTOR_SALESFORCE_BASE_URL`
- `CASTOR_SALESFORCE_CLIENT_ID`
- `CASTOR_SALESFORCE_CLIENT_SECRET`
- `CASTOR_SALESFORCE_PASSWORD`
- `CASTOR_SALESFORCE_SECURITY_TOKEN`
- `CASTOR_SALESFORCE_USERNAME`

### `castor-extract-salesforce-viz`

| Flag | Description |
| --- | --- |
| `-u, --username` | Salesforce username. |
| `-p, --password` | Password. |
| `-c, --client-id` | Client ID. |
| `-s, --client-secret` | Client secret. |
| `-t, --security-token` | Security token. |
| `-b, --base-url` | Instance URL. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_SALESFORCE_BASE_URL`
- `CASTOR_SALESFORCE_CLIENT_ID`
- `CASTOR_SALESFORCE_CLIENT_SECRET`
- `CASTOR_SALESFORCE_PASSWORD`
- `CASTOR_SALESFORCE_SECURITY_TOKEN`
- `CASTOR_SALESFORCE_USERNAME`

### `castor-extract-sigma`

| Flag | Description |
| --- | --- |
| `-H, --host` | Sigma host. |
| `-c, --client-id` | Client ID. |
| `-a, --api-token` | API key. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_SIGMA_API_TOKEN`
- `CASTOR_SIGMA_CLIENT_ID`
- `CASTOR_SIGMA_HOST`
- `CASTOR_SIGMA_GRANT_TYPE` (optional)

### `castor-extract-strategy`

| Flag | Description |
| --- | --- |
| `-u, --username` | Username. |
| `-p, --password` | Password. |
| `-b, --base-url` | Strategy URL. |
| `-i, --project-ids <list>` | Project IDs. Optional. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CATALOG_STRATEGY_BASE_URL`
- `CATALOG_STRATEGY_PASSWORD`
- `CATALOG_STRATEGY_USERNAME`
- `CATALOG_STRATEGY_PROJECT_IDS` (optional; comma-separated supported)

### `castor-extract-tableau`

| Flag | Description |
| --- | --- |
| `-u, --user` | Tableau user. |
| `-n, --token-name` | Token name. |
| `-p, --password` | Password. |
| `-t, --token` | Token. |
| `-b, --server-url` | Server URL. |
| `-i, --site-id` | Site ID. |
| `--skip-columns` | Skip column extraction. |
| `--skip-fields` | Skip field extraction. |
| `--with-pulse` | Extract Pulse assets. |
| `--page-size` | Custom pagination size. |
| `--ignore-ssl` | Disable SSL verification. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_TABLEAU_SERVER_URL`
- `CASTOR_TABLEAU_SITE_ID`
- `CASTOR_TABLEAU_USER` (required for username/password auth)
- `CASTOR_TABLEAU_PASSWORD` (required for username/password auth)
- `CASTOR_TABLEAU_TOKEN_NAME` (required for PAT auth)
- `CASTOR_TABLEAU_TOKEN` (required for PAT auth)

### `castor-extract-thoughtspot`

| Flag | Description |
| --- | --- |
| `-b, --base_url` | Base URL. |
| `-u, --username` | Username. |
| `-p, --password` | Password. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_THOUGHTSPOT_BASE_URL`
- `CASTOR_THOUGHTSPOT_USERNAME`
- `CASTOR_THOUGHTSPOT_PASSWORD`

### `castor-extract-confluence`

| Flag | Description |
| --- | --- |
| `-a, --account_id` | Confluence account ID. |
| `-b, --base_url` | Confluence base URL. |
| `-t, --token` | API token. |
| `-u, --username` | Username. |
| `--include-archived-spaces` | Include archived spaces. |
| `--include-personal-spaces` | Include personal spaces. |
| `--space-ids-allowed <list>` | Only include these space IDs. |
| `--space-ids-blocked <list>` | Exclude these space IDs. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_CONFLUENCE_ACCOUNT_ID`
- `CASTOR_CONFLUENCE_BASE_URL`
- `CASTOR_CONFLUENCE_TOKEN`
- `CASTOR_CONFLUENCE_USERNAME`

### `castor-extract-notion`

| Flag | Description |
| --- | --- |
| `-t, --token` | Notion token. |
| `-o, --output` | Output directory. |

These environment variables are supported:

- `CASTOR_NOTION_TOKEN`

---

[Castor Extractor]: /docs/catalog/developer/castor-extractor

---

## File Checker

The `castor-file-check` command validates generic warehouse CSV files for the [Warehouse Importer][] before upload. It does not validate JSON output from BI or warehouse extractors. Install and workflow context live on [Castor Extractor][].

This tool checks generic CSV files for data types, mandatory columns, broken references, and more.

## What the Tool Takes as Input

The tool takes two things as input:

* A `file template`, specifying the rules:
  * Mandatory columns
  * Data types
  * Reference (Foreign Keys)
  * Uniqueness (Primary Keys)

:::info[Templates Provided by Catalog]

The templates are provided by Catalog.

:::

* A set of `CSV files`, to be checked

## What the Tool Produces

After running, the tool produces two outputs:

* A `status`, indicating if the files are valid or not
* A summary of detected errors (empty when the files are valid)
* A detailed log of errors, when the `verbose` mode is activated

:::warning[Fix Issues Before Pushing]

If you push files without fixing issues, partial results will be loaded in Catalog.

:::

This tool allows you to track issues, fix them, and re-run the checks. Once you're happy with the % of valid rows, you can push the files to Catalog.

### Example

Here's an example of the output logs (without the verbose mode):

```bash
INFO - DATABASE -- 2 rows -- valid

INFO - SCHEMA -- 12 rows -- valid

INFO - TABLE -- 62 rows -- valid

INFO - COLUMN -- 616 rows -- ERROR (3 invalid rows)
| MISSING_VALUE: 1
| UNAUTHORIZED_VALUE: 2
| UNKNOWN_REFERENCE: 1

INFO - USER -- 4 rows -- valid

INFO - VIEW_DDL -- 2 rows -- valid

INFO - QUERY -- 153 rows -- valid
```

In that case, the files are almost valid, except for `COLUMNS`:

* 3 rows are not valid (4 issues were detected)
* If you push those files to Catalog, 613 columns will be loaded

### List of Issues

The following errors can be detected:

* `MISSING_VALUE`
  * some columns are mandatory and must be provided
  * check the header: some columns might be missing or have a wrong name
  * check the rows: some cells might be empty
* `WRONG_DATATYPE`
  * dates, floats, integers
* `UNAUTHORIZED_VALUE`
  * this column accepts only certain values
  * example: { `TABLE` | `VIEW` | `EXTERNAL`}
  * detailed logs show the list of authorized values
* `DUPLICATE_VALUE`
  * this column must be unique and a duplicate was detected
* `UNKNOWN_REFERENCE`
  * the column references another column which has not been found
  * check that the reference is not broken
  * check that the referred row is valid itself (otherwise it will be considered as unknown)
* `REPEATED_QUOTES`
  * the value is surrounded by repeated quotes that will be interpreted as such
  * the checker fails to avoid loading quoted text in Catalog, such as `"dim_client"`
  * it's probably not intentional and it generally happens when using spreadsheet to generate the CSV (File > Export As > CSV)

## Templates

### Generic Warehouse

It allows you to push generic warehouse CSV files to Catalog:

* Databases
* Schemas
* Tables
* etc.

More info about this template in the [warehouse importer documentation][].

:::info[Current Template Support]

For now, this is the only template provided with the tool.

:::

## Usage

```bash
castor-file-check [arguments]
```

* `--directory`: Path to the directory containing your CSV files
* `--verbose`: (optional) When provided, a log of each invalid row will be displayed

The given `directory` must contain all files mentioned in the [warehouse importer template][]:

* `database.csv`
* `schema.csv`
* etc.

:::info[File Prefixes]

The files can be prefixed (with timestamps or anything else):

* `978193-database.CSV`
* `978193-schema.CSV`

If several files are found, the most recently created will be used.

:::

[warehouse importer documentation]: /docs/catalog/developer/catalog-apis/warehouse-importer
[warehouse importer template]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Warehouse Importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Castor Extractor]: /docs/catalog/developer/castor-extractor

---

## Castor Extractor

The **castor-extractor** package is a set of command-line tools that connect to your data sources, read metadata only, and write local JSON or CSV files. You upload those files to Catalog when an integration is **client managed**: Catalog does not connect to your BI tool or warehouse directly. You control credentials, scheduling, and network access on your side.

**Developers** is the documentation area for engineers automating Catalog; **castor-extractor** is the installable Python package documented in this section.

## When to Use the Extractor

Choose client-managed extraction when Catalog cannot reach your source over the network, or when your security policy requires metadata to leave your environment only as files you send.

Typical cases include:

- BI tools listed as client managed in [Quick Set Up][] (for example during a trial one-time load, or after trial when you schedule daily pushes to a Catalog GCP bucket).
- Warehouses or other sources where you run extraction on your own hosts instead of granting Catalog a persistent connection.
- Custom or generic warehouse CSV flows where you build files yourself and validate them before upload.

If Catalog manages the connection for your technology, configure credentials in **Settings > Integrations** and Catalog runs extraction. You do not need the extractor for that path.

## How Extraction Fits in Catalog Setup

The extractor sits between your systems and Catalog ingestion. Catalog never reads your warehouse or BI tool during client-managed sync; it ingests the files you upload.

```mermaid
flowchart LR
  A[Your warehouse or BI tool] --> B[castor-extract-* CLI]
  B --> C[Local JSON or CSV files]
  C --> D[castor-file-check optional]
  D --> E[castor-upload]
  E --> F[Catalog GCP bucket]
  F --> G[Catalog ingestion]
```

In practice, follow this order:

1. Install the `castor-extractor` package and the extra for your source (for example `[tableau]` or `[snowflake]`). See [Installation][] below.
2. Extract metadata with the CLI for your technology (for example `castor-extract-tableau`). Output lands in a directory you choose.
3. Optionally validate warehouse CSV files with [File Checker][] before upload.
4. Upload files with `castor-upload` using your Catalog token and source ID.
5. **Wait for ingestion** in Catalog after the upload completes.

Technology-specific credentials, flags, and environment variables are in the [BI Tools extraction guides][] and [Warehouse extraction guides][].

## What Gets Extracted

The package writes structured metadata files to disk. Asset types depend on the source.

### Warehouse Assets

Warehouse metadata typically includes:

- `databases`
- `schemas`
- `tables`
- `columns`
- `queries`

### Visualization Assets

Visualization metadata typically includes:

- `dashboards`
- `users`
- `folders`

### Knowledge Assets

Knowledge content can come from tools such as Confluence and Notion when you use those extractors.

The package also includes utilities to push metadata into Catalog:

- [File Checker][] to validate your [Warehouse Importer][] CSV files before upload
- `castor-upload` to push extracted files to Google Cloud Storage (GCS) for Catalog ingestion

## Before You Begin

- Castor Extractor requires Python 3.10, 3.11, 3.12, or 3.13.
- A source ID from Catalog for the integration you are loading. The upload CLI flag is `--source_id`.
- A Catalog API token for upload. Catalog administrators create tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for generation and rotation steps.

## Installation

### Create an Isolated Environment

We recommend creating an isolated Python environment. The following example uses `pyenv`:

```bash
brew install pyenv pyenv-virtualenv

pyenv install -v 3.11.9
pyenv virtualenv 3.11.9 castor-env
pyenv shell castor-env
python --version   # should print 3.11.9
```

If `pyenv shell` doesn't work, add the following lines to your shell profile and restart your terminal:

```bash
eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"
```

### Install the Package

```bash
pip install --upgrade pip
pip install castor-extractor
```

Most sources need an extra. Install only the ones you use:

```bash
pip install castor-extractor[all]
# or only one integration, for example:
pip install castor-extractor[bigquery]
pip install castor-extractor[count]
pip install castor-extractor[databricks]
pip install castor-extractor[glue-athena]
pip install castor-extractor[looker]
pip install castor-extractor[lookerstudio]
pip install castor-extractor[metabase]
pip install castor-extractor[mysql]
pip install castor-extractor[omni]
pip install castor-extractor[powerbi]
pip install castor-extractor[qlik]
pip install castor-extractor[postgres]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
pip install castor-extractor[sqlserver]
pip install castor-extractor[strategy]
pip install castor-extractor[tableau]
```

### Create an Output Directory

```bash
mkdir -p /tmp/castor
```

## Example: Extract Tableau Metadata

After you install `castor-extractor[tableau]`, run the Tableau extractor from a terminal. The command writes JSON files under your output directory and prints progress in the log.

```bash
castor-extract-tableau \
  --username YOUR_TABLEAU_USER \
  --password YOUR_TABLEAU_PASSWORD \
  --server-url https://YOUR_SITE.online.tableau.com \
  --site-id YOUR_SITE_ID \
  --output /tmp/castor
```

Example log output:

```text
INFO - Logging in using user and password authentication
INFO - Signed into https://eu-west-1a.online.tableau.com as user with id ****
INFO - Extracting USER from Tableau API
INFO - Fetching USER
INFO - Querying all users on site
INFO - Wrote output file: /tmp/castor/1649078755-custom_sql_queries.json
INFO - Wrote output file: /tmp/castor/1649078755-summary.json
```

For PAT sign-in, flags, and environment variables, see [Tableau extraction][].

## 1. Extract Metadata

You can either use an extractor or add the files to the directory.

### Using an Extractor Package

Using one of the packages, for example, `pip install castor-extractor[snowflake]` you can extract the data directly and load it into the output directory created during setup to upload to Catalog.

```bash
castor-extract-snowflake \
  --account xy12345.eu-west-1 \
  --user svc_castor \
  --password secret \
  --output /tmp/castor
```

See the full list in [Castor Extractor Reference][].

### Using a Folder

Place your generic CSV files in the output directory.

## 2. Validate Data (Optional)

This step is optional. Run `castor-file-check` to validate your generic warehouse CSV files before uploading.

```bash
castor-file-check --directory /tmp/castor
```

See [Castor Extractor Reference][] for all `castor-file-check` flags.

## 3. Upload to Catalog

You need a source ID from Catalog and a Catalog API token from **Settings > API**. See [Getting Your Catalog API Keys][] for token creation and rotation.

```bash
castor-upload \
  --token YOUR_CATALOG_TOKEN \
  --source_id YOUR_SOURCE_ID \
  --file_type WAREHOUSE \
  --zone EU \
  --directory_path /tmp/castor
```

See [Castor Extractor Reference][] for all `castor-upload` flags and [zone selection][].

## Using Environment Variables

You can set environment variables for extract and upload commands instead of passing every flag on the command line.

```bash title="Snowflake extractor example"
export CASTOR_SNOWFLAKE_ACCOUNT="xy12345.eu-west-1"
export CASTOR_SNOWFLAKE_USER="svc_castor"
export CASTOR_SNOWFLAKE_PASSWORD="secret"
export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"

castor-extract-snowflake
```

```bash title="Castor uploader example"
export CASTOR_UPLOADER_TOKEN="your-token"
export CASTOR_UPLOADER_SOURCE_ID="your-source-id"
export CASTOR_UPLOADER_FILE_TYPE="WAREHOUSE"
export CASTOR_UPLOADER_ZONE="US"
export CASTOR_UPLOADER_DIRECTORY_PATH="/tmp/castor"

castor-upload
```

## Troubleshooting

If you have problems uploading your files, you can increase the timeout or configure retries with environment variables:

- `CASTOR_TIMEOUT_OVERRIDE`: seconds before timeout (default 60)
- `CASTOR_RETRY_OVERRIDE`: number of retries (default 1)

## What's Next?

- [BI Tools extraction guides][] for Looker, Tableau, Metabase, and other visualization sources
- [Warehouse extraction guides][] for Snowflake, Postgres, BigQuery, and other warehouses
- [File Checker][] for validating generic warehouse CSV files
- [Quick Set Up][] for Catalog managed versus client-managed onboarding
- [Warehouse Importer][] when you supply generic warehouse CSV files

[Castor Extractor Reference]: /docs/catalog/developer/castor-extractor/castor-extractor-reference
[Installation]: #installation
[File Checker]: /docs/catalog/developer/castor-extractor/file-checker
[BI Tools extraction guides]: /docs/catalog/developer/castor-extractor/bi-tools
[Warehouse extraction guides]: /docs/catalog/developer/castor-extractor/warehouse
[Tableau extraction]: /docs/catalog/developer/castor-extractor/bi-tools/tableau
[Quick Set Up]: /docs/catalog/quick-set-up
[Warehouse Importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[zone selection]: /docs/catalog/developer/castor-extractor/castor-extractor-reference#zone-selection

---

## BigQuery

Extract BigQuery metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We strongly recommend creating a dedicated user to extract your metadata.

Follow the [instructions for creating the Catalog user on BigQuery][] to create the `catalog` user.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-bigquery [arguments]
```

The script will run and display logs as following:

```bash
INFO - Credentials fetched from /.../keys/your-service-account.json
INFO - Available projects: ['project-1', 'project-2']

INFO - Extracting `DATABASE` ...
INFO - Results stored to /tmp/catalog/1649082442-database.csv

...

INFO - Extracting `USER` ...
INFO - Results stored to /tmp/catalog/1649082442-user.csv
INFO - Wrote output file: /tmp/catalog/1649082442-summary.json
```

### Credentials

* `-k`, `--token`: Token provided by Catalog

### Other Arguments

* `-o`, `--output`: Target folder to store the extracted files

### Optional Arguments

* `--skip-existing`: Skip files already extracted instead of replacing them
* `--db-allowed`: GCP projects you want to extract
* `--db-blocked`: GCP projects you do not want to extract

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials/service-account.json"
export CASTOR_OUTPUT_DIRECTORY="/tmp/catalog"
```

Then the script can be executed without any arguments:

```bash
castor-extract-bigquery
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-bigquery --output /tmp/catalog
```

## Database Filtering

:::info[GCP Projects]

In GCP, databases are often referred to as Projects.

:::

Database filters are optional. If you don't specify any filter, all available databases will be extracted (depending on the provided credentials)

```bash
# extract all but <...>
castor-extract-bigquery --db-blocked ZZ_DEPRECATED_PROJECT

# extract only <...>
castor-extract-bigquery --db-allowed PROJECT_1 PROJECT_2 PROJECT_3

# mixed (not really useful, but still doable)
castor-extract-bigquery --db-allowed PROJECT_1 --db-blocked ZZ_DEPRECATED_PROJECT
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating the Catalog user on BigQuery]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/bigquery#2-create-castor-user-on-bigquery

---

## Databricks

Extract Databricks metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We recommend creating a dedicated service principal to extract your metadata.

Follow the [instructions for creating a service principal on Databricks][] to create the OAuth credentials.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-databricks [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting `CATALOG` ...
INFO - Available databases: ['main', 'samples']
INFO - Extracted 8 databases to /tmp/catalog/1780550718-database.csv
INFO - Extracted 203 schemas to /tmp/catalog/1780550718-schema.csv

...

INFO - Extracted 128 view_ddl to /tmp/catalog/1780550718-view_ddl.csv
INFO - Wrote output file: /tmp/catalog/1780550718-summary.json
```

### Credentials

- `-H`, `--host`: Databricks workspace hostname (for example, `https://dbc-abc12345-6789.cloud.databricks.com`)
- `-p`, `--http-path`: SQL warehouse path (for example, `/sql/1.0/warehouses/xxxxx`)
- `--client-id`: OAuth client ID from your service principal
- `--client-secret`: OAuth client secret from your service principal

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--catalog-allowed`: Catalogs you want to extract
- `--catalog-blocked`: Catalogs you do not want to extract
- `--skip-existing`: Skip files already extracted instead of replacing them

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export CASTOR_DATABRICKS_HOST=https://dbc-abc12345-6789.cloud.databricks.com
export CASTOR_DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/xxxxx
export CASTOR_DATABRICKS_CLIENT_ID=your-client-id
export CASTOR_DATABRICKS_CLIENT_SECRET=your-client-secret

export CASTOR_OUTPUT_DIRECTORY="tmp/catalog_output"
```

Then the script can be executed without any arguments:

```bash
castor-extract-databricks
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-databricks --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating a service principal on Databricks]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/databricks#1-create-a-service-principal-for-oauth-m2m-authentication

---

## Warehouse

These pages document warehouse **`castor-extract-*`** commands for client-managed sync. Install the package and workflow steps on [Castor Extractor][], then follow the guide for your warehouse.

---

[Castor Extractor]: /docs/catalog/developer/castor-extractor

---

## MySQL

Extract MySQL metadata into Catalog using the castor-extractor package.

## Prerequisites

Follow the [castor-extractor installation instructions][] before running the extraction.

We strongly recommend creating a dedicated user to extract your metadata.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-mysql [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting `DATABASE` ...
INFO - Extracting database: query 1/1
INFO - Results stored to /tmp/mysql/1706004977-database.csv
...

INFO - Extracting `USER` ...
INFO - Extracting user: query 1/1
INFO - Results stored to /tmp/mysql/1706004977-user.csv

INFO - Wrote output file: /tmp/mysql/1706004977-summary.json
```

### Credentials

- `-H`, `--host`: hostname
- `-P`, `--port`: port number
- `-u`, `--user`: user
- `-p`, `--password`: password

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--skip-existing`: Skip files already extracted instead of replacing them

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export CASTOR_MYSQL_HOST=127.0.0.0
export CASTOR_MYSQL_PORT=3306
export CASTOR_MYSQL_USER=extraction_user
export CASTOR_MYSQL_PASSWORD=******

export CASTOR_OUTPUT_DIRECTORY="/tmp/catalog"
```

Then the script can be executed without any arguments:

```bash
castor-extract-mysql
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-mysql --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/

---

## Postgres

Extract Postgres metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We strongly recommend creating a dedicated user to extract your metadata.

Follow the [instructions for creating the Catalog user on Postgres][] to create the `catalog` user.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-postgres [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting `DATABASE` ...
INFO - Results stored to /tmp/catalog/1649083626-database.csv

...

INFO - Wrote output file: /tmp/catalog/1649083626-summary.json
```

### Credentials

- `-H`, `--host`: hostname
- `-P`, `--port`: port number
- `-d`, `--database`: database name
- `-u`, `--user`: user
- `-p`, `--password`: password

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--skip-existing`: Skip files already extracted instead of replacing them

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export CASTOR_POSTGRES_HOST=127.0.0.0
export CASTOR_POSTGRES_PORT=5439
export CASTOR_POSTGRES_DATABASE=db_name
export CASTOR_POSTGRES_USER=extraction_user
export CASTOR_POSTGRES_PASSWORD=******

export CASTOR_OUTPUT_DIRECTORY="/tmp/catalog"
```

Then the script can be executed without any arguments:

```bash
castor-extract-postgres
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-postgres --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating the Catalog user on Postgres]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/postgres#2-create-castor-user-on-postgres

---

## Redshift

Extract Redshift metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We strongly recommend creating a dedicated user to extract your metadata.

Follow the [instructions for creating the Catalog user on Redshift][] to create the `catalog` user.

:::danger[SSL Certificate Verification]

This client connects to Redshift using `sslmode=verify-ca`, which means your certificates must be up-to-date. For more information, see [AWS Redshift SSL support][].

:::

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-redshift [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting `DATABASE` ...
INFO - Results stored to /tmp/catalog/1649083626-database.csv

...

INFO - Extracting `USER` ...
INFO - Results stored to /tmp/catalog/1649083626-user.csv
INFO - Wrote output file: /tmp/catalog/1649083626-summary.json
```

### Credentials

- `-H`, `--host`: hostname
- `-P`, `--port`: port number
- `-d`, `--database`: database name
- `-u`, `--user`: user
- `-p`, `--password`: password

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--skip-existing`: Skip files already extracted instead of replacing them
- `--serverless`: Enables extraction for Redshift Serverless

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export CASTOR_REDSHIFT_HOST=127.0.0.0
export CASTOR_REDSHIFT_PORT=5439
export CASTOR_REDSHIFT_DATABASE=db_name
export CASTOR_REDSHIFT_USER=extraction_user
export CASTOR_REDSHIFT_PASSWORD=******

# Optional to enable Redshift Serverless
CASTOR_REDSHIFT_SERVERLESS=true

export CASTOR_OUTPUT_DIRECTORY="/tmp/catalog"
```

Then the script can be executed without any arguments:

```bash
castor-extract-redshift
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-redshift --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating the Catalog user on Redshift]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/redshift#2-create-castor-user-on-redshift
[AWS Redshift SSL support]: https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html

---

## Salesforce

Extract Salesforce metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We strongly recommend creating a dedicated user to extract your metadata.

Follow the [instructions for creating the Catalog user on Salesforce][] to create the `catalog` user.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-salesforce [arguments]
```

### Credentials

- `-u`, `--username`: Salesforce username
- `-p`, `--password`: Salesforce password
- `-c`, `--client-id`: Salesforce client ID
- `-s`, `--client-secret`: Salesforce client secret
- `-t`, `--security-token`: Salesforce security token
- `-b`, `--base-url`: Salesforce instance URL

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--skip-existing`: Skip files already extracted instead of replacing them

:::info[Help]

You can also get help with the `--help` argument.

:::

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating the Catalog user on Salesforce]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/salesforce

---

## Snowflake

Extract Snowflake metadata into Catalog using the castor-extractor package.

## Prerequisites

:::warning[Installation Required]

Follow the [castor-extractor installation instructions][] before running the extraction.

:::

We strongly recommend creating a dedicated user to extract your metadata.

Follow the [instructions for creating the Catalog user on Snowflake][] to create the `catalog` user.

## Run Extraction Script

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-snowflake [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting `DATABASE` ...
INFO - Results stored to /tmp/catalog/1649083626-database.csv

...

INFO - Extracting `USER` ...
INFO - Results stored to /tmp/catalog/1649083626-user.csv
INFO - Wrote output file: /tmp/catalog/1649083626-summary.json
```

### Credentials

- `-a`, `--account`: Snowflake [account identifier][]
- `-u`, `--user`: Snowflake user
- `-p`, `--password`: Snowflake password

If you use key pair authentication ([Snowflake key pair auth documentation][]), with no passphrase, replace password with:

- `-pk`, `--private_key`: Snowflake private key

:::warning[Private Key Format]

The key should be formatted as follows:

```text
-----BEGIN PRIVATE KEY-----\\xxxxxPRIVATE_KEYxxxxx\\n-----END PRIVATE KEY-----
```

:::

### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

- `--skip-existing`: Skip files already extracted instead of replacing them
- `--fetch-transient`: Will fetch transient tables if added
- `--warehouse`: Use a specific `WAREHOUSE` to run extraction queries
- `--role`: Use a specific `ROLE` to run extraction queries
- `--db-allowed`: Databases you want to extract
- `--db-blocked`: Databases you do not want to extract
- `--insecure-mode`: Turns off OCSP checking for Snowflake Client. Use when facing OCSP issues
- `--query_blocked`: Used to exclude specific SQL queries by applying filters expressed as `NOT ILIKE <pattern>`. Example for patterns:

```text
query_blocked = [
   "SELECT COUNT(*) FROM stats",  # Excludes queries matching this exact pattern (case-insensitive)
   "SELECT * FROM temp_%",        # Excludes all queries starting with 'SELECT * FROM temp_'
   "%pg_catalog%",                # Excludes queries referencing PostgreSQL system tables
]
```

:::info[Help]

You can also get help with the `--help` argument.

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```bash
export CASTOR_SNOWFLAKE_ACCOUNT=aCC0un7.eu-central-1
export CASTOR_SNOWFLAKE_USER=us3R
export CASTOR_SNOWFLAKE_PASSWORD=******
export CASTOR_SNOWFLAKE_INSECURE_MODE=True

export CASTOR_OUTPUT_DIRECTORY="tmp/catalog_output"
```

Then the script can be executed without any arguments:

```bash
castor-extract-snowflake
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-snowflake --output /tmp/catalog
```

## Database Filtering

Database filters are optional. If you don't specify any filter, all available databases will be extracted (depending on the provided credentials)

```bash
# extract all but <...>
castor-extract-snowflake --db-blocked ZZ_DEPRECATED_DB

# extract only <...>
castor-extract-snowflake --db-allowed DB_1 DB_2 DB_2

# mixed (not really useful, but still doable)
castor-extract-snowflake --db-allowed DB_1 --db-blocked ZZ_DEPRECATED_DB
```

## ROLE and WAREHOUSE

Here is how the extractor connects to Snowflake:

1. Use the provided `--role` and `--warehouse` if any
2. Otherwise, use the default `ROLE` and `WAREHOUSE`
3. If no defaults are set, use an arbitrary available `ROLE` and `WAREHOUSE`

The `--role` and `--warehouse` arguments are useful when you need to enforce specific values.

If you extract your data with user `catalog` (see [instructions for creating the Catalog user on Snowflake][]), you don't need to specify those values:

- `METADATA_WH` is the only available warehouse, it will be used by default
- `METADATA_VIEWER_ROLE` is the only available role, it will be used by default

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/
[instructions for creating the Catalog user on Snowflake]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/snowflake#2-create-castor-user-on-snowflake
[account identifier]: https://docs.snowflake.com/en/user-guide/admin-account-identifier.html
[Snowflake key pair auth documentation]: https://docs.snowflake.com/en/user-guide/key-pair-auth

---

## SQL Server

Extract metadata from your SQL Server databases using the Coalesce extraction script.

## Prerequisites

Before you begin, complete the following:

### Install Castor Extractor

Follow the [castor-extractor installation instructions][] on PyPI.

### Create a User

Create a dedicated user to extract your metadata.

## Run the Extraction Script

After installing the package, run the following command in your terminal:

```bash
castor-extract-sqlserver [arguments]
```

The script displays logs as it runs:

```bash
INFO - Extracting `DATABASE` ...
INFO - Extracting database: query 1/1
INFO - Results stored to /tmp/sqlserver/1706004977-database.csv
...
INFO - Extracting `USER` ...
INFO - Extracting user: query 1/1
INFO - Results stored to /tmp/sqlserver/1706004977-user.csv
INFO - Wrote output file: /tmp/sqlserver/1706004977-summary.json
```

## Arguments

### Credentials

Use these arguments to authenticate with your SQL Server instance:

- **-H, --host:** MSSQL host address.
- **-P, --port:** MSSQL port number.
- **-u, --user:** MSSQL username.
- **-p, --password:** MSSQL password.

### Output

- **-o, --output:** Target folder to store the extracted files.

### Optional Arguments

- **--skip-existing:** Skip files already extracted instead of replacing them.
- **--db-allowed:** List of databases that should be extracted.
- **--db-blocked:** List of databases that should not be extracted.

The values for `--db-allowed` and `--db-blocked` are case-sensitive. Make sure to use the exact values present in column `sys.databases.name` for the respective databases.

### Default Database

- **--default-db:** Default database to use when connecting to SQL Server.

This argument is optional and usually not required. The connector automatically discovers and extracts all databases the user has access to.

It becomes necessary in some environments, notably Azure SQL or restricted SQL Server setups, where the user doesn't have access to the master database. In those cases, SQL Server may fail during connection with the following error:

```text
The server principal "user" is not able to access the database "master" under the current security context. (DB-Lib error message 20018)
```

When this happens, SQL Server attempts to connect to master by default. Setting `--default-db` forces the connector to connect to a database the user can access, preventing the error.

## Use Environment Variables

If you don't want to specify arguments every time, set the following environment variables in your `.bashrc`:

```bash
export CASTOR_SQLSERVER_HOST=127.0.0.0
export CASTOR_SQLSERVER_PORT=3306
export CASTOR_SQLSERVER_USER=extraction_user
export CASTOR_SQLSERVER_PASSWORD=******
export CASTOR_SQLSERVER_DEFAULT_DB=database # optional
export CASTOR_OUTPUT_DIRECTORY="/tmp/catalog"
```

Then run the script without any arguments:

```bash
castor-extract-sqlserver
```

You can also run the script with partial arguments. The script looks in your environment variables as a fallback:

```bash
castor-extract-sqlserver --output /tmp/catalog
```

[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/

---

## BI Importer

BI tools share common structures. We have defined a format so you can load your dashboard metadata into [Catalog][]. Send a CSV file as described below to our endpoint using the [Catalog Uploader][].

:::info[API Token]

Catalog administrators generate API tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for creation steps and [rotation guidance][] when a token expires or changes.

:::

:::danger[File Naming]

Always prefix file names with a Unix timestamp.

:::

## File Expected Content

If you build these files in Excel or Google Sheets, save as **CSV (Comma delimited)**, not **CSV UTF-8**.

<!-- vale off -->

### 1. dashboards.csv

<ButtonLink to="/downloads/dashboards.csv">
  dashboards.csv example file
</ButtonLink>

<BetterMDTables
  headers={['', '✅ Required', 'Description', 'Example']}
  rows={[
    ['id 🔑', '✅', 'Id of the dashboard (usually from the API)', '`hlaAw17G`'],
    ['dashboard_type 🔑', '✅', 'Can be `DASHBOARD` or `TILE`. A tile is a dashboard element. A Viz Model is a semantic or modeling layer.', '`DASHBOARD|TILE``|VIZ_MODEL`'],
    ['name', '✅', 'The Dashboard name', '`MRR Report`'],
    ['folder_path', '✅', 'The full path of the dashboard in your folder structure (do not include the dashboard name in the path)', '`/Home/Finance/2021`'],
    ['url', '✅', 'The url of the dashboard (to redirect users from Catalog to it)', '`https://tableau.com/catalog/mrr_report-hlaAw17G`'],
    ['description', '❌', 'Dashboard description', '`Compute MRR records for 2021.`'],
    ['user_name', '❌', 'The email of the dashboard owner', '`arnaud@coalesce.io`'],
    ['view_count', '❌', 'Used to compute popularity', '`255`'],
    ['parent_tables', '❌', 'Tables used in the dashboard', "`['dbname.schema.table_1', 'dbname.schema.table_2']`"],
    ['parent_dashboards', '❌', 'If the dashboard is built using another dashboard, dashboard id', "`['dashboard_id_1', 'dashboard_id_2']`"]
  ]}
/>

To push dashboard lineage to your warehouse data in BigQuery, Snowflake, Redshift, and other warehouses, provide either the `Parent Tables` field or the `dashboard_queries.csv` file.

### 2. dashboard_queries.csv (optional)

<ButtonLink to="/downloads/dashboard_queries.csv">
  dashboard_queries.csv example file
</ButtonLink>

<BetterMDTables
  headers={['Column Name', '✅ Required', 'Description', 'Example']}
  rows={[
    ['dashboard_id', '✅', 'Relate to the dashboard the query belongs to.', '`hlaAw17G`'],
    ['dashboard_type', '✅', 'Relate to the dashboard the query belongs to.', '`dashboard|tile`'],
    ['text', '✅', 'The SQL query', 'SELECT id FROM my_schema.foo'],
    ['database_name', '✅', 'The technical name of the database the query was executed on *', '`dw_prod`']
  ]}
/>

Catalog needs the `database_name` column to resolve ambiguities when table names are not prefixed in the query. In the example above, the complete table name is `dw_prod.my_schema.foo`.

### 3. dashboard_fields.csv (optional)

<ButtonLink to="/downloads/dashboard_fields (2).csv">
  dashboard_fields.csv example file
</ButtonLink>

<BetterMDTables
  headers={['Column Name', '✅ Required', 'Description', 'Example']}
  rows={[
    ['dashboard_id', '✅', 'Relate to the dashboard the query belongs to.', '`hlaAw17G`'],
    ['external_id', '✅', 'The id of the field in the source', '`hlaAw17G_28`'],
    ['name', '✅', 'Name of the field', 'arr_certified'],
    ['role', '✅', 'Role of the field', '[MEASURE, DIMENSION]'],
    ['data_type', '❌', 'Type of data', 'Number'],
    ['description', '❌', 'Field description', 'Measure of ARR'],
    ['is_primary_key', '❌', 'Default to False', '`True | False`'],
    ['label', '❌', 'Label of the field', 'ARR'],
    ['view_name', '❌', 'Name of the view the field should be grouped in', 'financial_metrics'],
    ['view_label', '❌', 'Label of the view the field should be grouped in', 'Financial Metrics'],
    ['child_dashboards', '❌', 'Dashboards using the field', "`['dashboard_id_1', 'dashboard_id_2']`"],
    ['parent_columns', '❌', 'Columns used in the field', "`['dbname.schema.table_1.columnA', 'dbname.schema.table_2.columnB']`"]
  ]}
/>

Dashboard fields are only supported for dashboards with `dashboard_type = VIZ_MODEL`.

<!-- vale on -->

[Catalog]: https://app.castordoc.com/metadata-editor/tables
[Catalog Uploader]: /docs/catalog/developer/castor-extractor
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[rotation guidance]: /docs/catalog/developer/getting-your-api-keys#rotate-or-revoke-a-token

---

## Catalog APIs

Use the Catalog Public API to read and write metadata, or use the metadata importers to push dashboards, tables, and lineage from external sources.

## Public API

[Catalog Public API][]

## Metadata Importer

The following endpoints are available to push and update dashboards, tables, and lineage to Catalog:

- [BI Importer][]
- [Warehouse Importer][]
- [Manual Lineage Importer][]

:::info[API Token]

Catalog administrators generate API tokens in **Settings > API**. Use them with the importers below, the in-app API playground, MCP, and [Castor Extractor][] uploads.

- Create and manage tokens: [Getting Your Catalog API Keys][].
- For Quality API access, see [Getting Started Quality API][].

:::

[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[Warehouse Importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Manual Lineage Importer]: /docs/catalog/developer/catalog-apis/manual-lineage-importer
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Castor Extractor]: /docs/catalog/developer/castor-extractor
[Getting Started Quality API]: https://docs.synq.io/api-reference/getting-started

---

## Manual Lineage Importer

Catalog computes lineage from warehouse and BI metadata. When Catalog cannot infer a relationship, you can supply manual lineage in a CSV file. Contact [support@coalesce.io][] to set up ingestion. Manual lineage remains in Catalog until you delete it.

If you build the file in Excel or Google Sheets, save as **CSV (Comma delimited)**, not **CSV UTF-8**.

## File Expected Content

| Column | ✅ Required | Description | Example |
| -------- | ------------- | ------------- | --------- |
| `parent_type` | ✅ | Type of parent asset | `TABLE` |
| `parent_key` | ✅ | Key of parent asset | `my_warehouse_name.database_name.schema_name.table_name` |
| `child_type` | ✅ | Type of child asset | `TABLE` |
| `child_key` | ✅ | Key of child asset | `my_warehouse_name.database_name.schema_name.table_name` |

<ButtonLink to="/downloads/manual_lineage_example.csv">
  CSV example
</ButtonLink>

### Asset Types

Accepted asset types are:

* `TABLE`
* `COLUMN`
* `DASHBOARD`
* `DASHBOARD_FIELD`

Accepted link types are:

* `TABLE` > `TABLE`
* `TABLE` > `DASHBOARD`
* `DASHBOARD` > `DASHBOARD`
* `COLUMN` > `COLUMN`
* `COLUMN` > `DASHBOARD_FIELD`
* `COLUMN` > `DASHBOARD`
* `DASHBOARD_FIELD` > `DASHBOARD_FIELD`
* `DASHBOARD_FIELD` > `DASHBOARD`

### Asset Key Formats

Use the following conventions to identify assets:

* `TABLE` - `warehouse_name.database_name.schema_name.table_name`
* `COLUMN` - `warehouse_name.database_name.schema_name.table_name.column_name`
* `DASHBOARD` - `tool_name/folder_path/dashboard_name`
* `DASHBOARD_FIELD` - `tool_name/folder_path/dashboard_name/dashboard_field_label`

[support@coalesce.io]: mailto:support@coalesce.io

---

## Catalog Public API

The Catalog Public API lets you programmatically read and update metadata (tables, columns, dashboards, lineage, quality tests, and more) so you can:

- **Automate metadata workflows** (sync from external tools, enforce governance rules, keep systems in sync).
- **Build custom integrations** with internal apps, data products, or orchestration tools.
- **Power advanced use cases** such as AI assistants, reverse-ETL lineage, or custom reporting.

All endpoints and schemas are documented in the Catalog GraphQL API reference:

- **Full reference:** [Catalog GraphQL API][]

Each operation page includes a **Try it** panel to send live requests from the docs.

## Authentication and Tokens

The Public API uses a **Catalog API token** for authentication. Pass it as a bearer token in the `Authorization` header.

Catalog administrators create tokens in **Settings > API** with **Read Only** or **Read & Write** scope. See [Getting Your Catalog API Keys][] for step-by-step generation, expiration, rotation, and revocation.

Use your token in:

- Your own scripts and services calling the Public API
- **Metadata Importer** flows ([BI Importer][], [Warehouse Importer][], [Manual Lineage Importer][])
- The **in-app Catalog API playground** (see below)

Store tokens in a secrets manager and never commit them to Git.

## API Base URLs

Use the base URL that matches your region:

- **EU:** `https://api.castordoc.com`
- **US:** `https://api.us.castordoc.com`

## In-App Catalog API Playground

To reduce friction and make it easier to get started, the app includes an **in-app Catalog API playground**.

With the playground, you can:

- **Send real Catalog API requests directly from the product** in a safe, guided way.
- **Paste your Catalog API token** and immediately run live requests against your environment.
- Start from a **pre-configured example query** that works out of the box. Click **Run** to see results.
- Use the **autocomplete query editor** to discover available endpoints and parameters faster (for example, typing `folders` suggests valid folder paths).

This greatly **reduces time-to-first-successful-request** and helps teams:

- Validate that their token and permissions work.
- Explore the shape of the API before writing any code.
- Quickly prototype new initiatives leveraging the Catalog API.

[Catalog GraphQL API]: /docs/catalog/api
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[Warehouse Importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Manual Lineage Importer]: /docs/catalog/developer/catalog-apis/manual-lineage-importer

---

## Warehouse Importer

Warehouses share common structures. We have defined a format so you can load your metadata into Catalog. Fill in the 7 [files][] below and push them to our endpoint using the [Catalog Uploader][].

:::info[API Token]

Catalog administrators generate API tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for creation steps and [rotation guidance][] when a token expires or changes.

:::

:::danger[Required Files and Naming]

All 7 files are mandatory and data must make sense. If you add a column in the column file but the table that contains it is not in the table file, it will fail to load into Catalog.

Always prefix file names with a Unix timestamp.

:::

## CSV Formatting

If you build these files in Excel or Google Sheets, save as **CSV (Comma delimited)**, not **CSV UTF-8**. UTF-8 exports can include a byte-order mark that complicates ingestion.

Here's an example of a very simple CSV file:

### List Field Values

Some fields such as tags are typed as `list[string]`. In that case, several formats are accepted:

```text
- list "['a', 'b']"
- tuples "('a', 'b')"
- sets "{'a', 'b', 'c'}"

Empty list allowed: []
Singleton allowed: 'a'
Multiple types allowed: "['foo', 100, 19.8]"
```

Fields containing commas must be quoted. See **Quoting** below.

### Forbidden Characters

* Column separator is the comma `,`
* Row separator is the carriage return

### Quoting

Most string fields such as table names and column names should not contain commas or carriage returns. Generally the problem comes with large text fields, such as SQL queries or descriptions.

If you have any doubts, you can quote all your text fields:

<!-- vale off -->

## Files

🔑 Primary Key (must be unique)

🔐 Foreign Key (must reference an existing entry)

❓Optional (empty string in the CSV)

### **1. Database**

<ButtonLink to="/downloads/database.csv">
  database.csv
</ButtonLink>

#### Database Fields

`id` string 🔑

`database_name` string

### **2. Schema**

<ButtonLink to="/downloads/schema (3).csv">
  schema (3).csv
</ButtonLink>

#### Schema Fields

`id` string 🔑

`database_id` string > database.id 🔐

`schema_name` string

`description` string ❓

`tags` list[string] ❓

### **3. Table**

<ButtonLink to="/downloads/table (5).csv">
  table (5).csv
</ButtonLink>

#### Table Fields

`id` string 🔑

`schema_id` string > schema.id 🔐

`table_name` string

`description` string ❓

`tags` list[string] ❓

`type` enum `{TABLE | VIEW | EXTERNAL | TOPIC}`

`owner_external_id` string > user.id ❓

### **4. Column**

<ButtonLink to="/downloads/column (1).csv">
  column (1).csv
</ButtonLink>

#### Column Fields

`id` string 🔑

`table_id` string > table.id 🔐

`column_name` string

`description` string ❓

`data_type` enum: `{ BOOLEAN | INTEGER | FLOAT | STRING | ... | CUSTOM }`

`ordinal_position` positive integer ❓

### **5. Query**

<ButtonLink to="/downloads/query (7).csv">
  query (7).csv
</ButtonLink>

Upload the Query file even when it has no rows. The file itself is required.

We only ingest queries that ran the day before metadata ingestion. Include only those queries in the file; others are ignored.

#### Query Fields

`query_id` string > query.id

`database_id` string > database.id 🔐

`database_name` string > database.name

`schema_name` string > schema.name

`query_text` string

`user_id` string > user.id 🔐

`user_name` string > user name

`start_time` timestamp

`end_time` timestamp ❓

### **6. View DDL**

<ButtonLink to="/downloads/view_ddl (7).csv">
  view_ddl (7).csv
</ButtonLink>

Upload the View DDL file even when it has no rows. The file itself is required.

#### View DDL Fields

`database_name` string

`schema_name` string

`view_name` string

`view_definition` string

### **7. User**

<ButtonLink to="/downloads/user.csv">
  user.csv
</ButtonLink>

Upload the User file even when it has no rows. The file itself is required.

#### User Fields

`id` string 🔑

`email` string ❓

`first_name` string ❓

`last_name` string ❓

## Lineage

We compute lineage for your integration by analyzing and parsing the Queries and View DDL when possible.

Alternatively, you can complete the following lineage mapping for Tables and/or Columns and we will ingest them during each update.

### **1. Table Lineage**

<ButtonLink to="/downloads/external_table_lineage.csv">
  external_table_lineage.csv
</ButtonLink>

#### Table Lineage Fields

`parent_path` string 🔑: path of the parent table

`child_path`  string 🔑: path of the child table

### **2. Column Lineage**

<ButtonLink to="/downloads/external_column_lineage.csv">
  external_column_lineage.csv
</ButtonLink>

#### Column Lineage Fields

`parent_path` string 🔑: path of the parent column

`child_path`  string 🔑: path of the child column

<!-- vale on -->

[files]: /docs/catalog/developer/catalog-apis/warehouse-importer#files
[Catalog Uploader]: /docs/catalog/developer/castor-extractor
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[rotation guidance]: /docs/catalog/developer/getting-your-api-keys#rotate-or-revoke-a-token

---

## Dust Integration

Connect the Catalog AI Assistant to Dust so workspace users can ask data questions from Dust chat. You store a Catalog API token as a Dust app secret, build a Dust app that submits and polls the [AI Assistant API][], and attach that app to a Dust agent with a structured instruction prompt.

## Prerequisites

- You need `admin` access in Dust.
- You need a Catalog API token with **Read & Write** scope. Catalog administrators create tokens in **Settings > API**. See [Getting Your Catalog API Keys][] for generation and rotation steps.
- Review [Introduction to Dust apps][] if you are new to Dust app blocks.

## Installation Overview

The Dust agent is created at the workspace level. Once set up, it is accessible to all users in your workspace.

1. Set the Catalog API token as a Dust app secret named `CATALOG_API_TOKEN`.
2. Create and configure the `coalesceAiAssistant` Dust app with the blocks in [Build the Dust App Blocks][].
3. Create a Dust agent that runs the app as a tool and uses the [Agent instruction prompt][] below.

## Set the Catalog API Token as a Dust App Secret

Secrets let Dust apps reference sensitive values without exposing them in block definitions.

<VideoPlayer url="/img/catalog/assets/dust-app-catalog-api-token-secret.mov" subtitle="Create the CATALOG_API_TOKEN secret in Dust Developers settings" />

1. Open the **Admin** tab in the top left menu.
2. Go to **Developers** > **Secrets** in the left menu.
3. Click **Create Secret**.
4. Set the secret name to `CATALOG_API_TOKEN`, paste the token value, and click **Create**.

## Create the Dust App Shell

Create an empty app before you add blocks:

1. Go to the **Spaces** tab in the top left menu.
2. Go to **Open Spaces** > **Apps**.
3. Click **Create a new app**.
4. Set the app name to **coalesceAiAssistant** and the description to **Calling the Catalog AI Assistant to answer your question and show helpful assets**.

The app calls the Catalog public API in two phases: submit a question with `addAiAssistantJob`, then poll `getAiAssistantJobResult` until the job completes. See [AI Assistant API][] for the underlying GraphQL operations.

## Build the Dust App Blocks

Add the following blocks to `coalesceAiAssistant` in this order. Block names are case-sensitive and must match the names referenced in the code.

| Block | Type | Purpose |
| --- | --- | --- |
| `INPUT` | input | Receives `question`, `email`, and `conversationId` from the agent |
| `CONFIG` | code | Sets `MAX_POLLS` and `DELAY_SECONDS` reused by later blocks |
| `GET_JOB_ID` | curl | Submits the question to Catalog and returns a `jobId` |
| `CHECK_JOB_ID` | code | Parses the `GET_JOB_ID` response and surfaces errors |
| `POLLING` | while | Loops until the job reaches a terminal status or hits the poll limit |
| `POLL_API` | curl | Nested inside `POLLING`; polls for the AI Assistant result |
| `SUMMARY` | code | Formats the final `answer` and `assets` for the agent |

Use the API base URL that matches your Catalog region:

- **EU:** `https://api.castordoc.com/public/graphql`
- **US:** `https://api.us.castordoc.com/public/graphql`

Append the operation query parameter to each curl block URL, for example `?op=addAiAssistantJob` or `?op=getAiAssistantJobResult`.

### INPUT Block

1. On the **Specification** tab, click **Add Block** and add an **input** block.
2. Set the block name to `INPUT`.
3. Open **Datasets**, create a dataset named `INPUT`, and define these string fields: `conversationId`, `email`, and `question`.
4. Return to **Specification**, select the `INPUT` block, and set **Dataset** to `INPUT`.

### Configure Polling Constants

1. Add a **code** block and name it `CONFIG` (the block name must match exactly).
2. Paste this code:

```javascript
_fun = (env) => {
  const MAX_POLLS = 15;
  const DELAY_SECONDS = 5;
  return { MAX_POLLS, DELAY_SECONDS }
}
```

Your block should look like this:

### GET_JOB_ID Block

1. Add a **curl** block and name it `GET_JOB_ID`.
2. Set the request to **POST** with your region URL and `?op=addAiAssistantJob`.
3. Paste the **Headers** and **Body** code below into the block.
4. Enable **Results are computed at each run** on the block settings.

**Headers:**

```javascript
_fun = (env) => {
  return {
    "Content-Type": "application/json",
    "Authorization": `Token ${env.secrets.CATALOG_API_TOKEN}`
  };
}
```

**Body:**

```javascript
const _fun = (env) => {
  const { conversationId, email, question } = env.state.INPUT;

  const toGraphQLString = (value) =>
    JSON.stringify(String(value ?? '')).slice(1, -1);

  const body = {
    query: `
      query {
        addAiAssistantJob (
          data: {
            question: "${toGraphQLString(question)}"
            email: "${toGraphQLString(email)}"
            externalConversationId: "${toGraphQLString(conversationId)}"
            origin: DUST
          }
        ) {
          data {
            jobId
          }
        }
      }
    `
  };

  return JSON.stringify(body);
};
```

### CHECK_JOB_ID Block

1. Add a **code** block and name it `CHECK_JOB_ID`.
2. Paste this code:

```javascript
const _fun = (env) => {
  const { errors, data } = env.state.GET_JOB_ID.body;

  if (errors?.length) return { error: errors[0].message, jobId: '' };

  return { error: '', jobId: data?.addAiAssistantJob?.data?.jobId || '' };
};
```

Your block should look like this:

### POLLING and POLL_API Blocks

Add the polling loop and its nested curl block together.

1. Add a **while** block and name it `POLLING`.
2. Set the maximum iteration count to `20` in the block settings.
3. Paste the **condition** code below into the block.
4. Inside the `POLLING` block, add a **curl** block named `POLL_API`.
5. Set the `POLL_API` request to **POST** with your region URL and `?op=getAiAssistantJobResult`.
6. Reuse the same **Headers** code as `GET_JOB_ID`, paste the **Body** code below, and enable **Results are computed at each run** on `POLL_API`.

:::tip[POLL_API Nesting]

Create `POLL_API` inside the `POLLING` block, not at the top level of the app.

:::

**POLLING condition:**

```javascript
const _fun = (env) => {
  const { MAX_POLLS } = env.state.CONFIG;
  const { jobId } = env.state.CHECK_JOB_ID;
  const TERMINAL_STATUSES = ['completed', 'failed'];

  const iteration = env.map?.iteration ?? 0;
  const pollResponses = env.state.POLL_API;

  if (!jobId || iteration > MAX_POLLS) return false;
  if (!pollResponses?.length) return true;

  const lastStatus = pollResponses.at(-1)?.body?.data?.getAiAssistantJobResult?.data?.status;

  return lastStatus
    ? !TERMINAL_STATUSES.includes(lastStatus)
    : true;
};
```

**POLL_API Body:**

```javascript
const _fun = (env) => {
  const { DELAY_SECONDS } = env.state.CONFIG;
  const { jobId } = env.state.CHECK_JOB_ID;

  const body = {
    query: `
      query {
        getAiAssistantJobResult (
          data: {
            id: "${jobId}"
            delaySeconds: ${DELAY_SECONDS}
          }
        ) {
          data {
            status
            answer
            assets {
              id
              internalLink
              name
              url
            }
          }
        }
      }
    `
  };

  return JSON.stringify(body);
};
```

### SUMMARY Block

1. Add a **code** block and name it `SUMMARY`.
2. Paste this code:

```javascript
const _fun = (env) => {
  const { error } = env.state.CHECK_JOB_ID;
  if (error) return { answer: error, assets: [] };

  // Guard: POLL_API may be undefined if polling never ran (e.g. empty jobId)
  if (!env.state.POLL_API?.length) {
    return { answer: 'Unable to retrieve a job ID from the Catalog API. Please try again later.', assets: [] };
  }

  const result = env.state.POLL_API.at(-1)?.body?.data?.getAiAssistantJobResult?.data || {};
  const { status, answer } = result;

  if (status === 'failed') {
    return { answer: answer || 'The AI Assistant job failed. Please try again later.', assets: [] };
  }

  if (status !== 'completed') {
    return { answer: 'The AI Assistant job did not complete in time. Please try again later.', assets: [] };
  }

  const assets = (result.assets || []).map(({ name, url, internalLink }) => ({
    name,
    url: url || internalLink,
  }));

  return { answer: answer || '', assets };
};
```

Your block should look like this:

## Create the Dust Agent

The `CoalesceAiAssistant` agent runs the `coalesceAiAssistant` app as a tool and formats responses for Dust users.

1. Click **Create** > **New Agent**.
2. Set **Advanced settings**: Model selection **Claude Sonnet**, recommended. Creativity level **Deterministic**, recommended.
3. Copy and paste the [Agent instruction prompt][] into the instructions field, then click **Next**.
4. Remove all existing tools, then click **Add tools** > **Run Dust app**.
5. Select the `coalesceAiAssistant` app and click **Save**, then **Next**.
6. Name the agent `CoalesceAiAssistant`, update the icon, test in the bottom-right corner, and set access to **Published**.

### Agent Instruction Prompt

Paste this prompt into the agent instructions field:

```text
You are an assistant that answers user questions by querying a knowledge base tool. Your role is to process questions and return clean, structured responses.

## Process Flow

When a user asks a question:

1. Extract inputs
   - User's question from their message
   - User's email from system context
   - Conversation ID from system context: conversation_id

2. Call the tool with these parameters
   - question: string
   - email: string
   - conversationId: string

3. Process the JSON response

The tool returns this structure:

{
  "answer": "string - the answer to the user's question",
  "assets": [
    {
      "name": "string - display name for the asset",
      "url": "string - link to the asset"
    }
  ]
}

## Response Format

Structure your response exactly as follows:

- Display the answer from the answer field
- List assets when the assets array is not empty:
  - Format each asset as a Markdown link: [asset.name](asset.url)
  - Put each asset link on a separate line

## Error Handling

- If the answer is "NotFound: User was not found", respond: "You need a Coalesce Catalog account to use this assistant. Please contact your organization's admin to request access. They can set up your account with the necessary permissions."
- If the tool fails or returns an error: "I'm unable to process your request right now. Please try again later."

## Important Rules

- No additional commentary. Only include the answer and asset links.
- No explanations beyond what the tool provides.
- Always validate that assets have both name and url fields before formatting.
- Handle empty responses gracefully. Display the answer even when there are no assets.
- Maintain consistent formatting for all asset links.
```

## Debug

- **Tool inspection in the chat:** Click **Tool inspection** in your chat box to view both the input and output of the tool.
- **Tool logs:** Each response from `CoalesceAiAssistant` triggers a tool call to `coalesceAiAssistant`. View a detailed log of every call, including step-by-step output, by going to the app **Logs** > **API**.

## Update

- The Dust agent uses the AI Assistant public API, so updates to the AI Assistant on the Catalog side are reflected in Dust automatically. Updates on the Catalog side are designed to be backward compatible. Catalog notifies you if an update requires action on your part.
- Because the Dust app and agent live in your Dust workspace, you are responsible for updating them when needed. Catalog notifies you if an update is required.

## What's Next?

- [AI Assistant API][]
- [MCP Integration][]
- [Getting Your Catalog API Keys][]

[AI Assistant API]: /docs/catalog/developer/ai-assistant-api
[MCP Integration]: /docs/catalog/developer/mcp-integration
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Introduction to Dust apps]: https://docs.dust.tt/reference/introduction-to-dust-apps
[Build the Dust App Blocks]: #build-the-dust-app-blocks
[Agent instruction prompt]: #agent-instruction-prompt

---

## Getting Your Catalog API Keys

Catalog **API tokens** (also called API keys) authenticate programmatic access to Coalesce Catalog. You need one for the [Catalog Public API][], [MCP Integration][], metadata importers, the in-app API playground, [Castor Extractor][] uploads, and custom automation.

This page covers how to create a token in the app and how to manage expiration, rotation, and revocation. Integration-specific setup steps stay on each guide; return here when you need to rotate credentials or fix `401 Unauthorized` errors.

## Generate a Catalog API Token

Catalog administrators generate tokens in the Catalog app. You need admin access to **Settings > API**.

1. Sign in to Coalesce Catalog.
2. Open **Settings > API**.
3. Click **Create Token**.
4. Enter an optional **Name** (for example `Production MCP` or `BI importer`).
5. Choose a **Scope**:
   - **Read Only** for search, lineage reads, and MCP use cases that do not write metadata.
   - **Read & Write** for metadata importers, uploads, automation that creates or updates catalog assets, and AI Assistant integrations (including the [Dust Integration][]).
6. If a token already exists for that scope, read the warning, acknowledge it, and confirm **Generate new token**.
7. Copy the token from the success dialog immediately. For security, Catalog **does not show the full token again** after you close that dialog.

<figure><figcaption></figcaption></figure>

Store the token in a secrets manager or your team's credential store. Never commit it to Git or paste it into public channels.

## Token Scopes and Limits

Each Catalog account can have **one active token per scope**:

- One active **Read Only** token
- One active **Read & Write** token

You can use both at the same time when different integrations need different permissions. For example, point MCP at a Read Only token and keep Read & Write for `castor-upload` or importer scripts.

The token list shows each active token's **masked value**, **scope**, **name**, and **expiration date**.

## Where to Use Your Token

Pass the token in the `Authorization` header (or the auth field your client expects) when calling Catalog APIs or configuring tools:

| Use case | Documentation |
| --- | --- |
| GraphQL Public API and playground | [Catalog Public API][] |
| Claude, Cursor, Dust, and other MCP clients | [MCP Integration][] |
| BI, warehouse, and manual lineage importers | [Catalog APIs][] |
| Client-managed extraction uploads | [Castor Extractor][] |
| AI Assistant polling via API | [AI Assistant API][] |
| Dust agent tool | [Dust Integration][] |

The AI Assistant API and Dust integration require **Read & Write** scope. MCP and read-only GraphQL queries can use **Read Only**.

Use the API base URL that matches your region:

- **EU:** `https://api.castordoc.com`
- **US:** `https://api.us.castordoc.com`

## Expiration and Email Notifications

New Catalog API tokens are valid for **one year** from creation.

Catalog emails **account administrators** about token status:

- **30, 7, and 1 days** before a token expires
- When a token **expired within the last 24 hours**
- When a token is **generated** or **revoked**

Expiring emails include a link to **Settings > API** and remind you that generating a replacement token **revokes the current token for that scope** immediately.

## Rotate or Revoke a Token

### Rotate Before Expiration

Plan rotation when you receive an expiration notice or as part of your security cadence.

1. Inventory every integration that uses the token for that scope (MCP client configuration, extractor jobs, CI secrets, importer scripts, playground, and partner tools).
2. In **Settings > API**, create a token with the **same scope**.
3. Copy the new token from the one-time dialog.
4. Update each integration with the new value.
5. Verify API calls and MCP connections succeed.

Generating the new token **revokes the previous active token for that scope right away**. API and MCP requests fail with **`401 Unauthorized`** until every consumer uses the new secret.

### Revoke Without Replacing

To remove access immediately, open **Settings > API**, find the token row, and click **Revoke**. Administrators receive an email notification. Revoked tokens cannot be restored; create a new token if you still need API access.

### If You Lost the Token Value

Catalog cannot display a token again after the creation dialog closes. Generate a new token for that scope and update your integrations. That action revokes the lost token.

## SCIM Tokens Are Separate

[SCIM provisioning][] uses a **different token** from the Catalog Public API token. Rotating a Public API token in **Settings > API** does not update SCIM credentials in Okta or Microsoft Entra ID. Contact [Coalesce Support][] for SCIM token rotation.

## Troubleshooting

### `401 Unauthorized` from the API or MCP

- Confirm the token is not expired on the **Settings > API** list.
- Confirm the integration uses the **correct scope** (Read Only vs Read & Write).
- Confirm nobody rotated or revoked the token without updating your config.
- For MCP, update the `Authorization` value in your client config and restart the app.

### Integrations Broke After Someone Created a Token

Creating a token for a scope replaces the prior token for that scope. Coordinate with your team before rotating shared credentials, or use separate scopes when Read Only access is enough for some tools.

### Expiration Email but Integrations Still Work

The email refers to the token that will expire on the listed date. Rotate before that date to avoid a hard cutoff when the token expires.

## What's Next?

- [Catalog Public API][]
- [MCP Integration][]
- [Castor Extractor][]

[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[MCP Integration]: /docs/catalog/developer/mcp-integration
[Castor Extractor]: /docs/catalog/developer/castor-extractor
[Catalog APIs]: /docs/catalog/developer/catalog-apis
[AI Assistant API]: /docs/catalog/developer/ai-assistant-api
[Dust Integration]: /docs/catalog/developer/dust-integration
[SCIM provisioning]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-okta
[Coalesce Support]: /docs/catalog/support/how-to-reach-us

---

## GitHub Actions for dbt Sync

Use [GitHub Actions][] to synchronize your dbt manifest file with GCS and keep your data catalog up to date.

## Create the Action

You need to create a folder in your dbt repository with the path `<repository>/.github/workflows/`

Inside you need to create a YAML file, we will name ours `send_to_catalog.yml`

The content is :

```bash
name: Send manifest to Catalog
on: [push]
jobs:
  send-manifest-to-catalog:
    runs-on: ubuntu-latest
    steps:
      - name: Imports GitHub code source
        uses: actions/checkout@v2 
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install castor-extractor
      - name: Launch python script
        run: castor-upload -c '${{ secrets.SERVICE_ACCOUNT_JSON }}' -s <source_id> -f dbt_manifest.json -t DBT
```

## Create the Secrets

There's a [reference guide](https://docs.github.com/en/actions/reference/encrypted-secrets) in order to understand the manipulation of secrets using GitHub Actions.

Here is a screenshot of the [minified](https://www.cleancss.com/json-minify/) version of the Service Account JSON content stored and passed to the Python function.

## Push the Code

Once that all the code is pushed to the repository, the workflow will be triggered. Then you will see something like this in the `Actions` tab of the repository.

The date and time of the file will be in UTC.

[GitHub Actions]: https://github.com/features/actions

---

## Developer Guide

This section is for engineers who automate Catalog with APIs, the extractor CLI, CI pipelines, or agent integrations.

Start with [Getting Your Catalog API Keys][] when you need a Catalog API token for the Public API, MCP, importers, or extractor uploads.

## Catalog APIs

Use [Catalog APIs][] when you push metadata over HTTP: the [Public API][], [BI Importer][], [Warehouse Importer][], and [Manual Lineage Importer][].

## Castor Extractor

Use [Castor Extractor][] when an integration is **client managed**. Install `castor-extractor` on your infrastructure, run `castor-extract-*` commands, optionally validate warehouse CSV files with `castor-file-check`, then upload with `castor-upload`.

## Integrations and Automation

- [GitHub Actions and dbt][]
- [AI Assistant API][]
- [MCP Integration][]
- [Dust Integration][]

Catalog-managed integrations are configured in **Settings > Integrations** and do not require the extractor. See [Quick Set Up][] for managed versus client-managed onboarding.

## What's Next?

- [Catalog APIs][]
- [Castor Extractor][]
- [Quick Set Up][]

[Catalog APIs]: /docs/catalog/developer/catalog-apis
[Public API]: /docs/catalog/developer/catalog-apis/public-api
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[Warehouse Importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Manual Lineage Importer]: /docs/catalog/developer/catalog-apis/manual-lineage-importer
[Castor Extractor]: /docs/catalog/developer/castor-extractor
[GitHub Actions and dbt]: /docs/catalog/developer/github-actions-dbt
[AI Assistant API]: /docs/catalog/developer/ai-assistant-api
[MCP Integration]: /docs/catalog/developer/mcp-integration
[Dust Integration]: /docs/catalog/developer/dust-integration
[Quick Set Up]: /docs/catalog/quick-set-up
[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys

---

## MCP Integration

Connect your AI assistant to Coalesce Catalog through the Model Context Protocol (MCP). Your assistant can search catalog metadata, traverse lineage, and answer data questions in natural language from Claude Desktop, Cursor, Dust, or any MCP-compatible client.

For the hosted Catalog AI Assistant polling API instead of MCP tools, see [AI Assistant API][].

## What Is MCP?

The Model Context Protocol is an open standard that lets AI assistants connect to external data sources and tools. With Coalesce MCP, your assistant reads the same catalog search and metadata that powers Coalesce Catalog, without switching out of your chat or IDE.

Ask your assistant a question such as *"Find all customer tables owned by the `{your-team-name}` team"* and get results grounded in your catalog.

## Before You Begin

You need:

1. A **Coalesce Catalog** account with API access.
2. A **Catalog API token** from **Settings > API**. See [Getting Your Catalog API Keys][] for generation and rotation steps. Use **Read Only** scope when you do not need write access. Treat the token as a secret.
3. An **MCP-compatible client**. This page covers Claude Desktop, Cursor, and Dust.

## API Endpoints

Use the MCP server URL that matches your Catalog region:

- **EU:** `https://api.castordoc.com/mcp/server`
- **US:** `https://api.us.castordoc.com/mcp/server`

## Set Up Claude Desktop

Claude Desktop does not support quick MCP setup for API token-based authentication yet, so you edit the configuration file manually.

### 1. Create or Edit Your Configuration File

Choose either direct file or using the MCP server.

#### Direct File Access

- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
- **Linux:** `~/.config/Claude/claude_desktop_config.json`

#### Through Claude Desktop Settings

1. Navigate to **Claude Settings > Developer**.
2. Click **Edit Config**.

### 2. Add the Coalesce MCP Server

Add a `coalesce-catalog` entry to `mcpServers` using the endpoint from **API Endpoints** that matches your region. Replace `YOUR_API_TOKEN_HERE` with your Catalog API token.

```json title="EU endpoint configuration"
{
  "mcpServers": {
    "coalesce-catalog": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://api.castordoc.com/mcp/server",
        "--header",
        "Authorization: YOUR_API_TOKEN_HERE"
      ]
    }
  }
}
```

```json title="US endpoint configuration"
{
  "mcpServers": {
    "coalesce-catalog": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://api.us.castordoc.com/mcp/server",
        "--header",
        "Authorization: YOUR_API_TOKEN_HERE"
      ]
    }
  }
}
```

### 3. Restart Claude Desktop

Quit and reopen Claude Desktop so it loads the updated configuration.

### 4. Test the Connection

Ask Claude: *"Can you search for `{your-data-topic}` tables in Coalesce?"*

## Set Up Cursor

Add the Coalesce MCP server by editing your local MCP configuration file.

### 1. Open Cursor Settings

In Cursor, go to **Tools & MCP** > **New MCP server**.

### 2. Configure the MCP Server

Create or edit `~/.cursor/mcp.json`. Add a `Catalog` entry using the endpoint from **API Endpoints** that matches your region. Replace `YOUR_API_TOKEN_HERE` with your Catalog API token.

```json title="EU endpoint configuration"
{
  "mcpServers": {
    "Catalog": {
      "url": "https://api.castordoc.com/mcp/server",
      "headers": {
        "Authorization": "YOUR_API_TOKEN_HERE"
      }
    }
  }
}
```

```json title="US endpoint configuration"
{
  "mcpServers": {
    "Catalog": {
      "url": "https://api.us.castordoc.com/mcp/server",
      "headers": {
        "Authorization": "YOUR_API_TOKEN_HERE"
      }
    }
  }
}
```

### 3. Restart Cursor

Restart Cursor so it picks up the new MCP configuration.

### 4. Verify in Cursor Agent

Open Cursor Agent and ask: *"Find tables with `{your-data-topic}` data in Coalesce"*

## Set Up Dust

Connect Catalog MCP from your Dust workspace tools settings. For a custom Dust app that polls the Catalog AI Assistant API instead, see [Dust Integration][].

### 1. Access Dust Workspace Settings

1. Navigate to your Dust workspace.
2. Go to **Spaces** > **Tools** > **Add Tools**.

### 2. Add a New MCP Connection

1. Click **Add Tools** > **Add MCP Server**.
2. Set **URL** to the EU or US endpoint from **API Endpoints**.
3. For **Authentication**, type **Bearer Token**, then set Bearer Token to `YOUR_API_TOKEN_HERE`.

### 3. Save the Connection

Change name and tool settings if needed, then save changes.

### 4. Add the MCP Server to a Dust Agent

1. Go to **Chat**.
2. Create or select an existing **Agent** > **Edit Agent**.
3. Go to **Knowledge & Tools** > **Add Tools**.
4. Add the Catalog MCP.
5. Click **Test & Save**.

In any Dust conversation, your agent can now query Catalog. For example: *"@your-agent Search Catalog for `{your-data-topic}` tables tagged as `{your-tag}`"*

## Authorizations

The MCP server uses Public API Token authentication. OAuth is not yet supported.

The server does not enforce role-based access control inside Catalog. It returns every resource your API token can access through the Public API, including search results, lineage, tags, teams, and users.

## Available Tools

The Coalesce MCP server exposes 10 tools for catalog search and metadata reads, grouped below.

### Search Tools

These tools use the same search engine as Coalesce Catalog AI search. Each call returns at most 30 results, even when a higher `limit` is requested.

| Tool | Purpose | Key Inputs | Result Limit |
| --- | --- | --- | --- |
| `search_tables` | Find data warehouse tables | `query` required, `name`, `ownerName`, `tag`, `limit` | Default 10, capped at 30 |
| `search_dashboards` | Find BI dashboards | `query` required, `name`, `ownerName`, `tag`, `limit` | Default 10, capped at 30 |
| `search_columns` | Find specific columns | `query` required, `name`, `tag`, `limit` | Default 20, capped at 30 |
| `search_terms` | Search business glossary | `query` required, `name`, `ownerName`, `tag`, `limit` | Default 10, capped at 30 |

### Catalog API Tools

These tools cover lineage traversal and other catalog metadata exposed through MCP. The lineage tools, `get_asset_lineage` and `get_field_lineage`, walk upstream and downstream relationships from a starting asset or field using the same lineage graph that powers Coalesce Catalog. Each call returns up to 100 related nodes per direction, and `depth` is capped at 10 levels.

| Tool | Purpose | Inputs | Result Limit |
| --- | --- | --- | --- |
| `get_asset_lineage` | Upstream or downstream lineage for a data asset such as a table or dashboard | `id` required, `asset_type` required, `depth` optional, `direction` optional | `depth` defaults to 1, capped at 10. `direction` defaults to `BOTH`. Up to 100 nodes per direction, up to 200 total with `BOTH` |
| `get_field_lineage` | Field-level lineage for a specific column or dashboard field | `id` required, `field_type` required, `depth` optional, `direction` optional | `depth` defaults to 1, capped at 10. `direction` defaults to `BOTH`. Up to 100 nodes per direction, up to 200 total with `BOTH` |
| `get_company_context` | Retrieve organizational context for the authenticated account | None | Returns a single company context object |
| `get_tags` | Retrieve tags used to categorize data assets | `ids` optional, `labelContains` optional | Returns all matching tags |
| `get_users` | Get all users and their profiles | None | Returns all users in the account |
| `get_teams` | Get all teams and configurations | None | Returns all teams in the account |

## Troubleshooting

### Connection Issues

If you see **Failed to connect to MCP server**, check the following:

1. Verify the API token is valid and not expired. See [Getting Your Catalog API Keys][] for rotation steps if the token was revoked or replaced.
2. Confirm the endpoint URL matches your Catalog region, EU or US.
3. Test network connectivity to your Catalog API host.

### AI Not Using Tools

If your assistant answers from general knowledge instead of calling MCP tools:

- Be explicit in the prompt, for example *"Search Coalesce Catalog MCP for…"* using the server name from your config.
- Ask about specific asset types: tables, columns, dashboards, or glossary terms.
- Mention owners, teams, or tags by name when you know them.

## FAQ

### Does This Work With All AI Assistants?

Any MCP-compatible client can connect with custom configuration. This page documents Claude Desktop, Cursor, and Dust.

### What Data Does the AI Assistant See?

The assistant can search and read catalog metadata: table and column names, descriptions, owners, tags, lineage, and glossary terms available to your API token.

### What Happens If the Token Expires?

MCP calls return `401 Unauthorized`. Generate a replacement in **Settings > API**, update the `Authorization` value in your MCP client config, and restart the client. See [Getting Your Catalog API Keys][] for expiration notices and rotation impact on other integrations.

## Resources

- [Model Context Protocol][]
- [Getting Your Catalog API Keys][]
- [Catalog Public API documentation][]

## What's Next?

- [Getting Your Catalog API Keys][]
- [AI Assistant API][]
- [Dust Integration][]

[Getting Your Catalog API Keys]: /docs/catalog/developer/getting-your-api-keys
[Catalog Public API documentation]: /docs/catalog/developer/catalog-apis/public-api
[Model Context Protocol]: https://modelcontextprotocol.io/
[AI Assistant API]: /docs/catalog/developer/ai-assistant-api
[Dust Integration]: /docs/catalog/developer/dust-integration

---

## Catalog Auto Doc

The Catalog automatically documents your data by retrieving descriptions from well-known sources and adding them to your tables and columns.

You may be asking yourself "but what is a documentation repository?"

## What Is a Documentation Repository?

Among columns and tables that you have in your warehouse, there are some that come from well-known sources (Google Ads, Intercom, HubSpot, etc.).

What we did for you:

1. Search online for the data documentation of these tools
2. Sort out the descriptions for the tables and columns
3. Look inside your tables and columns to figure out what you are using, and automatically add the description for you

## Some Numbers

### Sources

Doc Master is now able to retrieve information from 38 **different sources**. See below for some examples.

If there are some sources that you are missing, feel free to contact us. We will be always adding new sources to Doc Master.

## Full Source List

The following source identifiers are supported (these are technical identifiers used in the integration):

* `ad_reporting`
* `asana`
* `facebook_ads`
* `facebook_ads_creative_history`
* `fivetran_log`
* `github`
* `go_cardless`
* `googl_ads`
* `googl_ads_api`
* `hubspot`
* `intercom`
* `iterable`
* `jira`
* `klaviyo`
* `linkedin_ads`
* `linkedin_hiring`
* `linkedin_marketing`
* `linkedin_sales`
* `marketo`
* `microsoft_ads`
* `netsuite`
* `pardot`
* `pendo`
* `pinterest`
* `quickbooks`
* `salesforce`
* `salesforce-marketing-cloud`
* `shopify`
* `snapchat_ads`
* `stripe`
* `twilio`
* `twitter`
* `wootric`
* `xero`
* `zendesk`

---

## Column Description Propagation

It's common for a column to be reused "as is" by downstream tables. In that case, the column will be called in the "Select" SQL clause, without any modification.

In that case, why should you document the downstream columns? You're right, you should not have to. Catalog solves that for you.

In the image below, `user.account_id` inherits its description from its parent column in the `opportunity` table.

## Requirements

There are just 3 simple requirements for this feature to be activated:

1. Parent and child column need to have the same exact name
2. Child column is generated directly from parent without any SQL transformations
3. Child column has one and only one parent

:::warning[Dashboard Fields Not Supported]

Dashboard fields are out of scope for this feature. Descriptions cannot propagate from table columns to dashboard fields.

:::

## Troubleshooting

Use this table to diagnose why descriptions might not propagate.

| Symptom | Likely cause | Action |
| :------ | :----------- | :----- |
| Propagation not working | One or more requirements not met | Verify: (1) parent and child column have the exact same name, (2) child column has no SQL transforms, (3) child column has exactly one parent. |
| No lineage visible | Lineage not computed or not available | See [Lineage Troubleshooting][] for common causes. |
| Descriptions don't propagate to dashboard fields | Feature limitation | Dashboard fields are not supported. Use the [Metadata Editor][] to add descriptions manually. |

### When Propagation Doesn't Apply

Propagation does not apply when:

* **Columns have transforms**—The child column is derived with SQL (for example, `CAST`, `CONCAT`, or expressions).
* **Multiple parents**—The child column is built from more than one upstream column.
* **Different names**—Parent and child column names differ.

In these cases, use the [Metadata Editor][] with the **Ancestors** filter to select downstream tables and bulk-apply descriptions.

[Lineage Troubleshooting]: https://docs.coalesce.io/docs/catalog/navigate/lineage/lineage-troubleshooting
[Metadata Editor]: https://docs.coalesce.io/docs/catalog/document-your-data/metadata-editor

---

## Describe With AI

The AI-Powered Auto-Description feature helps contributors document dashboards, tables, and Knowledge pages using all available metadata.

## Overview

Previously, AI-generated suggestions for asset documentation were only available if a SQL source query was present. With this feature, the AI leverages **all available metadata**—not just SQL queries—to generate rich, accurate, and context-aware descriptions for dashboards and tables.

## Why This Matters

* **Broader Coverage:** Previously, only assets with a SQL source query could benefit from AI suggestions. Now, assets **without SQL queries**—such as most dashboards and many tables—can also be auto-documented using their metadata.
* **Efficiency:** Contributors can document assets quickly and completely, without needing to write content manually or rely on external tools.
* **Scalability:** This unlocks auto-descriptions for up to **40x more dashboards** (from 8,000 to over 309,000), plus similar gains for tiles and viz models.

## How It Works

### 1. Metadata-Driven AI Suggestions

When you ask the AI to suggest a description for a dashboard or table:

* The AI gathers **all relevant metadata** associated with the asset, including:
  * Dashboard/Table name, path, and type
  * Technology (for example, Power BI, Looker)
  * Tags and labels
  * Editor and owner information
  * IsVerified / IsDeprecated status
  * Folder path and source system
  * Descriptions (internal and external)
  * All source SQL queries (if any)
  * Parent and child asset relationships
  * Column/field names, types, and descriptions
  * Frequently used users, pinned assets, mentioned assets
  * Additional context such as report page names, business objects, and data update frequency

### 2. No SQL? No Problem

* If an asset has no SQL source query, the AI uses all other available metadata to generate a high-quality description.
* For dashboards, the AI leverages the same context used in the Dashboard Q&A feature (including information from tiles, not just the dashboard itself).
* For tables, **all SQL queries** associated with the table are used (not just the primary one), along with rich column and relationship metadata.

### 3. User Experience

* **AI Suggestion Button:** When documenting an asset, click the AI button to generate a suggested description.
* **Multiple Queries:** If multiple queries are available, the AI analyzes them all automatically—no need for a modal asking you to choose.
* **Preview and Edit:** Review, accept, or edit the suggested description before saving.

### 4. Leverage Company Description

When documenting a Knowledge page, you can leverage AI and the company description your admin provided. Use the toggle to decide if the AI should leverage it to generate the description or not.

Admins can edit the description following this documentation: [AI Administration](/docs/catalog/administration/ai-administration.md)

<figure><figcaption></figcaption></figure>

## Example Use Case

Suppose you have a Power BI dashboard called "GHG Monitoring Dashboard - PROD" with no SQL source query. When you click the AI button, the system will:

* Use the dashboard's metadata: name, description, folder path, tags (for example, "domain:finance"), verification status, source system ("Power BI Corp"), and more.
* Include information about editors, frequent users, owners, and related business objects or pinned assets.
* Reference parent tables and fields, metrics definitions, update frequencies, and other context.
* The AI will then generate a comprehensive, readable description, summarizing the dashboard's purpose, content, data sources, and key metrics.

## Supported Metadata (Not Exhaustive)

* **Dashboards:** Name, folder path, descriptions, tags, technology, type, source, editors, frequent users, pinned assets, owners, parent/child assets, page names, etc.
* **Tables:** Name, database/schema, technology, columns (names, types, user/external descriptions), joins, all source SQL queries, parent/source tables, verification/deprecation status, editors, owners, tags, pinned/mentioned assets, etc.
* **Knowledge Pages:** path, parent page, tags, company description

---

## Description Auto-Complete

Use Catalog to reuse column descriptions across tables. When you see `See suggestions from other tables` in a column description, that column is described in another table and you can add it here.

## Column Description Suggestion

Whenever you see `See suggestions from other tables` in a column description, that column is described in another table and you can add it here.

---

## Catalog Scribe ✨

## The Magic

In spite of our best effort to create a beautiful app, writing documentation just isn't fun 😢. Catalog AI is here to support you 🦸‍♂️

## Table and Dashboard Readmes

Catalog AI can generate for you a detailed README describing the content and use cases for tables and dashboards that have source SQL queries.

## Table's Column Descriptions

Leverage AI to document your columns as well. On any table you just have to click on this button

<figure><figcaption></figcaption></figure>

It will generate the documentation for first thirty columns at a time. For each documented column you'll see "From AI Assistant" so you can easily identify them.

<figure><figcaption></figcaption></figure>

Catalog also completes your column documentation with the following features:

<ButtonLink to="./catalog-scribe/castor-auto-doc">
  Catalog Auto Doc
</ButtonLink>

<ButtonLink to="./catalog-scribe/column-description-propagation">
  Column Description Propagation ✨
</ButtonLink>

<ButtonLink to="./catalog-scribe/description-auto-complete">
  Description Auto-Complete
</ButtonLink>

## Knowledge Pages

Catalog AI can also come to your help to create knowledge pages. The scribe is tailored for different types of documentation:

* Metrics
* Data entities
* Data domains
* Acronyms

<YouTube id="baqG4Z86Lb0" />

You simply need to specify the page name and type for the AI to do its' magic.

If you want none of the specified types correspond to your needs, you can always chose the type "Other" 😉.

## Safety

The Scribe uses OpenAI models. The feature requires admin approval (before then, no metadata will be shared with OpenAI). Once the feature is activated, only queries for which a user requests an explanation are shared with OpenAI. It is worth noting that we filter out PII information from queries. For more details, please view the Catalog AI [safety page](/docs/catalog/support/catalog-ai-safety).

## Activate Catalog AI on Your Account

To see this live in your Catalog account, an admin just needs to reach out to [support@coalesce.io](mailto:support@coalesce.io)

---

## Knowledge Map

Catalog's Knowledge Map provides a visual representation of your organization's knowledge pages. With Knowledge Map, you can navigate and explore the hierarchy of your organization's data, spanning domains, metrics, and documented KPIs.

<figure><figcaption></figcaption></figure>

## The Significance of Knowledge Map

Effective data management and organization are paramount for any organization's success. The ability to understand and communicate how data is structured within your organization is critical. Knowledge Map bridges these gaps by providing teams with the tools to comprehend the context around their data and establish a unified company language regarding data domains, metrics, and KPIs.

## A Business-Centric Approach

Knowledge Map invites you to explore your data ecosystem from a business perspective. Move beyond technical hierarchies and dive into your data with a business lens. Start by understanding the business logic, follow the business hierarchy, and then delve into the data assets that best represent these business metrics.

### The Bird's Eye View

One of the standout features of Knowledge Map is its ability to offer a high-level view of key knowledge pages. Instead of searching through a sea of knowledge pages, Knowledge Map presents this information in a relational diagram. This diagram clearly and concisely illustrates how each knowledge page is related, complete with descriptions, owners, and tags. You can grasp the importance of knowledge pages and quickly drill down into specific topics, such as Metrics, Churn Rate, MMR, or Active Users.

Knowledge Map is more than a diagram. Its strength lies in connecting data assets to knowledge pages, serving as a gateway for business professionals to explore the data warehouse. This transformation makes knowledge pages accessible and familiar, enabling you to step into the realm of data using concepts you are accustomed to.

## Understanding From Every Angle

Knowledge Map offers a multi-dimensional understanding of knowledge by combining hierarchy and tags. You can see how different pieces of knowledge relate to each other, providing a holistic perspective. For example, Knowledge Map can help you understand the various types of revenues and their relationships.

It facilitates a clearer understanding of ownership of key performance indicators and encourages collaboration among teams. By identifying overlaps and redundancies in responsibilities, it optimizes efficiency and effectiveness within the organization.

Understanding these interconnections within your knowledge is vital for data-driven decision-making and helps everyone in your organization grasp how knowledge is interconnected.

## Speedy Knowledge Navigation

In the world of data accessibility, speed is crucial. With Knowledge Map, teams can navigate much faster than traditional data catalogs. Instead of sifting through lists of folders documenting company knowledge, you can quickly find what you need on the map in seconds. This ensures that company knowledge is easily discoverable and readily available when teams need it most.

---

## Lineage To Service Accounts

It is key to have a bigger picture than just tables and dashboards, since your data flows to various places to fulfill different purposes. Bringing clarity to where some data is used will help your user in both ways. Going upstream, one can identify which tables are used by your data apps. Additionally, when manipulating tables of a warehouse, it is essential to be aware some table are key resources to data apps (on top of other tables or dashboards).

Catalog detects service accounts reading into your warehouses and shows them in the app. Each service account will be featured as a table, with lineage to the warehouse tables it reads from.

<figure><figcaption>CastorDoc represents Streamlit, Census, and Hightouch reading into a Snowflake table</figcaption></figure>

Reach out on Slack or to [support@coalesce.io](mailto:support@coalesce.io) to activate this Beta feature.

---

## Metadata Editor

The Metadata Editor lets you make bulk changes to owners, tags, descriptions, and more across your Catalog assets. Admins can perform bulk metadata edits. When your account uses **custom ownership labels**, you can also use **Replace owner** to apply labels to existing assignments without changing the principal.

## For Tables, Dashboards and Knowledge

Actions you can perform in bulk on your assets are:

1. **Owners** - Add owners, use **Replace owner** to update the assignment or assign a **custom ownership label** while keeping the same principal, or remove owners
2. Add tags
3. Certify or uncertify
4. Deprecate or undeprecate

It's a 3-step process:

### Step 1: Select the Tables You Want to Modify

Use the enhanced filtering system with intelligent cascading filters and lineage-based options.

#### Basic Filters

These filters narrow assets by location, tags, and owner:

- **Source, Database, Schema**: Hierarchical selection where database options depend on selected sources, and schema options depend on selected databases
- **Tags**: Filter by existing tags
- **Owner**: Filter by asset owners

#### Advanced Lineage Filters

These filters use lineage to find upstream relationships:

- **Ancestors**: Filter tables by their upstream dependencies using lineage relationships
  - Supports flexible search formats: `database.schema.table` or `database schema table`
  - Case-insensitive search
  - Considers all lineage types: `AUTOMATIC`, `MANUAL_CUSTOMER`, `MANUAL_OPS`, and `OTHER_TECHNOS`

#### Smart Cascading Selection

The **Source > Database > Schema** filters work hierarchically to prevent invalid combinations. When you select a source, only databases from that source appear. When you select databases, only schemas from those databases appear.

#### Lineage-Based Filtering

Use the **Ancestors** filter to find all tables that depend on a specific upstream table. This is perfect for propagating ownership, tags, or other metadata changes downstream through your data lineage.

<figure></figure>

### Step 2: Click on the Action You Want to Perform

After you select assets, use the toolbar to pick the bulk action, such as **Owners**, **Tags**, **Certification**, or **Deprecation**.

<figure></figure>

### Step 3: Apply Your Changes

Complete the modal or panel for your chosen action, for example picking owners and confirming **Assign owners**.

<figure></figure>

Your assets are immediately updated.

### Review Changes on Each Asset

Bulk updates apply to every asset in your selection or filter scope. To confirm what changed, open the **History** tab on individual tables, dashboards, or knowledge pages. **History** lists owners, tags, descriptions, certification, deprecation, and other metadata edits with author and timestamp. See [History][] for the full list of logged change types and governance workflows.

## Assign Custom Ownership Labels in Bulk

When your account uses **custom ownership labels**, existing owner assignments keep the **No label** status until you update them, even after admins create labels under **Account Configuration > Ownership Labels**. The steps here apply the right label while keeping the same owner or team on each asset.

:::warning[No label until you migrate]

If you enable labels but do not update existing rows, owners continue to show as **No label** anywhere Catalog groups or displays ownership by label.

:::

### Before You Begin

Admins create and manage label names under **Account Configuration > Ownership Labels**. You need at least one label before you can assign it from the Metadata Editor.

### How to Apply Labels With Replace Owner

Use **Replace owner** when the accountable person or team stays the same and you only need to set or change the label.

1. Open the **Metadata Editor** for tables, dashboards, or knowledge you plan to update.
2. Use **Owner** filters so the grid lists the assets for one owner, or another scope you intend to change.
3. Select the assets that should receive the same label for that owner.
4. Choose **Replace owner**.
5. Select the **same** owner or team, pick the **ownership label** you need, and confirm.
6. Repeat for other owners or filter scopes until **No label** no longer appears where you expect labeled ownership.

For large result sets, you can use **Select all matching assets** and **Apply All** with the same care you use for other bulk actions. Read [When There Are More Results Than the List Shows][] first so your filters match the assets you intend to change.

After migration, owners appear grouped by label wherever Catalog shows ownership that way. For **new** assignments when labels are enabled, choose a label that matches the relationship you are recording. For what ownership means in Catalog, see [Ownership][].

<VideoPlayer url="/img/catalog/assets/label-setup.mov" subtitle="See how to make custom labels and bulk update" />

## Advanced Use Cases

These patterns help when you need the same metadata change across many related assets.

### Propagating Governance Downstream

The **Ancestors** filter is particularly powerful for data governance workflows.

#### Same Owner on Downstream Tables

You have a core customer table `prod.core.customers` and want to assign the same owner to all downstream tables that depend on it.

1. In the **Metadata Editor**, use the **Ancestors** filter
2. Search for `prod.core.customers` or `prod core customers`
3. All tables with lineage relationships to this table will be displayed
4. Select the tables you want to update
5. Apply ownership changes in bulk

This eliminates the need to manually navigate through lineage tabs for each dependent table.

### Multi-Source Environment Management

The **hierarchical filtering** system makes it easy to work across complex data environments.

#### Tags Across Multiple Snowflake Schemas

You need to tag all tables in specific schemas across multiple Snowflake databases.

1. Select your Snowflake source first
2. Choose the relevant databases. Only Snowflake databases from that source appear.
3. Select the specific schemas. Only schemas from the chosen databases appear.
4. Apply tags to all selected tables

This prevents accidentally selecting tables from the wrong source or invalid database and schema combinations.

## Columns

Actions you can perform in bulk on your assets are:

1. Add, replace, or remove descriptions
2. Add, replace, or remove tags
3. Flag as PII
4. Flag as Primary Key

Columns that share a name often need a shared description. The following video walks through that workflow.

<Loom id="51e002591585417fa5e6b99c3a101d5b" />

When you run **Apply All** on columns, you use the same **Select all matching assets** flow and limits as for other asset types. Before a large run, read [When There Are More Results Than the List Shows][] for how paging works, how filters set scope, and the maximum number of assets you can include in one operation.

### Bulk Descriptions Workflow

Use the **Ancestors** filter to select downstream tables that depend on a source table, then add or replace descriptions in bulk. This works well when [column description propagation][] doesn't apply, for example columns with transforms or multiple parents.

For initial import of descriptions from a CSV, see [Upload existing descriptions][]. Use the Metadata Editor for ongoing bulk updates after import.

## When There Are More Results Than the List Shows

The Metadata Editor loads a limited number of matching assets at a time, for example the first **500**. When your filters match more assets than appear in the grid, you can still update every matching asset in one operation.

Use **Select all matching assets** to start **Apply All**. Coalesce runs your metadata action against the full filtered set in the **background**, not only the rows you see on the page. While the job runs, watch the Metadata Editor header for progress and status. You can keep working, or check back when the job finishes.

Before you start **Apply All**, remember that filters set the scope for the whole run. Whatever filters are active when you start **Apply All** apply to every matching asset, not only the rows on the page. Narrow or widen filters first so the scope matches what you intend.

### What's Different From Selecting Rows on the Page?

If you are choosing between selecting on the page and **Select all matching assets**, compare these two behaviors:

- Selecting checkboxes on the loaded page updates **only** those visible or selected assets.
- **Select all matching assets** targets **all** assets that match the filter, including those not yet loaded in the grid.

---

[When There Are More Results Than the List Shows]: /docs/catalog/document-your-data/metadata-editor#when-there-are-more-results-than-the-list-shows
[column description propagation]: /docs/catalog/document-your-data/catalog-scribe/column-description-propagation
[Upload existing descriptions]: /docs/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions
[Ownership]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/ownership
[History]: /docs/catalog/collaborate/history

---

## PII Tagging Suggestions

Catalog uses an auto-detection mechanism to handle Personally Identifiable Information (PII). The process analyzes metadata to identify and categorize PII.

## Auto-Detection Process

### Metadata Scanning

Catalog automatically scans metadata to identify potential PII. By analyzing metadata, it can detect sensitive information without accessing the actual content of the data.

### Detection Patterns

The auto-detection model uses predefined detection patterns to identify PII. These patterns include rules and regular expressions tailored to recognize common forms of PII.

## Categorization of PII

### Identification and Tagging

Once Catalog detects potential PII, it categorizes and tags the information accordingly. Tagging the detected PII helps classify and manage sensitive information effectively, ensuring that it is handled appropriately.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

---

## Pinned Assets

On an asset page, you might want to give it more context by pinning the relevant tables or dashboards to a given asset.

Here's how it works:

## Pin an Asset

Click the "+" or `See suggestions` to open the modal and select your relevant assets.

<figure><figcaption></figcaption></figure>

The first "AI Suggestions" tab offers suggestions of assets to pin based on their semantic proximity. For example, if your table is called users, you may get suggestions of dashboards dealing with user activity.

<figure><figcaption></figcaption></figure>

Other tabs allow you to search and pick assets of different types.

<figure><figcaption></figcaption></figure>

When you're done, close the modal, now any Catalog can see the assets you've just pinned.

<figure><figcaption></figcaption></figure>

## Mentioned In

When you're done pinning assets and you're looking at your favorite table, you can visualize all the assets that pinned your table. Those are grouped in the "Mentioned in" section of the sidebar.

---

## Query Explainer ✨

## The Magic

Reading SQL takes time and is not accessible to all. Catalog AI is here to support you 🦸‍♂️

Catalog AI explains in natural English what a SQL query does.

The Query Explainer is accessible from:

* A table's queries tab
* A table's source SQL
* A dashboard's source SQL
* A user's queries

## Safety

The Query Explainer uses OpenAI models. The feature requires admin approval (before then, no metadata will be shared with OpenAI). Once the feature is activated, only the queries for which a user requests an explanation are shared with OpenAI. It is worth noting that we filter out PII information from queries. For more details, please view the Catalog AI [safety page](/docs/catalog/support/catalog-ai-safety).

## Activate Catalog AI on Your Account

To see this live in your Catalog account, an admin just needs to reach out to [support@coalesce.io](mailto:support@coalesce.io) 😊

---

## Add Tags to Assets

Learn how to add and manage tags on your Catalog assets. You can import existing tags from your sources or create new tags directly in Catalog.

## Import Tags

Catalog retrieves metadata from all your warehouses and visualization tools. That metadata can include tags from your sources. In the right sidebar, imported tags appear with a lock icon.

<figure><figcaption></figcaption></figure>

## Create Tags in Catalog

To create a tag in Catalog, click the "+" in the Tags section of the sidebar and add or create tags on the fly.

<figure><figcaption></figcaption></figure>

Tags appear on the asset page and on asset list pages.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

## Link a Tag to a Knowledge Page

From a knowledge page, determine which tag relates to the definition you are writing so all tagged assets appear on the page directly and dynamically.

See [Link a tag to a knowledge page][] for instructions.

## Search by Tags

In the search bar at the top of your screen, you can search for a tag across your entire data.

For example, to retrieve everything tagged with "finance":

<figure><figcaption></figcaption></figure>

## What's Next?

- [Link a tag to a knowledge page][]
- [Tagging and Classification][]
- [Tag Manager][]

[Link a tag to a knowledge page]: https://docs.coalesce.io/docs/catalog/assets/knowledge#link-a-tag
[Tagging and Classification]: https://docs.coalesce.io/docs/catalog/document-your-data/tags
[Tag Manager]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/tag-manager

---

## Tagging and Classification

Learn how to tag and classify your data assets in Coalesce Catalog so you can discover, govern, and secure your data effectively.

## Why Tag and Classify Data?

Tags improve how you find and manage data assets. Here's what they do:

1. **Improve discoverability:** You can filter and search by tags instead of relying on asset names or schemas alone.
2. **Support governance, compliance, and security:** Tags help you apply retention rules, access policies, and compliance controls (for example, PII or sensitive data).
3. **Provide semantic context:** Tags describe content, usage, ownership, quality, and business relevance in a way that both technical and non-technical audiences can understand.

**Tagging in Catalog:** Coalesce Catalog lets you import existing tags from your sources and add new tags directly in Catalog.

:::info[Imported vs Catalog-Managed Tags]
We recommend keeping imported tags focused on technical and lifecycle metadata (for example, source, platform, stage). Use Catalog-managed tags for governance, compliance, and discovery tags that non-technical audiences need. This keeps technical metadata in sync with your sources while allowing you to layer business semantics in Catalog.
:::

## Tag Origins and Terminology

Catalog uses three tag origins. These map to the concepts above:

| Origin | Meaning |
| -------- | --------- |
| **User tags** | Catalog-managed tags created by your team in Catalog |
| **External tags** | Imported from your sources (warehouses, BI tools) |
| **Catalog tags** | Auto-added by Catalog when it parses queries or detects sources (for example, source detection) |

Only User tags can be edited. See [Tag Manager][] for admin workflows.

## Add Tags to Your Assets

To add tags to assets, you can import them from your sources or create them in Catalog. See [Add tags to assets][] for step-by-step instructions.

## Faceted Tag Format

We recommend using faceted tags that specify `type:meaning`. This format makes tags easier to search, filter, and govern.

**Examples:**

- `domain:finance`, `domain:marketing`
- `function:compliance`, `function:risk`
- `concept:customer`, `concept:invoice`
- `useCase:ML`, `useCase:reporting`
- `sensitivity:PII`, `sensitivity:confidential`
- `quality:incomplete`, `quality:trusted`
- `stage:UAT`, `stage:production`
- `source:CRM`, `source:GoogleAds`
- `expirydate:20270101`

## Types of Tags

To be effective, tags should be meaningful, consistent, and governed. The categories below are a starting point, not an exhaustive list.

### Business Context Tags

Business context tags add business meaning to data assets.

**Examples:**

- **Domain:** `Finance`, `Marketing`, `HR`
- **Sub-domain or function:** `Compliance`, `Inbound`, `Talent`
- **Business entity (concept):** `Customer`, `Invoice`, `Product`
- **Use case (optional):** `Reporting`, `Forecasting`, `ML Feature Store`

**Benefits:** Helps you find data by business relevance rather than technical schemas.

### Sensitivity and Compliance Tags

These tags classify data by sensitivity, privacy, or legal requirements.

**Examples:**

- `PII`, `PHI`, `Confidential`, `Public`
- `GDPR`, `HIPAA`, `Crown Jewels`

**Purpose:**

- Drives access control policies
- Enables audit readiness
- Lets you mark sensitive assets without exposing their contents

### Quality and Trust Tags

These tags are optional. You can replace them with data quality tests and certification or deprecation workflows, or use them alongside. They capture data quality and reliability.

**Examples:**

- `High Quality`, `Stale`, `Incomplete`
- `Trusted`, `Reviewed`, `Deprecated`

**Purpose:**

- Guides you before consuming data
- Integrates with governance and stewardship workflows in Catalog

### Technical Metadata and Lifecycle Tags

These tags are very often imported from your source systems. They provide technical context and lifecycle state.

**Examples:**

- **Data source type:** `Snowflake`, `Redshift`, `Postgres`
- **Lifecycle maturity:** `Development`, `Staging`, `Production`, `ExpiryDate`
- **Type:** `KPI`, `Microservices`
- **Source:** `CRM`, `GoogleAds`. [Source tags][] are one example of the `source:` facet that Catalog adds automatically.

**Purpose:**

- Enables filtering and grouping in user interfaces
- Aligns with data lifecycle policies

## Knowledge Structure

Not every tag needs a corresponding entry in the Knowledge section. We recommend adding a knowledge page for each tag you add so you can explain what that tag means and how to use it.

Here's an example of how you might structure knowledge around your tags:

**Business Domain** (key domains)

- **Finance** (list of functions)
  - Compliance
  - Risk
- **HR**
  - Talent
  - Recruitment

**Business Entity** (typically conceptual, optional)

- Customer
- Product

**Metrics** (list of metrics with tags linking to the relevant domain and function)

- **Revenue** (groupings by metric type)
  - ARR
  - NRR
  - GRR

**Glossary terms** (groupings, acronyms, and business jargon)

- **Territory**
  - EMEA
  - Central
- QBR
- GDPR

**Sources** (data sources)

- Google Ads
- Salesforce
- HubSpot

## New Assets

Catalog highlights assets added in the last 30 days so contributors know what to document and review. For warehouse tables, tables are not flagged as new if their schema was created in the last 7 days.

## What's Next?

- [Add tags to your assets][]
- [Link tags to knowledge pages][]
- [Configure tag settings][]

[Add tags to assets]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/add-tags-to-assets
[Add tags to your assets]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/add-tags-to-assets
[Link tags to knowledge pages]: https://docs.coalesce.io/docs/catalog/assets/knowledge#link-a-tag
[Configure tag settings]: https://docs.coalesce.io/docs/catalog/administration/tag-settings
[Tag Manager]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/tag-manager
[Source tags]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/source-tags

---

## Source Tags

Source tags are automatically added based on the source tool for your data. They follow the `source:` facet pattern described in the [Tagging and Classification][] overview. These tags appear alongside other tags for all your assets so you can easily see where your data originates. [See all 120 sources Catalog identifies][].

You can see them in the **`Catalog Insights`** section of the right panel of Tables and Dashboards.

<figure><figcaption></figcaption></figure>

## How It Works

1. Catalog automatically finds data coming from most SaaS tools
2. Catalog adds a "Catalog Insights" tag on the raw tables and propagates it to children tables and dashboards

## Catalog Insights Knowledge

Additionally, Catalog creates a Knowledge page so you can see all tables and dashboards using data from a specific source (_HubSpot_ in the above example)

<figure><figcaption>Find a dedicated section for the source tags Catalog identifies and documents, with all their related tables and dashboards.</figcaption></figure>

<!-- vale off -->
<details>

<summary>120 Sources identified</summary>

* Active Campaign
* Google Ads
* Adyen
* Aircall
* Airship
* Amplitude
* Apple Search Ads
* Asana
* Beamer
* Billfoward
* Bing Ads
* BlueDot
* Blueshift
* Braintree
* Braze
* Candu
* ChargeDesk
* Chatlio
* CleverTap
* CommandBar
* Customer.io
* Datadog
* Delighted
* Dialpad
* Drip
* Ethoca
* Facebook
* Facebook Ads
* Fluctuo
* Foursquare
* Fox Intelligence
* Freshdesk
* Friendbuy
* Freshpaint
* GeoNames
* GitHub
* Gladly
* GoCardless
* Google
* Google Analytics
* Google Calendar
* Google Play
* Google Sheet
* Greenhouse
* HEROW
* HighJump
* HubSpot
* InMoment
* Insider
* Intercom
* Iterable
* iTunes
* Jebbit
* Jira
* Klaviyo
* Klenty
* Leanplum
* LinkedIn ads
* Livestorm
* Mailjet
* Mandrill
* Marketo
* Mastercard
* Mavenlink
* Microsoft Ads
* Mixpanel
* MixRank
* MoEngage
* Moesif
* NetSuite
* Notion
* Nudgespot
* Omeda
* OpenStreetMap
* Optimizely
* Outreach
* PagerDuty
* Salesforce
* PayPal
* Pendo
* Pinterest
* PostHog
* ProuveSource
* Qonto
* QuickBooks
* Recurly
* Refiner
* Revolut
* Sailthru
* SalesRep
* Selligent
* SendGrid
* Sentry
* MailChimp
* Shopify
* Slack
* Snapchat
* Sprig
* Staffomatic
* Statsig
* Stripe
* Taboola
* ThoughtSpot
* TikTok
* TikTok Ads
* Trello
* Twilio
* Twitter
* Typeform
* Unit4
* USPS
* Vero
* When I Work
* Wootric
* Xero
* YouTube
* YouTube Ads
* Zendesk
* Zenhub
* Zingtree
* Zuora

</details>
<!-- vale on -->

[See all 120 sources Catalog identifies]: https://docs.coalesce.io/docs/catalog/document-your-data/tags/source-tags#120-sources-identified
[Tagging and Classification]: https://docs.coalesce.io/docs/catalog/document-your-data/tags

---

## Tag Manager

The Tag Manager lets admins manage tags associated with assets in your Catalog instance. It ties into the [Tagging and Classification][] overview: User tags are Catalog-managed, External tags are imported from sources, and Catalog tags are auto-added by Catalog.

## Accessing Tag Manager

To access the Tag Manager, follow these steps:

1. Log in to your Catalog admin account.
2. Navigate to the "Governance" page.
3. Go to the "Tag Manager" tab.

Or open [Tag Manager in Governance](https://app.castordoc.com/governance/tag-manager).

<figure><figcaption></figcaption></figure>

For each tag you will find the number of assets linked to it and the origin of the tag. Origins map to the [Tagging and Classification][] overview:

* **User tag:** Catalog-managed, created by your team in Catalog
* **External tag:** Imported from your sources (warehouses, BI tools)
* **Catalog tag:** Auto-added by Catalog when parsing queries or detecting sources

## Edit Tags

By clicking the pen icon on the right, you will be able to edit your tags: rename it and change its color.

:::info[User Tags Only]

Only User tags can be edited.

:::

<figure><figcaption></figcaption></figure>

## Navigate

From the Tag Manager you can also navigate to the Tag Page or the Knowledge Page associated with the tag.

### Tag Page

Click the tag line to land on this page which will display:

1. The linked Knowledge Page
2. All the assets tagged

<figure><figcaption></figcaption></figure>

### Knowledge Page

By clicking the book icon on the right you'll land on the Knowledge Page associated with this tag.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

[Tagging and Classification]: /docs/catalog/document-your-data/tags

---

## Templates

Templates for asset documentation in the Catalog allow administrators to establish standardized formats for documentation, ensuring consistency and clarity across all materials. By defining templates, admins can guide contributors in structuring their documentation, thereby improving its quality and usability.

## Accessing Templates

Follow the [governance templates link][], or:

1. Log in to the Catalog with your admin credentials.
2. Navigate to the governance page.
3. Select the "Template" tab.

## Creating Templates

:::info[Define Asset Types]

When creating or editing a template, don't forget to define the type of assets it should be usable on: tables, dashboards, or Knowledge pages.

:::

1. Click on the "Create New Template" button.
2. Name the template to reflect its purpose or content.
3. Choose the type of assets the templates should be usable on.
4. Define the structure of the template by adding sections, subsections, and any necessary formatting guidelines.

<figure><figcaption>Create and fill the template</figcaption></figure>

## Applying Templates

Contributors (and admins) can apply templates when editing documentation for tables, dashboards, or Knowledge articles:

1. Access the relevant documentation section for editing.
2. Click the template icon in the edition bar.
3. Select the desired template from the dropdown menu.
4. The template's structure and formatting guidelines will be applied to the document.
5. Then you just have to input your documentation.

:::info[Apply Before Writing]

Remember to apply the template before writing your documentation.

:::

<figure><figcaption></figcaption></figure>

[governance templates link]: https://app.castordoc.com/governance/templates

---

## Upload Existing Documentation

Your team already created documentation and you want to import it into Catalog. Choose the option that fits your content:

- [Upload existing descriptions][]—Load table and column descriptions from a CSV. Use when migrating from Snowflake or BigQuery exports, spreadsheets, or other tools.
- [Upload existing documentation][]—Import full documentation (readmes, guides) from third-party tools into Catalog.

[Upload existing descriptions]: https://docs.coalesce.io/docs/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions
[Upload existing documentation]: https://docs.coalesce.io/docs/catalog/document-your-data/upload-existing-documentation/upload-existing-docs

---

## Upload existing Descriptions

:::info[Import Does Not Overwrite]

This process imports initial descriptions only. It does not overwrite existing user-written descriptions in Catalog.

:::

Upload your existing table and column descriptions into Catalog using a CSV file. This works well when you have descriptions in spreadsheets, exported from another tool, or from a warehouse that Catalog doesn't ingest directly.

## When To Use This Method

Upload existing descriptions is ideal when:

* **Descriptions from an unconnected warehouse** - You've exported table and column descriptions from a warehouse that Catalog doesn't connect to, or you need to load them before the connection is set up.
* **Retrofitting older assets** - Legacy tables and columns lack descriptions; you've documented them elsewhere and need to bulk load.
* **Importing from spreadsheets or other tools** - Descriptions live in Excel, Google Sheets, or a data dictionary tool and you want them in Catalog.

If your descriptions already exist in a source that Catalog ingests, they are brought in automatically. See [Ingested descriptions from source][]. Use this CSV upload only when descriptions come from elsewhere. For ongoing bulk edits after import, use the [Metadata Editor][].

## Pre-Submission Checklist

Before sending your CSV:

* Valid CSV with no malformed rows.
* No duplicate keys in the `key` column.
* No empty values in any column.
* Use `quote_all` when exporting to CSV.
* **Excel or Microsoft 365:** Save as **CSV (Comma delimited)**, not **CSV UTF-8**. UTF-8 exports can include a byte-order mark that complicates ingestion and may delay processing.
* **Google Sheets:** Use **File > Download > Comma-separated values**.

## CSV Format

Send your descriptions in the following CSV format. Required columns:

* **`type_d`** - Type of the asset: `table` or `column`.
* **`key`** - Database path of the asset. Must match Catalog's asset keys exactly, including warehouse or source name when your Catalog uses it.
  * Table: `<database_name>.<schema_name>.<table_name>`
  * Column: `<database_name>.<schema_name>.<table_name>.<column_name>`
* **`description`** - The description for the asset. Markdown is supported for table descriptions only.

**Example row (table):**

```text
type_d,key,description
table,my_db.my_schema.my_table,"This table stores customer records."
```

**Example row (column):**

```text
type_d,key,description
column,my_db.my_schema.my_table.customer_id,"Unique identifier for each customer."
```

:::warning[Key Must Match Catalog]

The `key` must match the asset path as it appears in Catalog. The `key` column should not contain duplicates and no empty values should be provided in any of the columns. If your Catalog includes a warehouse or source name in the path, include it. Assets that don't exist in Catalog will be skipped.

:::

<ButtonLink to="/downloads/csv_template (1).csv">
  Load to Catalog - CSV Template
</ButtonLink>

## Submit Your File

Send the CSV to your Catalog Sales Rep or [support@coalesce.io][]. We'll ingest the file and inform you of the results.

## Troubleshooting

| Symptom | Likely cause | Action |
| :------ | :----------- | :----- |
| Descriptions not applied | Key format doesn't match Catalog, or assets don't exist | Check that keys match Catalog asset paths, including warehouse or source name when your Catalog uses it. Confirm assets exist in Catalog. |
| Duplicates or empty values | Invalid CSV data | Validate the CSV before sending. Remove duplicate keys and ensure no empty cells. |
| Processing delayed | File exported as CSV UTF-8 from Excel | Re-export as **CSV (Comma delimited)** and resubmit. |
| Existing descriptions unchanged | Expected behavior | Import does not overwrite user-written descriptions. |

For bulk edits after import, use the [Metadata Editor][].

[Ingested descriptions from source]: /docs/catalog/use-cases/document/content-and-context/descriptions/ingested-descriptions-from-source
[Metadata Editor]: /docs/catalog/document-your-data/metadata-editor
[support@coalesce.io]: mailto:support@coalesce.io

---

## Upload Existing Documentation(Upload-existing-documentation)

Import full documentation pages into Catalog by sending a CSV file with your content. Your Catalog Sales Rep can process the upload for you.

## CSV Format

Use the following columns in your CSV:

- **`path`** - Documentation page path. For root pages, use `/`.
- **`title`** - Title of the documentation page.
- **`content`** - The content of the documentation page.
- **`owner`** - Optional. Documentation page owner in email format.

Download the template:

<ButtonLink to="/downloads/csv_template.csv">
  Load to Catalog - CSV Template
</ButtonLink>

:::info[Export from Excel or Google Sheets]

We strongly recommend using the `quote_all` setting when exporting to CSV.

When you save from a spreadsheet, use a comma-delimited CSV file:

- **Excel or Microsoft 365:** Save as **CSV (Comma delimited)**, not **CSV UTF-8**.
- **Google Sheets:** Use **File > Download > Comma-separated values**.

:::

Send the completed CSV to your Catalog Sales Rep and they will load it into Catalog.

---

## Communication

Connect the Catalog to your communication tools. Integrate with Slack and Microsoft Teams to ask questions and get answers directly in your workflow.

<!-- import DocCardList from '@theme/DocCardList';

<DocCardList /> -->

---

## Microsoft Teams

The Microsoft Teams connector enables your users to interact with Coalesce Catalog directly within Teams.

Your users can:

- Ask questions directly to the Catalog Data assistant and get insights instantly.
- Receive automated notifications about assets they own and stay updated on changes that affect them.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

This document walks you through the steps required to install, use, and remove the custom Microsoft Teams extension within your organization.

## Prerequisites

Before you begin, ensure you have the following:

- **Microsoft 365 Admin Privileges:** You must have the appropriate permissions to manage apps within the [Microsoft 365 admin center](https://admin.microsoft.com/). For more information on admin roles, see [About admin roles](https://learn.microsoft.com/en-us/microsoft-365/admin/add-users/about-admin-roles).
- **Microsoft Teams Admin Center Access:** You need access to the [Microsoft Teams Admin Center](https://admin.teams.microsoft.com/) to add custom apps and manage policies.
- **Custom Extension Package (Manifest):** You should have the custom extension files ready for upload. Refer to the Teams App manifest schema for details.

Download the appropriate package for your region:

- **US Region:** For accounts using `app.us.castordoc.com` [appPackage.prod-us.zip](/downloads/appPackage.prod-us.zip)
- **EU Region:** For accounts using `app.castordoc.com` [appPackage-1.0.2.prod-eu.zip](/downloads/appPackage-1.0.2.prod-eu.zip)

- **Subscription or License for Microsoft Teams:** Ensure your organization is licensed to use Microsoft Teams. You can learn more at [Microsoft Teams licensing](https://learn.microsoft.com/en-us/microsoftteams/teams-add-on-licensing/microsoft-teams-service-description).
- **Access to the Integration Page:** You need the ability to visit [app.castordoc.com/settings/integrations](http://app.castordoc.com/settings/integrations) as an Admin user to initiate the integration.

## Microsoft Teams Integration

This section covers how to create the integration from the Coalesce Catalog Integrations Page.

### Step 1: Navigate to the Integration Page

Visit [app.castordoc.com/settings/integrations](http://app.castordoc.com/settings/integrations).

### Step 2: Select Microsoft Teams

Click the **Teams** card to start the installation process.

<figure><figcaption></figcaption></figure>

### Step 3: Grant Required Permissions

You're redirected to your [Microsoft Online sign-in page](https://login.microsoftonline.com/). During this stage, you may be prompted to grant the necessary permissions for the [Azure Bot Service](https://azure.microsoft.com/en-us/services/bot-service/), including reading chat messages, user profiles, and other [Microsoft Graph](https://learn.microsoft.com/en-us/graph/overview) API capabilities. Only a **Microsoft 365 admin** can grant these permissions.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

### Step 4: Complete Administrative Consent

Once the admin has consented, you're redirected back to [app.castordoc.com/settings/integrations](http://app.castordoc.com/settings/integrations), where a confirmation toast appears indicating the installation was successful.

<figure><figcaption></figcaption></figure>

Once the integration is set up, a **user synchronization** from Azure Active Directory (Entra) to the **Coalesce Catalog** is initiated. This synchronization is automatically re-scheduled to run **daily**, ensuring user data remains up to date.

The AI Assistant in Microsoft Teams only responds to users with **active accounts in the Coalesce Catalog**. If a user doesn't have an active account, the AI Assistant ignores their messages.

## Microsoft Teams Extension Installation

This section covers how to install the Teams extension in your organization.

### Step 1: Enable Custom App Upload

If your organization has restricted the use of custom apps, you must enable the upload of custom apps before proceeding.

1. **Log into the Admin Center:** Sign in at the [Microsoft Teams Admin Center](https://admin.teams.microsoft.com/).
2. **Permission Policies:** Navigate to [Teams apps > Permission policies](https://admin.teams.microsoft.com/policies/app-setup).
3. **Allow Custom Apps:** Under **Global (Org-wide default)** or the relevant policy, ensure **Custom Apps** is set to **Allowed**.
4. **Save:** Click **Save**.

### Step 2: Prepare the Package for Upload

1. Download the appropriate Teams package for your region:
   - **US Region:** For accounts using `app.us.castordoc.com` [appPackage.prod-us.zip](/downloads/appPackage.prod-us.zip)
   - **EU Region:** For accounts using `app.castordoc.com` [appPackage-1.0.2.prod-eu.zip](/downloads/appPackage-1.0.2.prod-eu.zip)
2. Locate the downloaded `.zip` file containing the custom extension manifest.
3. If you need to edit the manifest file to update icons, descriptions, or version, do so before proceeding. You can refer to the [Teams App Studio](https://learn.microsoft.com/en-us/microsoftteams/platform/concepts/build-and-test/app-studio-overview) for help.
4. Confirm that your manifest file references the correct application endpoints.

### Step 3: Install the Extension

Microsoft Teams offers 2 ways to install custom extensions: using the Teams Client or the Microsoft Teams Admin Center Panel.

#### Install Using the Teams Client

If your organization has **Allow custom app uploads** enabled for end users, you can install the extension directly from within the Microsoft Teams client.

1. **Open Teams:** In the Microsoft Teams desktop or web client, select **Apps** on the left navigation.

    <figure><figcaption></figcaption></figure>

2. **Upload a Custom App:** At the bottom of the Apps pane, choose **Upload a custom app**.

    <figure><figcaption></figcaption></figure>

    <figure><figcaption></figcaption></figure>

3. **Select File:** Browse to the `.zip` file containing your custom extension.

    <figure><figcaption></figcaption></figure>

4. **Consent (If Required):** If prompted for app permissions, only a Microsoft 365 admin can grant these permissions organization-wide.
5. **Complete Installation:** Once uploaded, you should see a confirmation message, and your new extension appears on your app list.

#### Install Using the Admin Center

If you're an admin and want to upload the extension directly through the Microsoft Teams Admin Center, follow these steps.

1. **Manage Apps:** In the [Microsoft Teams Admin Center](https://admin.teams.microsoft.com/), go to [Teams apps > Manage apps](https://admin.teams.microsoft.com/teamsapps/manage-apps).

    <figure><figcaption></figcaption></figure>

    <figure><figcaption></figcaption></figure>

2. **Upload:** Click **Upload**.

    <figure><figcaption></figcaption></figure>

3. **Select File:** Browse to the `.zip` file for your custom extension and select it.
4. **Complete Upload:** Wait for the upload process to complete.

### Step 4: Assign the Extension to Users Using App Setup Policies

If you want specific users or groups to receive notifications from the Coalesce Catalog, the Teams Catalog app must be installed for those users. You can either have users install the extension through self-service, or you can deploy it to them using **App setup policies**.

1. **Setup Policies:** Under [Teams apps](https://admin.teams.microsoft.com/policies/manage-apps) in the Microsoft Teams Admin Center.

    <figure><figcaption></figcaption></figure>

2. **Select Coalesce Catalog, then select the Users and Groups tab.**

    <figure><figcaption></figcaption></figure>

3. **Search and Add:** Search for your newly uploaded custom extension and add it.
4. **Save:** Click **Save**.

### Step 5: Create a Custom App Setup Policy

1. In the left navigation of the Teams Admin Center, expand **Teams apps** and select **Setup policies**.
2. Click **+ Add** to create a new policy.
3. Name it something recognizable (e.g., "**Coalesce App Install Policy**").
4. Scroll down to the **Installed apps** section.
5. Click **Add apps**.
6. Search for **Coalesce**, click **Add**, and then click **Add** again at the bottom of the pane.
7. *(Optional but recommended)* Under **Pinned apps**, you can also add **Coalesce** here if you want the app icon to be permanently pinned to their left-hand Teams navigation bar so they don't miss it.
8. Click **Save** at the bottom of the policy page.

### Step 6: Assign the Policy to Your Specific Groups

Now you need to apply this pushing policy to the groups you identified.

1. Still on the **Setup policies** page, look for the **Group policy assignment** tab at the top (next to '**Manage policies**').
2. Click **+ Add group**.
3. Search for the group that you want the application pushed to.
4. Select a **rank** (rank determines which policy wins if a user belongs to multiple groups; rank 1 is the highest).
5. Under **Select a policy**, choose the "**Coalesce App Install Policy**" you just created.
6. Click **Apply**.
7. Repeat this process for the **Toll Analytics - Data and BI** group.

### Step 7: Done

Once the extension is installed, the user has access to it, and they have a provisioned account in catalog, they can interact with the AI Assistant in 2 ways:

1. In **1-to-1 chat**.
2. In a **group chat**, by mentioning **@Coalesce Assistant**, followed by your question.

## Microsoft Teams Extension Uninstall

This section covers how to remove the Teams extension from your organization.

### Uninstall Using the Teams Client

If you installed the extension directly through the **Teams client**, you can remove it at the user level.

<figure><figcaption></figcaption></figure>

1. **Open Teams:** In the Microsoft Teams desktop or web client, select **Apps** on the left navigation.
2. **Find Your Extension:** Look for your custom extension under **Built for your org** or **Uploaded**.
3. **Open the Extension Menu:** Click the extension's name or right-click.
4. **Uninstall:** Select **Uninstall** or **Remove** and confirm if prompted.

### Uninstall Using the Admin Center

If you want to remove or uninstall the custom extension from your **organization**, follow these steps.

<figure><figcaption></figcaption></figure>

1. **Unpublish or Block the Extension:**
   - Navigate to [Teams apps > Manage Apps](https://admin.teams.microsoft.com/teamsapps/setup-policies) in the Microsoft Teams Admin Center.
   - Locate the custom extension.
   - Select **Block** to disable it temporarily, or **Delete** to permanently remove it from your tenant.
2. **Confirm Removal:**
   - Wait a few minutes for the changes to propagate.
   - Sign out and sign back in to Microsoft Teams to ensure the extension is no longer visible.

## Troubleshooting

This section covers common issues and how to resolve them.

**Extension Not Showing Up:** Double-check the app permissions and setup policies in the [Teams admin center](https://admin.teams.microsoft.com/).

**Missing Permissions:** Ensure your account has the necessary [Microsoft 365 admin roles](https://learn.microsoft.com/en-us/microsoft-365/admin/add-users/about-admin-roles).

**Provide Tenant ID:** If you contact support, your Microsoft 365 tenant ID helps track activity in the logs, making diagnosing and resolving any issues more straightforward.

For further assistance or clarifications, reach out to the support team.

---

## Slack

Using the Catalog Slack app, your users will be able to:

- Ask any question to the Catalog Data Assistant
- Get notifications about assets they own
- Provide in-context information on any Catalog URL shared

## Ask Questions From Slack to the Data Assistant

When you are in a channel, type `@Coalesce` and ask your question. The Assistant will answer you. Make sure to invite the Assistant first in your channel.

See more on the Assistant in the [AI Assistant introduction][].

<figure><figcaption></figcaption></figure>

## User and Team Notifications in Slack

### Notification of New Comments

Whenever someone posts a comment on your owned assets, you will receive a notification.

In case an asset does not have an owner, admins will get notified (probably a good sign that the asset should have one of several owners).

### Notification of Metadata Changes

:::info[Batch Notifications]

To avoid spamming, the following notifications are grouped in hourly batches. Catalog waits for 60 minutes without new notifications to send them as a batch.

:::

The following actions will trigger a notification to be sent to the table owner or the admins if there is no owner:

- A table or column description is edited
- An owner is added
- A tag is added or updated
- The table is Certified or Uncertified
- The table is Deprecated or Undeprecated

<figure><figcaption></figcaption></figure>

## Catalog Link Prettified and Sync Back to Catalog

### Get People To Share Catalog URLs When They Have a Data Question

That way, you are sure that they have checked the doc before asking a question.

### The Catalog Slack App Adds Context to the URL

### It Saves Your Thread on Catalog App

By clicking on **Save thread link in Catalog**, the message URL is saved in Catalog, on that Table, Dashboard, or Knowledge page, in the Comments tab.

## How To Set Up the Catalog Slack App

At Catalog, we love to make things simple.

Requirements: be an Admin.

1. Go to the [Integrations Set Up][] page
2. Click on the Slack tile
3. Add the Catalog app on Slack

Make sure to add the Catalog Slack Application in your Public Channel where you want it to run. By default it will not be added, to give you full control on where it can act.

## Set Up Your Teams Notifications Channel

By following the [documentation on teams notifications][].

[AI Assistant introduction]: https://docs.coalesce.io/docs/catalog/ai-assistant/introduction
[Integrations Set Up]: https://app.castordoc.com/settings/integrations
[documentation on teams notifications]: https://docs.coalesce.io/docs/catalog/collaborate/notifications/notifications-for-teams

---

## Domo

Connect Catalog to Domo to sync Pages, Cards, data sets, and relationships between them. This guide describes what Catalog extracts, how Cards appear as tile-style dashboards when you browse and search, credential options for Catalog-managed or client-managed sync, IP allowlisting, and the extractor package.

## Requirements

You need a Warehouse-type integration configured before the first ingestion for this integration can complete.

For all Domo clients:

- Domo admin credentials
- Domo Developer Token
- Internal alignment if you use Domo as a warehouse

## What Catalog Extracts from Domo

Catalog extracts Domo **Pages**, **Cards**, and **data sets**, along with metadata Domo exposes through its APIs, for example names, descriptions where available, links to the asset in Domo, and ownership or editing context when present.

- **Pages** - Domo dashboards and stories appear in Catalog as dashboard assets so you can browse and document the containers your teams use for reporting.
- **Cards** - In Domo, Cards are the visualizations you add to Pages, such as KPIs and charts. Catalog syncs Cards alongside Pages and data sets so you can document and govern each visualization, not only the Page that contains it. Each Card is stored as a **dashboard** asset with dashboard type **TILE**, the same tile-style type used in other BI integrations, and appears under the **/Cards** path in the asset tree next to Page-level dashboards.
- **Data sets** - Domo data sets are cataloged as data assets so you can relate Domo content back to the data that powers it.
- **Lineage** - Catalog records **asset-level** relationships among Domo data sets, Cards, and Pages. The lineage graph reflects how those Domo objects connect; it is not a substitute for field-level or expression-level detail inside a Card.
- **Search and governance** - Card and Page metadata is available in Catalog search, stewardship workflows, and the AI Assistant alongside other dashboard assets. For the fields used in AI Search, see [AI Assistant Context][].

When you validate warehouse-to-BI lineage and a Card sits in the path, open the Card in Catalog to see how it connects to upstream data sets and related Pages.

## If Needed, Allowlist Catalog IPs

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

## Catalog Managed

To get things started with Domo in Catalog, you'll need to be a Domo admin and provide us with:

- A `Client ID` - Domo [documentation][]
- A Domo `API token`, which Domo calls the Client Secret - Domo [documentation][]
- The `Base URL` of your Domo instance as `https://XXXX.domo.com/`
- A `Developer Token` - Domo [developer token documentation][]
- The `Cloud Id` of your external Warehouse. Omit this field when it does not apply.

For more information on how to retrieve those values, see the [Domo API quickstart][].

You can input your credentials directly into your Catalog account under the following format:

```json
{
    "apiToken": "*****",
    "clientId": "*****",
    "baseUrl": "https://XXXX.domo.com/",
    "developerToken": "*****",

    "cloudId": "my-external-warehouse-cloud-id"
}
```

:::warning[SSO, Sign-In, and Dashboard Access]

If your Domo instance uses SSO, the Catalog integration account must have direct sign-on enabled. The Domo account that creates the client ID and secret in the developer portal must reach every dashboard you want in Catalog. In Domo, go to **Admin > Dashboards** and share those dashboards with that account.

:::

- `apiToken`: secret for the Catalog Domo integration connection
- `clientId`: client ID for the Catalog Domo integration connection
- `baseUrl`: Domo base URL and API endpoint, usually your Domo homepage URL
- `developerToken`: Token to access the Domo private API
- `cloudId`: ID of the external warehouse used within Domo

For your first sync, allow up to 48 hours. The Catalog team notifies you when it completes.

If you are not comfortable giving us access to your credentials, continue to [Client Managed](#client-managed).

## Client Managed

For your trial, you can supply a one-time extract view of your Domo BI assets. Use the Catalog package workflow below when you manage sync yourself.

### Running the Catalog Package

#### Install the PyPI Package

```bash
pip install castor-extractor
```

For further details, see the [castor-extractor PyPI page][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-domo [arguments]
```

### Arguments

The Domo extractor accepts the following flags:

- `-c`, `--client-id`: Domo client ID
- `-a`, `--api-token`: Domo access token
- `-d`, `--developer-token`: Domo developer token
- `-b`, `--base-url`: Domo base URL
- `-o`, `--output`: target folder to store the extracted files

### Optional Arguments

Use this flag when you rely on an external warehouse instead of Domo as the warehouse:

- `-C`, `--cloud-id`: Required when you use an external warehouse and not Domo as a warehouse

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Domo content in Catalog. Follow these steps:

The Catalog team will provide you with:

1. `Catalog Identifier`, an ID the Catalog team uses to match your Domo files to your Catalog instance
2. `Catalog Token`, an API token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

### Castor-Upload Arguments

`castor-upload` supports these flags:

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: Account ID provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are `DBT`, `VIZ`, or `WAREHOUSE`

### Target Files

Choose one upload target:

- `-f`, `--file_path`: Push a single file.
- `-d`, `--directory_path`: Push every file in a folder at once.

When you use `--directory_path`, `castor-upload` sends every file in that folder. Limit the folder to extracted Domo files only before you run the command.

Then schedule the extract script and the upload to Catalog using your preferred scheduler.

## What's Next?

- Explore other [BI Tools][] integrations and onboarding options.
- Read [AI Assistant Context][] for technical detail on which dashboard fields power AI Search.
- For Domo as a warehouse source, follow [Domo Data][].

---

[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[documentation]: https://developer.domo.com/portal/1845fc11bbe5d-api-authentication#step-1-create-client-id-and-secret
[developer token documentation]: https://domo-support.domo.com/s/article/360042934494?language=en_US
[Domo API quickstart]: https://developer.domo.com/portal/8ba9aedad3679-ap-is#quickstart
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install
[BI Tools]: /docs/catalog/integrations/data-viz
[AI Assistant Context]: /docs/catalog/ai-assistant/ai-assistant/ai-assistant-context
[Domo Data]: /docs/catalog/integrations/data-warehouses/domo-data

---

## Holistics.io

## Requirements

- You must be an organization Admin to create a Holistics API key.

## Catalog Managed

To get things started with Holistics in Catalog, you will need to be a Holistics admin and provide us with a Holistics `API token` associated to an `Admin` account.

For more information on how to retrieve that information, see the [Holistics documentation][].

You can input your credentials directly into your Catalog account under the following format:

```json
{
    "apiToken": "*****",
}
```

For your first sync, it will take up to 48 h and we will let you know when it is complete.

### Holistics to Catalog Translation

- "Dataset" = Catalog "Model"
- "Model" = Catalog "View"
- "Card" = Catalog "Tile"

[Holistics documentation]: https://docs.holistics.io/api/v2#section/Authentication

---

## BI Tools(Data-viz)

## Integrating BI Tools in Catalog

Your BI tool onboarding can either be:

- **Catalog managed:** You give us admin credentials and you have nothing else to do.
- **Client managed:** You use the `castor-extractor` package (which we provide for you) to extract the information we need and you then send them over to Catalog.

To integrate Looker, Metabase, Tableau, Qlik, or Mode, Catalog provides a PyPI package (for other BI tools see [BI Importer][]). Power BI has a dedicated native integration ([Power BI setup][]); use that path instead of BI Importer unless your Catalog team scoped a custom upload integration. Unlike Warehouses, we do not have direct access to your tool. We let you run the agent to extract the information that we need and send us the output files.

## Client Managed

### During Your Trial

- We will only ask you to run the package once; you can upload the resulting files directly in the app.
- The Dashboard section in Catalog will thus not be synced throughout your trial.
- Not to worry, after your trial, we have everything needed to keep your Dashboard sections up to date.

To get things working quickly, here is a [Google Colab][] to run our package swiftly. Just create a copy and input your credentials.

You can also follow the [castor-extractor installation instructions][] and run the castor-extractor locally.

Check out the requirements by technology below.

### Once Your Trial Is Over

- To keep your Dashboard section up to date after your trial, we will ask you to schedule the extraction at your desired frequency (up to 1x/day).
- To send us the output files, we will provide you with a GCP bucket so that you can automatically push them.

## Check Out Your Tool

<DocCardList />

## What If a BI Tool Technology Is Not Covered

If your BI tool technology is not listed as one of our integrations, we have got you covered with:

<ButtonLink to="/docs/catalog/developer/catalog-apis/bi-importer">
  BI Importer
</ButtonLink>

Basically, we provide a list of the necessary metadata we need to get things to show up in Catalog. You should explore the metadata you can access and format it into our templates.

If you are feeling overwhelmed by the magnitude of the task, just try filling in a couple examples by hand. This way, you will be able to test out Catalog's main functionalities.

Please reach out to the sales or ops team for more details.

[BI Importer]: https://docs.coalesce.io/docs/catalog/developer/catalog-apis/bi-importer
[Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Add an API Key to a Looker User

[View the Looker documentation on API keys][].

1. Go to the User List and select the user you want to edit.
1. As in the screen below, click on **Edit Keys**.

<figure><figcaption></figcaption></figure>

1. Click on **New API Key**.

<figure><figcaption></figcaption></figure>

1. Copy the `Client Id` and the `Client Secret`. You will add them in the Catalog App later.

<figure><figcaption></figcaption></figure>

[View the Looker documentation on API keys]: https://cloud.google.com/looker/docs/admin-panel-users-users#api_keys

---

## Looker(Looker)

Integrate Looker with the Catalog to sync dashboards, looks, and metadata.

After your Looker content syncs to Catalog, use the [Catalog browser extension in Looker][] while viewing dashboards for descriptions, ownership, and related Catalog features.

## Requirements

:::warning[Warehouse Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

To get things started with Looker in the Catalog, you will need:

* Credentials of a Looker admin
* An admin Looker API key. Follow the steps below "Create a Looker API Key"
* Read-only access to your Looker Git repository. Follow the steps below "Provide access to your LookML"

Related pages:

<!-- import DocCardList from '@theme/DocCardList';

<DocCardList /> -->

## Catalog Managed

Enter your credentials directly in the [Catalog App integration settings][]. You need:

* `base-url`: for example, `http://looker.catalog.com`
* `client-id`: API Key Client ID
* `client-secret`: API Key Client Secret

Give Catalog read-only access to your Looker Git repository. See how to [provide access to your LookML code][].

For your first sync, it can take up to 48 hours before ingestion completes. Catalog notifies you when it finishes.

If you prefer not to share credentials with Catalog, continue to [Client Managed](#client-managed).

## Client Managed

### Doing a One Shot Extract

For your trial, you can simply give us a one shot view of your BI tool.

To get things working quickly, here's a [Google Colab][] to run our package.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[looker]
```

For further details on the extractor PyPI package, see the [castor-extractor PyPI page][].

#### Run the PyPI Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-looker [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting users from Looker API
INFO - POST(https://cloud.looker.com/api/4.0/login)
INFO - GET(https://cloud.looker.com/api/4.0/users/search)
INFO - Fetched page 1 / 7 results
INFO - GET(https://catalog.cloud.looker.com/api/4.0/users/search)
INFO - Fetched page 2 / 0 results

...

INFO - Wrote output file: /tmp/catalog/1649079699-projects.json
INFO - Wrote output file: /tmp/catalog/1649079699-summary.json
```

#### Credentials

* `-c`, `--client-id`: API Key Client ID (mandatory)
* `-s`, `--client-secret`: API Key Client Secret (mandatory)

#### Other Arguments

* `-b`, `--base-url`: Looker base URL (mandatory)
* `-o`, `--output`: Target folder to store the extracted files (mandatory)
* `-t`, `--timeout`: Timeout (in s) parameter for Looker API
* `--log-to-stdout`: Will write all log outputs to `stdout` instead of `stderr`

#### Specific Export Methods for Looks and Dashboards

* `--search-per-folder`: Will export looks and dashboards per folder using multiple threads (see below argument)
* `--thread-pool-size`: Number of parallel threads, defaults to 20

Run any extractor command with `--help` to print the full argument list.

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Looker content in the Catalog. Here is how to do it:

The Catalog team will provide you with:

1. `Catalog Identifier` (an id for us to match your Looker files with your Catalog instance)
2. `Catalog Token` An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

* `-k`, `--token`: Token provided by Catalog
* `-s`, `--source_id`: account id provided by Catalog
* `-t`, `--file_type`: source type to upload. Currently supported are { `DBT` | `VIZ` | `WAREHOUSE` }

#### Target Files

To specify the target files, provide **one** of the following:

* `-f`, `--file_path`: to push a single file

 or

* `-d`, `--directory_path`: to push several files at once (\*)

:::warning[Directory Contents]

(\*) The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you'll have to schedule the script run and the push to the Catalog. Use your preferred scheduler to create this job.

## Troubleshooting

These scenarios apply to core Looker onboarding and first ingestion.

### Lineage Drops After Database or Connection Changes

When Explores resolve to a warehouse database your Warehouse integration does not ingest, lineage that depended on matching those tables can disappear after a successful Looker sync. That often follows a switch between development and production databases in LookML or a connection change. For symptoms, prevention, recovery steps, and verification, see [Lineage troubleshooting for Looker database and connection mismatches][].

### First Ingestion Runs Longer Than a Few Hours

Warehouse integrations must finish before visualization ingestion can complete. Large LookML repositories extend parsing time.

**Resolution:**

1. Confirm the Warehouse integration listed under **Requirements** is healthy in **Settings > Integrations**.
2. Verify read-only Git access to LookML still works from Catalog's perspective (SSH keys, tokens, branch protections).
3. If ingestion stays unfinished beyond forty-eight hours while both prerequisites are green, contact [Coalesce Support][].

### Looker Studio Versus LookML-Backed Looker Instances

Looker Studio connections follow Google's OAuth and connector scopes. Classic Looker uses API keys and Git-backed LookML. Mixing setup steps between products causes auth errors.

**Resolution:**

1. Follow the integration path that matches your product (Looker with LookML vs Looker Studio OAuth apps).
2. After changing Google Cloud scopes for Looker Studio, wait for the provider cache to refresh before re-testing ingestion.

### Cannot Validate API Credentials

Wrong base URL format or revoked secrets surface as login failures in extractor logs.

**Resolution:**

1. Confirm `base-url` matches Looker's documented pattern (`https://company.looker.com`) without typos.
2. Rotate API keys in Looker admin, update Catalog credentials immediately, and revoke old secrets.

### Missing Dashboards or Explores After a Partial Git Migration

Catalog indexes what appears in the Looker API plus accessible LookML.

**Resolution:**

1. Confirm the Git branch Catalog reads matches production LookML.
2. Run a fresh extraction after merges that rename explores or move files between folders.

### Browser Extension Panels Empty Inside Looker

The Chrome extension depends on Catalog indexing your Looker assets first.

**Resolution:**

1. Finish at least one successful Looker ingestion before expecting extension tiles.
2. See [Catalog browser extension in Looker][] for install steps.

[Catalog browser extension in Looker]: /docs/catalog/integrations/data-viz/looker/looker-chrome-extension
[Catalog App integration settings]: https://app.castordoc.com/settings/integrations
[provide access to your LookML code]: /docs/catalog/integrations/data-viz/looker/provide-access-to-your-lookml
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install
[Coalesce Support]: mailto:support@coalesce.io
[Lineage troubleshooting for Looker database and connection mismatches]: /docs/catalog/navigate/lineage/lineage-troubleshooting#lineage-disappears-after-a-database-or-connection-change-in-lookml

---

## Looker and the Catalog Browser Extension

While you view dashboards in Looker, the Catalog browser extension connects what you see in the browser to metadata synced in Catalog. This guide describes what you need in place, how different Looker dashboard URL patterns relate to Catalog, how to confirm the extension picked the right dashboard, and what to try when something looks off.

## Before You Begin

You need the following before the extension can show useful Catalog metadata for a Looker dashboard:

- **Looker content in Catalog** - Dashboards and related Looker assets must be synced through the [Looker integration][]. If extraction has not completed or a dashboard was added recently, wait for the next sync or run your extraction process according to your setup.
- **Access to Catalog** - Sign in to your Catalog Workspace in the browser where you use the extension so the extension can load metadata you are allowed to see.
- **A supported browser** - Install the extension in Chrome or Edge. Edge users install from the Chrome Web Store with extensions from other stores allowed. Full browser notes are on the [Browser Extensions][] page.

:::tip[Looker Admin Versus Extension Use]

Configuring the Looker integration requires Looker admin credentials and API access. Day-to-day extension use only requires that your organization finished syncing Looker to Catalog and that you can sign in to Catalog.

:::

## Install the Catalog Browser Extension

If you have not installed the extension yet, follow [How to Install the Extension][] on the Browser Extensions page. Your organization can also push the extension using the extension ID on the [Company-Wide Install][] page.

## How Looker Dashboard URLs Relate to Catalog

Looker dashboard addresses vary by instance. You might see a numeric dashboard ID in the path, a readable slug, or another URL shape your Looker admin configured. Catalog is built to recognize the dashboard you have open from these URL patterns so the extension can load the matching Catalog asset.

Use this flow when you verify behavior:

- When the URL includes a **numeric dashboard identifier**, Catalog associates the open page with the corresponding dashboard metadata ingested from Looker.
- When the URL uses a **slug** or another stable identifier in place of a bare numeric segment, Catalog still resolves the open dashboard to the same Catalog asset when extraction captured that dashboard.

If your instance switches URL styles between environments or after publishes, wait until Catalog finishes ingesting the revision you care about, then refresh the Looker page and open the extension again.

## Confirm the Extension Shows the Right Dashboard

Work through these checks when you first use the extension or after large Looker changes:

1. Open a Looker dashboard that you know exists in Catalog. If you are unsure, search for it in Catalog first.
2. Activate the Catalog extension from the browser toolbar. The extension panel opens beside your Looker content.
3. Confirm metadata matches what you expect for that dashboard, including descriptions, ownership, and popularity fields shown for your Workspace.
4. If your Workspace licenses Dashboard Q&A, open that capability from the extension and ask a simple factual question about the dashboard to confirm context lines up with the asset you selected in Catalog.

If labels or owners differ between Looker and Catalog, resolve extraction or synchronization first; the extension reflects what Catalog stores after integration runs.

## Troubleshooting

These situations cover the most common extension issues in Looker workflows.

### The Extension Does Not Appear or Stay Pinned

Confirm the extension is installed and enabled for your browser profile. On Edge, confirm you allowed extensions from the Chrome Web Store. Reload the Looker tab after enabling the extension. If your organization manages extensions centrally, ask your IT administrator to confirm the Catalog extension is allowed using the ID on [Company-Wide Install][].

### Metadata Describes a Different Dashboard or Looks Empty

Check that you are signed in to Catalog in the same browser and that the dashboard URL matches what was extracted, whether the path uses a slug or a numeric dashboard identifier. Run through [How Looker Dashboard URLs Relate to Catalog][], then refresh the Looker page and open the extension again after extraction completes.

### Sign-In or Permission Errors in the Extension

Clear stale Catalog sessions by signing out and back into Catalog in the browser, then reload Looker. If your organization uses more than one Catalog region or tenant, confirm you are logging into the Workspace that matches your Looker integration. When errors persist after those checks, capture the browser message you see and contact [Catalog Support][] with your Workspace name and the Looker dashboard URL you had open.

:::tip[Catalog Web App Versus the Browser Extension]

Problems that appear only inside the Catalog web application are separate from the Chrome or Edge extension. Note whether an issue reproduces on `app.castordoc.com` alone or only when the extension panel is open so support can route the report quickly.

:::

## What's Next?

- Configure or refine the [Looker integration][] so dashboards stay current in Catalog.
- Learn how to use [Dashboard Q&A][] from supported BI tools through the extension.
- Deploy the extension broadly with [Company-Wide Install][].

---

[Browser Extensions]: https://docs.coalesce.io/docs/catalog/navigate/browser-extensions
[How to Install the Extension]: https://docs.coalesce.io/docs/catalog/navigate/browser-extensions#how-to-install-the-extension
[How Looker Dashboard URLs Relate to Catalog]: #how-looker-dashboard-urls-relate-to-catalog
[Company-Wide Install]: https://docs.coalesce.io/docs/catalog/navigate/browser-extensions/company-wide-install
[Dashboard Q&A]: https://docs.coalesce.io/docs/catalog/ai-assistant/dashboard-q-and-a
[Looker integration]: https://docs.coalesce.io/docs/catalog/integrations/data-viz/looker
[Catalog Support]: https://docs.coalesce.io/docs/catalog/support/how-to-reach-us

---

## Provide Access to Your LookML

## During Your Trial

A simple way, yet static, is to clone your LookML repository, zip it, and send it over via Slack or email.

Once again, this is a static method: your lineage will not update throughout your trial.

## After Your Trial

### Grant Catalog Access to Your Git Repo (Read-Only)

#### 1. Send to the Catalog Team Your Repo SSH URL

- GitHub: You can find it by clicking **Clone** on your GitHub Looker repo.
- GitLab: You can find it by clicking **Code** on your GitLab Looker repo.

#### 2. Add the Deploy Key to Your Repo

In return, Catalog will provide you with a deploy key.

- GitHub: You can add it to your repo via Settings > Security > Deploy keys.
- GitLab: You can add it to your repo via Settings > Repository > Deploy keys.

---

## Looker Studio(Data-viz)

## Requirements

:::warning[Warehouse Integration Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

To get things started with Looker Studio you will need:

- To enable the [Looker Studio API][] in the chosen GCP project
- To enable the [Admin SDK API][] in the chosen GCP project

You will also need to create a Google Service Account. As such, you need to have at minimum one of the following roles:

- Service Account Admin
- Editor Basic

Finally, to give the needed rights to the service account you will need the following role:

- Google Workspace Admin

## If Needed, Whitelist Catalog IP

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

## Catalog Managed

### 1. Create Google Service Account for Catalog

Client creates a service account for Catalog from the [Google Console][]. See how to [create and manage service accounts][].

Make sure to create and download a JSON key for that service account. See how to [create service account keys][].

Take note of the Service Account **Client ID**, also known as **Unique ID**.

### 2. Authorize the Google Service Account for Catalog

Authorize the service account for an organization. See how to [authorize the app for Looker Studio][].

The scopes to add are:

- [`https://www.googleapis.com/auth/datastudio`](https://www.googleapis.com/auth/datastudio)
- [`https://www.googleapis.com/auth/userinfo.profile`](https://www.googleapis.com/auth/userinfo.profile)
- [`https://www.googleapis.com/auth/admin.reports.audit.readonly`](https://www.googleapis.com/auth/admin.reports.audit.readonly)
- [`https://www.googleapis.com/auth/admin.directory.user.readonly`](https://www.googleapis.com/auth/admin.directory.user.readonly)

### 3. Add Credentials in Catalog App

:::info[Catalog Admin Required]

You must be a Catalog admin to do it.

:::

You can now enter the newly created credentials in the Catalog App at [app.castordoc.com/settings/integrations][].

- Go to **Settings > Integrations**
- Click on **Looker Studio Add**

Share the following credentials:

- The JSON credentials of the service account created in step 1
- The email of a Google Workspace user with **admin** access

:::info[Service Account Impersonation]

The service account will impersonate this admin user in order to obtain the view activity of each Looker Studio asset. Rest assured that the service account's actions are limited by the scopes and IAM roles or permissions.

:::

Combine both into this expected format:

```json
{
    "admin_email": "<str>",
    "auth_provider_x509_cert_url": "<str>",
    "auth_uri": "<str>",
    "client_email": "<str>",
    "client_id": "<str>",
    "client_x509_cert_url": "<str>",
    "private_key": "<str>",
    "private_key_id": "<str>",
    "project_id": "<str>",
    "token_uri": "<str>",
    "type": "<str>"
}
```

## Client Managed

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[lookerstudio]
```

#### Optional: BigQuery Access

If your Looker Studio data sources are connected to BigQuery:

- Make sure to install the BigQuery dependencies as well:

```bash
pip install castor-extractor[lookerstudio,bigquery]
```

- You will require a Service Account with BigQuery access. Consider using the Service Account linked to your BigQuery integration. Note that this can be the same Service Account as for Looker Studio, or a separate one.

:::info[Extractor Package]

For further details on our extractor PyPI package, see the [castor-extractor installation instructions][].

:::

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-looker-studio [arguments]
```

The script will run and display logs as following:

```bash
INFO - Loading credentials from /looker/studio/credentials.json
INFO - Loading credentials from /bigquery/credentials.json
INFO - Extracting ASSETS from API
INFO - Refreshing authentication token...
INFO - Refreshing authentication token...

...

INFO - Wrote output file: /tmp/ls/1742219344-summary.json
```

#### Credentials

The required arguments depend on your extraction goal:

**For a full Looker Studio extraction:**

- `-c`, `--credentials`: File path to Service Account credentials with Looker Studio access
- `-a`, `--admin-email`: Email of a Google Workspace user with admin access

**For source-queries-only mode:**

- `-b`, `--bigquery-credentials`: File path to Service Account credentials with BigQuery access

#### Optional Arguments

- `-o`, `--output`: Directory to write the extracted files to
- `--source-queries-only`: If selected, only extracts BigQuery source queries (bypasses Looker Studio extraction)
- `--skip-view-activity-logs`: Skip extraction of activity logs (use if credentials lack required scopes)
- `--users-file-path`: Optional path to a JSON file with user email addresses as a list of strings (for example, `["foo@bar.com", "fee@bar.com"]`). If provided, only extracts assets owned by the specified users.
- `--db-allowed`: Optional list of GCP projects to allow for source queries extraction
- `--db-blocked`: Optional list of GCP projects to block from source queries extraction

:::info[Help]

You can always get help with the argument `--help`.

:::

### Scheduling and Push to Catalog

When moving out of trial, you will want to refresh your Looker Studio content in Catalog. Here is how to do it:

The Catalog team will provide you with:

1. `Catalog Identifier` (an id for us to match your Looker files with your Catalog instance)
2. `Catalog Token` - An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: account id provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are `DBT`, `VIZ`, or `WAREHOUSE`

#### Target Files

To specify the target files, provide **one** of the following:

- `-f`, `--file_path`: to push a single file

or

- `-d`, `--directory_path`: to push several files at once

:::warning[Directory Contents]

The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will have to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

You are done.

[Looker Studio API]: https://console.cloud.google.com/apis/library/datastudio.googleapis.com
[Admin SDK API]: https://console.cloud.google.com/apis/api/admin.googleapis.com
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[Google Console]: https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts?supportedpurview=project
[create and manage service accounts]: https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating
[create service account keys]: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys
[authorize the app for Looker Studio]: https://developers.google.com/looker-studio/integrate/api#authorize-app
[app.castordoc.com/settings/integrations]: https://app.castordoc.com/settings/integrations
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Metabase(Data-viz)

Integrate Metabase into Coalesce Catalog so dashboard and metadata content stays available alongside your warehouse lineage.

## Requirements

:::warning[Warehouse Integration Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

Catalog ingests Metabase metadata using one of two technical paths: the **Metabase HTTP API**, or **read-only SQL** against the **PostgreSQL database Metabase uses as its application database**. Metabase stores questions, dashboards, and collections there. That PostgreSQL instance is not your warehouse.

How you connect breaks down along two choices: the **connector** and whether extraction is **Catalog managed** or **client managed**. The connector is either the API or the PostgreSQL application database. **Catalog managed** means credentials live in the Coalesce App. **Client managed** means you run the extractor. Use this table to match your setup to the connector:

| Setup | Connector | What you use |
| ----- | --------- | ------------ |
| Catalog managed | Metabase API | JSON credentials in the Coalesce App. See [Catalog Managed][]. |
| Client managed | Metabase API | `castor-extract-metabase-api` with API credentials. See [Client Managed][]. |
| Client managed | PostgreSQL application database | `castor-extract-metabase-db` with read-only PostgreSQL credentials for Metabase application tables. See [Client Managed][]. |

### Metabase API Prerequisites

For Catalog managed onboarding or the API extraction script, the Metabase account must meet Metabase privilege requirements for metadata access through the API. The castor-extractor documentation refers to this as the Metabase `superuser` role. That is a permission on the Metabase account, not a requirement tied to a person's job title. For API limitations such as popularity metrics, see [Metabase package documentation][].

### PostgreSQL Application Database Path

Read-only access to the PostgreSQL instance Metabase uses for application metadata applies when you run **client-managed** extraction with `castor-extract-metabase-db`. Catalog managed onboarding does not collect PostgreSQL host or database credentials for Metabase. See [Metabase configuration documentation][] for how Metabase configures its application database.

## Allowlist Catalog IP

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

## Catalog Managed

You can enter your credentials directly in the Coalesce App.

Using the API:

```json
{
    "baseUrl": "http://company.cloud.metabase.com",
    "user": "catalog",
    "password": "abcdefgh"
}
```

For your first sync, it will take up to 48 hours and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, continue to [Client Managed][].

## Client Managed

### Doing a One-Shot Extract

For your trial, you can simply give us a one-shot view of your BI tool.

To get things working quickly, here is a [Google Colab][] to run our package swiftly.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[metabase]
```

For further details, see the [castor-extractor installation instructions][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-metabase-db [arguments]

# or, if you use the API connector:
castor-extract-metabase-api [arguments]
```

The script will run and display logs as follows:

```bash
INFO - Getting session_id: {'id': '****'}
INFO - Fetching USER (15 results)
INFO - Wrote output file: /tmp/catalog/1649081473-user.json
INFO - Fetching COLLECTION (41 results)

...

INFO - Wrote output file: /tmp/catalog/1649081473-dashboard_cards.json
INFO - Wrote output file: /tmp/catalog/1649081473-summary.json
```

#### Credentials

You can sign in using one of the following methods:

- **Postgres connector**
  - `-H`, `--host`: Host name where the server is running
  - `-P`, `--port`: TCP/IP port number
  - `-d`, `--database`: Database name
  - `-s`, `--schema`: Schema name where the views or tables are located
  - `-u`, `--user`: User
  - `-p`, `--password`: Password
  - `-o`, `--output`: Directory to write to
  - `--require_ssl`: Flag to require SSL
- **API connector**
  - `-b`, `--base-url`: Metabase base URL, such as `http://company.cloud.metabase.com`
  - `-u`, `--user`: Metabase user
  - `-p`, `--password`: Metabase password

#### Other Arguments

- `-o`, `--output`: target folder to store the extracted files

Run any extractor command with `--help` to list all arguments.

### Scheduling and Push to Catalog

When moving out of trial, you will want to refresh your Metabase content in Catalog. Here is how to do it:

- Your source id from Catalog. In the code examples, this value is called `source_id`.
- Your Catalog Token from the Catalog team

The Catalog team will provide you with:

1. `Catalog Identifier`: An id we use to match your Metabase files with your Catalog instance
2. `Catalog Token`: An API token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: account id provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are `DBT`, `VIZ`, or `WAREHOUSE`

#### Target Files

To specify the target files, provide **one** of the following:

- `-f`, `--file_path`: to push a single file

or

- `-d`, `--directory_path`: to push several files at once

:::warning[Directory Contents]

The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will have to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

After you schedule the extractor runs and uploads, Metabase content stays current in Catalog.

## Troubleshooting

These checks cover both Catalog-managed Metabase onboarding and client-managed extractors.

### You Chose the API Path but Ingestion Stays Shallow on Dashboards

Metabase labels certain accounts as **superuser** for API access in extractor documentation. Without that privilege tier, popular metrics and some collections may not export.

**Resolution:**

1. Confirm the Metabase account tied to Catalog meets the API prerequisites in [Metabase package documentation][].
2. Retry ingestion after promoting the account or switching to an administrative service account permitted by your Metabase policy.

### Confusion Between Metabase Application PostgreSQL and Your Warehouse PostgreSQL

Catalog never asks for your warehouse credentials when you pick **Catalog managed** onboarding for Metabase. Client-managed extraction with `castor-extract-metabase-db` connects to the PostgreSQL database Metabase itself uses for app metadata, not your analytical warehouse.

**Resolution:**

1. Point `--host` and `--database` at Metabase's application database host from Metabase admin settings.
2. Keep read-only accounts limited to Metabase metadata tables only.

### Catalog-Managed Credentials Fail Immediately

Incorrect base URL format or blocked outbound traffic stops login before extraction begins.

**Resolution:**

1. Allowlist Catalog IPs from **Allowlist Catalog IP** on this page when Metabase sits behind a firewall.
2. Validate JSON keys such as `baseUrl`, `user`, and `password` against Metabase's HTTPS endpoint.

### Extractor Logs Stop Mid-Collection

Large Metabase instances sometimes hit API rate limits or disk limits on the runner.

**Resolution:**

1. Re-run with a larger output directory (`-o`) on the extractor host.
2. Split collections by scheduling incremental pushes instead of one oversized upload directory.

### Upload Succeeds but Dashboards Never Refresh

Schedulers often point to stale directories or wrong `source_id`.

**Resolution:**

1. Confirm `castor-upload` receives the current `source_id` from Catalog operations.
2. Verify cron jobs replace **all** JSON outputs between runs so partial uploads do not leave mixed timestamps.

---

[Catalog Managed]: #catalog-managed
[Client Managed]: #client-managed
[Metabase package documentation]: /docs/catalog/developer/castor-extractor/bi-tools/metabase
[Metabase configuration documentation]: https://www.metabase.com/docs/latest/operations-guide/configuring-application-database.html#postgres
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Mixpanel

Connect Mixpanel to Catalog to bring dashboards, reports, and event-oriented metadata into your Workspace. You can use Catalog-managed credentials or prepare uploads yourself using the BI Importer format. Catalog-managed Mixpanel sync is still maturing alongside client-side tooling, so this page focuses on what you can configure today and how Mixpanel objects translate into Catalog.

## Requirements

You need to be a Mixpanel Project owner or Admin to create the credentials Catalog uses for Mixpanel.

## Catalog Managed

Use this section when Catalog stores your Mixpanel credentials and runs scheduled sync for your Workspace.

:::info[Integration in progress]

Catalog-managed Mixpanel onboarding is still maturing. Treat this path as the supported way to hand credentials to Catalog for scheduled sync when your Workspace is enabled for it. If you need a fully self-run pipeline today, use [Client Managed][] with BI Importer instead.

:::

To connect Mixpanel with Catalog-managed credentials, supply a Mixpanel service account. Enter the following JSON in Catalog:

```json
{
    "username": "<service account username>",
    "password": "<service account password>"
}
```

For instructions on creating service accounts, see [Mixpanel service accounts documentation][].

You can enter credentials directly in Catalog.

For your first sync, allow up to 48 hours. Catalog notifies you when the initial sync completes.

If you prefer not to share credentials with Catalog operations, continue to [Client Managed][].

## Client Managed

### Doing a One-Time Extract

For a trial or one-time view of Mixpanel content in Catalog, extract or assemble Mixpanel metadata into [BI Importer expected format][], then upload it using the path your Catalog contact provides.

The Catalog extractor package published on PyPI includes dedicated command-line entry points for several BI platforms. There is not yet a published `castor-extract-mixpanel` command in that package. Until a Mixpanel-specific command ships, rely on BI Importer formatting and upload flows rather than a named extractor CLI. You can still use the shared [Google Colab][] notebook to experiment with extraction patterns for other supported tools while you shape Mixpanel files by hand or with your own scripts.

### Mixpanel to Catalog Translation

Use this mapping when you structure BI Importer files or interpret synced objects:

- **Event** - Catalog visualization model
- **Report** - Tile
- **Dashboard** - Dashboard

Additional Mixpanel objects surface in Catalog only when your payload or connector ingestion includes them.

### Scheduling and Upload After Trial

When you move beyond a one-time upload, Catalog typically provides a Catalog identifier and token so you can push files with `castor-upload` on a schedule you control. Align scheduling and upload steps with your Catalog contact so uploads stay aligned with your Mixpanel estate.

## Common Issues

- **Credentials rejected** - Confirm the service account is active, has access to the Mixpanel projects you expect, and matches the JSON field names Catalog expects for Mixpanel.
- **Sparse dashboards or reports** - Catalog reflects the metadata Mixpanel exposes for the account and projects you connected. Narrow project or workspace scope in Mixpanel can limit what appears in Catalog.

## What's Next?

- Browse related setup patterns on [BI Tools][].
- Review file layout and columns for uploads in [BI Importer][].

---

[BI Tools]: /docs/catalog/integrations/data-viz
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[BI Importer expected format]: /docs/catalog/developer/catalog-apis/bi-importer#file-expected-content
[Client Managed]: #client-managed
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[Mixpanel service accounts documentation]: https://developer.mixpanel.com/reference/service-accounts

---

## Mode

Connect Mode Analytics to Coalesce Catalog using Catalog-managed credentials or the client-managed extractor and upload flow.

## Requirements

:::warning[Warehouse Integration Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

To use Mode with Catalog, you need:

- Admin credentials on your Mode Analytics instance.
- An API token from Mode. Follow the [Mode API token instructions][].

## Catalog Managed

Send the API key details through [Safenote][] with Slack or email.

For your first sync, it can take up to 48 hours before ingestion completes. Catalog notifies you when it finishes.

If you prefer not to share credentials with Catalog, continue to [Client Managed](#client-managed).

## Client Managed

### Doing a One Shot Extract

For your trial, you can give us a one-shot view of your BI tools.

To get things working quickly, here is a [Google Colab notebook][] to run the package.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[mode]
```

For further details, see the [castor-extractor PyPI page][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-mode [arguments]
```

The script will run and display logs as following:

```bash
INFO - Authentication succeeded.
INFO - Starting extraction for DATASOURCE...
INFO - Calling https://modeanalytics.com/api/...
INFO - 1 rows extracted
INFO - Wrote output file: /tmp/catalog/1649080890-datasource.json
INFO - Starting extraction for COLLECTION...

...

INFO - Wrote output file: /tmp/catalog/1649080890-member.json
INFO - Wrote output file: /tmp/catalog/1649080890-summary.json
```

#### Credentials

- `-H`, `--host`: the host name or IP address where your Mode Analytics instance runs. If you use Mode's managed cloud, use `https://modeanalytics.com`.

- `-w`, `--workspace`: The name of your workspace. More information in the [Mode organizations documentation][]
- `-t`, `--token`: The `Token` value from the API token created in Mode.
- `-s`, `--secret`: The `Password` value from the API token created in Mode.

#### Optional Arguments

- `-o`, `--output`: target folder to store the extracted files

Run any extractor command with `--help` to print the full argument list.

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```text
CASTOR_MODE_ANALYTICS_HOST
CASTOR_MODE_ANALYTICS_WORKSPACE
CASTOR_MODE_ANALYTICS_TOKEN
CASTOR_MODE_ANALYTICS_SECRET

CASTOR_OUTPUT_DIRECTORY
```

Then the script can be executed without any arguments:

```bash
castor-extract-mode
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-mode --output /tmp/catalog
```

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Mode content in Catalog. Here is how to do it:

The Catalog team will provide you with

1. `Catalog Identifier` (an id for us to match your Mode files with your Catalog instance)
2. `Catalog Token` An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: account id provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are { `DBT` | `VIZ` | `WAREHOUSE` }

#### Target Files

To specify the target files, provide **one** of the following:

- `-f`, `--file_path`: to push a single file

 or

- `-d`, `--directory_path`: to push several files at once (\*)

:::warning[Directory Contents]

(\*) The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will need to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

## Troubleshooting

Use these steps when Mode ingestion or uploads fail before Mode assets appear in Catalog.

### Authentication Succeeds in Logs but Catalog Stays Empty

Large Mode organizations span multiple workspaces or collections. Extractors may finish without pushing files if upload arguments omit required identifiers.

**Resolution:**

1. Confirm `castor-upload` runs with the `source_id` and token Catalog operations issued for Mode.
2. Verify the upload directory contains the JSON outputs listed in extractor logs before each scheduled push.

### Wrong Host Value for Managed Mode Analytics

Managed tenants should target Mode's SaaS hostname while self-hosted installs need your private host.

**Resolution:**

1. Use `https://modeanalytics.com` when your contract references Mode's managed cloud.
2. Replace host names after DNS or TLS migrations so extractor scripts match the endpoint analysts use in browsers.

### API Token Rejected After Rotation

Mode invalidates older tokens when admins rotate secrets.

**Resolution:**

1. Generate a fresh token per [Mode API token instructions][].
2. Update Catalog-managed credentials or extractor CLI arguments the same day you revoke the prior token.

### Warehouse Prerequisite Missing

Mode lineage depends on warehouse integrations Catalog already ingested.

**Resolution:**

1. Finish configuring the Warehouse integration referenced under **Requirements** before expecting cross-tool lineage.
2. Re-run Mode extraction after warehouse assets finish indexing.

### Scheduled Jobs Overlap

Overlapping runs can leave half-written JSON files in the upload folder.

**Resolution:**

1. Add file locks or stagger schedules so `castor-upload` starts only after `castor-extract-mode` exits cleanly.
2. Clear stale artifacts before each upload when troubleshooting failures.

[Safenote]: https://safenote.co/
[Google Colab notebook]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install
[Mode organizations documentation]: https://mode.com/help/articles/organizations/
[Mode API token instructions]: https://mode.com/help/articles/api-reference/#generating-api-tokens

---

## Omni

Catalog offers a native Omni connector in beta to bring workbook, dashboard, and query metadata into your Workspace. Coverage and how Omni objects map into Catalog may change. Onboarding is coordinated with [Coalesce Support][].

## Requirements

:::warning[Warehouse Integration Required]

A warehouse integration must already be configured in Catalog before the first Omni ingestion completes.

:::

Before you contact support, confirm:

- You have access to an Omni organization API key, or ask your Omni admin to create one. See [Omni API authentication][] for how Omni issues API keys.
- You know your Omni instance base URL: the hostname you use to sign in, for example `https://yourorg.omniapp.co`.
- If Omni sits behind a firewall and Catalog runs extraction for you, your network team can allow outbound HTTPS from Catalog infrastructure to that host. See [Allowlist Catalog IP][] for fixed egress IPs.

## How Omni Objects Appear in Catalog

Catalog ingests three Omni asset types. Each maps to a Catalog asset type as follows:

| Omni object | Catalog asset type |
| ----------- | ------------------ |
| Workbook | Viz model |
| Query | Tile |
| Dashboard | Dashboard |

Workbook and dashboard names in Catalog match the Omni document name. Query names come from Omni as well.

## Supported Today

The following capabilities are available.

### Metadata

| Capability | Workbook | Query | Dashboard |
| ---------- | -------------------- | ------------ | --------- |
| Asset name | Yes, same as Omni document | Yes | Yes, same as Omni document |
| External description | No | Yes, for published queries | No |
| Tags from Omni | Yes, inherited from document | No | Yes |
| Link back to Omni | Yes, opens the workbook through the query URL | Yes | Yes |
| Editors | Yes, document owner | Yes, document owner | Yes, document owner |
| Field descriptions | Yes | Not applicable | Not applicable |

### Lineage

Catalog builds asset-level lineage, not column-level lineage, when your warehouse integration and Omni metadata allow it:

| Relationship | Supported |
| ------------ | --------- |
| Warehouse table, workbook, and query | Yes |
| Workbook, query, and dashboard | Yes |
| Warehouse table, query, and dashboard | Yes |
| Query and dashboard | Yes |
| Column- or field-level lineage | No |

### Behind the Scenes

Catalog also extracts Omni connections, view definitions from model YAML, and users to support lineage and ownership. Coalesce Support configures these paths during onboarding.

## Not Supported Today

The following are not available for Omni in Catalog today:

- Popularity metrics such as view counts
- Frequent users
- External descriptions on workbooks or dashboards
- Tags on queries
- Column- or field-level lineage
- Chrome extension metadata for Omni assets
- Sync back to Omni

Folder paths from Omni are not surfaced in Catalog today. Browse assets by type after onboarding.

## Get Started

Email [Coalesce Support][] and ask to connect Omni to Catalog. Include your Catalog Workspace name, Omni base URL, and whether you prefer Catalog-managed extraction or a client-managed upload path.

Coalesce Support will confirm prerequisites, validate that the connector fits your Omni usage, enable the integration on your Workspace, and walk you through credentials and timing for the first sync. Allow up to 48 hours for the initial ingestion after configuration is complete.

## Allowlist Catalog IP

When Omni sits behind a firewall and Catalog runs extraction for you, allow outbound HTTPS from Catalog infrastructure to your Omni host.

Catalog uses these fixed egress IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

Share your Catalog region and Omni base URL with Support so the allowlist matches your deployment.

## Catalog Managed

Coalesce Support configures Catalog-managed sync for your Workspace. Support enables the integration and applies credentials on your behalf.

When your Workspace is enabled, provide:

- **Base URL** - Your Omni instance root URL, the hostname you use to sign in.
- **Token** - Bearer token from an Omni organization API key.

Example credential shape:

```json
{
    "base_url": "https://yourorg.omniapp.co",
    "token": "*****"
}
```

Field names may vary. Use the names Support confirms for your Workspace.

Catalog calls Omni's `/api/v1/...` endpoints under that host. Do not add `/api` to the base URL unless that matches how you reach your deployment.

If you are not comfortable sharing credentials with Catalog operations, ask Support about [Client Managed][] instead.

## Client Managed

Use this path when your organization runs extraction and upload on your own schedule. Coalesce Support provides your Catalog `source_id` and upload token during onboarding.

### Doing a One-Shot Extract

For a trial, you can supply a one-time extract of your Omni BI assets. To run the package quickly, use this [Google Colab][] notebook.

### Running the Extraction Package

#### Install the PyPI Package

Install the Omni extractor from PyPI:

```bash
pip install castor-extractor[omni]
```

For further details, see the [castor-extractor installation instructions][].

#### Run the Package

After installation, run:

```bash
castor-extract-omni [arguments]
```

Example:

```bash
castor-extract-omni -u "https://yourorg.omniapp.co" -t "*****" -o "/output/directory"
```

#### Credentials

Pass your Omni host and API token with these flags:

- `-u`, `--base-url`: Omni instance root URL, the hostname you use to sign in.
- `-t`, `--token`: Bearer API token from an Omni organization API key.

#### Other Arguments

Set the output location with:

- `-o`, `--output`: Target folder for extracted JSON files.

Run `castor-extract-omni --help` to list all arguments.

### Scheduling and Push to Catalog

When you move out of trial, schedule extraction and upload so Omni content stays current in Catalog. You need:

- Your `source_id` from Catalog.
- Your Catalog token from the Catalog team.

Use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

`castor-upload` accepts these flags:

- `-k`, `--token`: Token provided by Catalog.
- `-s`, `--source_id`: Source id provided by Catalog.
- `-t`, `--file_type`: Source type to upload. Use `VIZ` for Omni visualization extracts.

#### Target Files

Provide exactly one of the following:

- `-f`, `--file_path`: Push a single file.
- `-d`, `--directory_path`: Push every file in a directory.

:::warning[Directory Contents]

The tool uploads all files in the given directory. Confirm the directory contains only Omni extract outputs before you push.

:::

Schedule the extractor run and upload on your preferred scheduler.

## API Limits and Scope

### API Access

Catalog only ingests objects and fields your API token can read. Restricted workbooks or documents stay out of sync until Omni grants the integration account access.

### Rate Limits

Omni enforces per-key API rate limits. Omni documents a limit of 60 requests per minute per API key. Heavy workspaces can return 429 Too Many Requests during sync. Retry after the limit window or reduce parallel activity against the same key.

For Omni API reference material, see [Omni REST APIs][] and [Omni API authentication][].

## Troubleshooting

### First Ingestion Fails with No Warehouse

See [Requirements][] for the warehouse prerequisite. Catalog needs that integration before the first Omni ingestion can complete.

**Resolution:**

1. Connect the warehouse Omni queries use, such as Snowflake, BigQuery, or Databricks.
2. Contact [Coalesce Support][] to re-run or wait for the scheduled Omni sync after the warehouse integration succeeds.

### Authentication or Connection Errors

Incorrect base URL format or an expired token stops extraction before metadata collection.

**Resolution:**

1. Use the sign-in hostname only, for example `https://yourorg.omniapp.co`, without appending `/api/v1` unless your deployment requires it.
2. Create a fresh organization API key in Omni and share the updated token with Support or update your extractor `-t` value.
3. For Catalog-managed sync behind a firewall, allowlist the IPs in [Allowlist Catalog IP][].

### 429 Too Many Requests

Omni returned a rate-limit response while Catalog or your extractor called the API.

**Resolution:**

1. Wait for the limit window to reset, then retry the sync.
2. Avoid running multiple large Omni extractions against the same API key at the same time.
3. For very large workspaces, prefer off-peak schedules for client-managed runs.

### Workbooks Appear as Viz Models

Omni workbook documents are stored in Catalog as viz models so you can document semantic workbook content separately from dashboard presentations.

**Resolution:**

1. Open the asset from search or lineage and confirm the Omni link opens the expected workbook in Omni.
2. Use dashboard assets for document presentations flagged as dashboards in Omni.

### Lineage Stops at Omni Without Warehouse Tables

View SQL lineage requires a matching warehouse integration and SQL that Catalog can parse in synced workbook views.

**Resolution:**

1. Confirm the warehouse that powers the Omni connection is integrated in Catalog.
2. Open the viz model or tile in Catalog and verify view SQL synced from Omni.
3. Contact [Coalesce Support][] to re-sync after you add or fix the warehouse integration.

## What's Next?

- Browse related BI setup on [BI Tools][].
- Install optional extractor extras in [Castor Extractor][].
- Review warehouse prerequisites on [Data Warehouses][].

---

[Allowlist Catalog IP]: #allowlist-catalog-ip
[Requirements]: #requirements
[BI Tools]: /docs/catalog/integrations/data-viz
[Castor Extractor]: /docs/catalog/developer/castor-extractor
[Client Managed]: #client-managed
[Coalesce Support]: mailto:support@coalesce.io
[Data Warehouses]: /docs/catalog/integrations/data-warehouses
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[Omni API authentication]: https://docs.omni.co/api/authentication
[Omni REST APIs]: https://docs.omni.co/docs/API
[app.castordoc.com]: https://app.castordoc.com/
[app.us.castordoc.com]: https://app.us.castordoc.com/
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Periscope

Connect Periscope Data to Catalog to sync dashboards, charts, and related BI metadata. Where this page says Periscope, it means Periscope Data. You can use Catalog-managed credentials or prepare uploads yourself using the BI Importer format. Catalog-managed Periscope sync is still maturing, so this page describes current credential shapes, client-managed options, and how Periscope objects map into Catalog.

## Requirements

You need a Periscope account that can sign in with the username and password Catalog will use for API or UI-backed extraction. Confirm with your Periscope administrator that programmatic or automated access is allowed for that account and that single sign-on policies do not block password-based login if Catalog relies on it.

If your organization routes Periscope through single sign-on only, work with Periscope support or your admin on a service-style account before onboarding Catalog.

## Catalog Managed

Use this section when Catalog stores your Periscope credentials and runs scheduled sync for your Workspace.

:::info[Integration in progress]

Catalog-managed Periscope onboarding is still maturing. Use this section when Catalog operations enable credential-based sync for your tenant. For a self-contained path you fully control today, use [Client Managed][] with BI Importer.

:::

To connect Periscope with Catalog-managed credentials, supply:

- Username
- Password

Enter credentials in Catalog using this JSON shape:

```json
{
    "username": "xxxx",
    "password": "xxxx"
}
```

Some deployments require a host or organization URL in addition to username and password. If Catalog prompts for extra fields or your sync fails with connection errors, share your Periscope base URL with your Catalog contact so they can confirm the expected JSON shape for your environment.

For your first sync, allow up to 48 hours. Catalog notifies you when the initial sync completes.

If you prefer not to share credentials with Catalog operations, continue to [Client Managed][].

## Client Managed

Use this section when you prepare Periscope metadata or uploads yourself, or when you use BI Importer instead of Catalog-managed sync.

### Doing a One-Time Extract

For a trial or one-time snapshot, assemble Periscope metadata into [BI Importer expected format][], then upload using the channel your Catalog contact provides.

The Catalog extractor package published on PyPI includes dedicated command-line entry points for several BI platforms. There is not yet a published `castor-extract-periscope` command in that package. Until a Periscope-specific command ships, rely on BI Importer formatting and upload flows. You can use the shared [Google Colab][] notebook to explore extraction patterns for other tools while you build Periscope files manually or with custom scripts.

### Periscope to Catalog Translation

Periscope dashboards and charts align with Catalog as follows:

- **Periscope dashboard** - Dashboard in Catalog
- **Periscope chart or widget** - Tile in Catalog
- **SQL snippets, views, and upstream dependencies** - May appear as visualization models in Catalog when that metadata is included in your payload or sync

Use Periscope-specific names for dashboards, charts, and SQL assets in your exports so Catalog labels match what analysts see in Periscope. If your organization merged Periscope with another BI platform, confirm naming with your Catalog contact before you expand the mapping list.

## Common Issues

If onboarding stalls or the metadata looks incomplete in Catalog, start with these checks:

- **Authentication failures** - Verify the account is allowed to access every Periscope site or space you expect Catalog to read.
- **Unexpected empty folders** - Confirm you exported or described every dashboard and dependency you need in the BI Importer files.

## What's Next?

- Browse related setup patterns on [BI Tools][].
- Review file layout and columns for uploads in [BI Importer][].

---

[BI Tools]: /docs/catalog/integrations/data-viz
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[BI Importer expected format]: /docs/catalog/developer/catalog-apis/bi-importer#file-expected-content
[Client Managed]: #client-managed
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing

---

## Power BI Dataflows in Catalog

Learn how Catalog uses Power BI Dataflow metadata to compute lineage through Dataflows, which mashup patterns resolve to warehouse tables, and how to validate and troubleshoot Dataflow-backed paths from the semantic data sets and reports that consume them.

## What Is a Power BI Dataflow?

A Power BI Dataflow is an optional data preparation layer in Power BI. Dataflows use Power Query M to connect to sources, shape data, and publish entities that data sets and other Dataflows can reuse. In many architectures, data moves from a warehouse into a Dataflow, then from the Dataflow into one or more data sets, then into reports and dashboards.

Microsoft's overview explains concepts and authoring: [Introduction to Dataflows][].

## Which Power BI Assets Connect to Warehouse Lineage

Use this section to see which kinds of Power BI content Catalog is built to trace deeply through semantic data sets and related metadata, and where expectations should be lower. Catalog ties BI assets to warehouse objects your integrations have already synced; the shape of Microsoft's APIs and the asset type both influence how complete that graph is.

| Asset | Role in lineage | What to expect |
| ----- | ---------------- | -------------- |
| **Semantic data sets** | Models that store imported or DirectQuery tables and relationships Power BI reports consume. | Primary path from warehouse tables into Power BI. Strong table-level and column-level lineage when warehouse objects exist in Catalog, admin APIs expose metadata, and models stay refreshed as described in [Power BI setup][]. |
| **Standard Power BI reports** | Interactive reports authored against semantic data sets you build in Power BI Desktop or the Power BI service. | Catalog treats these as the main report type for dependency graphs: upstream through the linked data set toward warehouse tables when that link exists in ingested metadata. |
| **Dashboards** | Collections of tiles and pinned visuals that reference underlying reports and visuals. | Lineage flows through those references into reports and data sets as Microsoft's metadata and Catalog ingestion allow. |
| **Power BI Dataflows** | Optional Power Query layers that publish entities other Dataflows and data sets reuse. | Catalog ingests Dataflow metadata during extraction to compute lineage only. Dataflows are not searchable semantic models, Catalog assets, or semantic representations in the UI. See [How Catalog Uses Dataflow Metadata for Lineage][] and [Supported Mashup Patterns and Limits][]. |
| **Paginated reports** | Print-oriented or operational layouts, often built in Report Builder and saved as `.rdl`, backed by a different reporting model than typical interactive Power BI reports. | Paginated reports use a different reporting stack; the metadata available for them does not support the same table-level and column-level resolution. Field-level detail in Catalog is usually lighter as well. For regulatory or document-style outputs, validate those assets directly rather than assuming the same depth you get when you trace interactive reports through data sets and Dataflows to the warehouse. |

When you're validating lineage for a specific asset, confirm whether it is an interactive report on a semantic data set or a paginated report. Mixed expectations across those types are a common reason two teams see different Catalog depth for “reports” in Power BI.

:::tip[Paginated URLs and Catalog ingestion]

Some product surfaces can recognize Power BI links whose paths include paginated report locations. That only means the URL is parsed as a Power BI resource. It does not change how Catalog ingests metadata or builds lineage for paginated reports relative to standard reports.

:::

For credentials, admin APIs, extraction schedules, and environment-specific behavior, use [Power BI setup][] as the setup reference alongside this Dataflows guide.

## How Dataflows Relate to Lineage in Catalog

Catalog maintains a metadata graph that links BI assets to warehouse objects your integrations have already synced into Catalog. When a data set table loads from a Dataflow, Catalog must resolve that intermediate Dataflow layer internally before it can link the data set to warehouse tables. Dataflows are not published as Catalog assets or semantic representations, so you will not see a Dataflow node in the lineage graph. If Catalog cannot resolve the Dataflow layer from ingested metadata and pattern-matched mashup text, lineage can stop at the data set even when warehouse sync is healthy.

Catalog reads Dataflow metadata during Power BI extraction so table-level and column-level lineage can resolve through the Dataflow layer when mashup text matches supported patterns and upstream warehouse objects exist in Catalog. You don't turn on a separate Dataflows option in Catalog. The same Power BI credentials, **Admin API Settings**, and extraction runs cover Dataflow metadata when your tenant and models expose it.

Catalog does not publish Dataflows as semantic models or visualization models you can open in **Dashboards** or [Catalog search][]. Validate Dataflow-backed lineage from the **semantic data set** or **report** that loads from the Dataflow.

Lineage quality still depends on both sides of the graph:

- **Power BI** - Admin settings, refresh or republish behavior, and what the Power BI APIs return for mashup and related metadata. See [Power BI setup][] for details.
- **Warehouse and other sources** - Tables and columns must be present in Catalog through your warehouse integration or another integration. If Catalog doesn't know about a database or connection your Dataflow uses, lineage can be incomplete even when Power BI extraction succeeds.

The following sections explain how Catalog uses that Dataflow layer during lineage computation, how to inspect results in the Catalog UI, what to expect for sync cadence, and which mashup patterns Catalog can match.

## How Catalog Uses Dataflow Metadata for Lineage

During Power BI extraction, Catalog ingests Dataflow mashup metadata from Microsoft's admin APIs and uses it internally to compute lineage. That metadata supports links between warehouse tables and the **semantic data sets** and **reports** Catalog publishes. Dataflows themselves are not represented as semantic models or searchable visualization models in the Catalog UI.

When a semantic data set table loads from a Dataflow, its mashup text often uses the Power BI Dataflows or Power Platform Dataflows connectors. Catalog does not parse Power Query M. It applies pattern matching and heuristics to mashup strings and related admin API output for connector shapes it tests, such as `PowerBI.Dataflows`, `Value.NativeQuery`, connector navigation records, and direct `Table.RenameColumns` mappings. When those patterns align, Catalog can infer a **Dataflow ID** and **entity** name and resolve warehouse table paths through the Dataflow layer when linked-entity patterns and upstream objects are available.

Catalog resolves the Dataflow layer internally when it builds lineage links you see in the UI: successful resolution typically shows **warehouse tables** connected to **semantic data sets** and **reports**, not a separate searchable Dataflow asset.

Column-level lineage uses the same resolution path. It can be weaker than table-level lineage when an entity draws from several warehouse sources in one logical table, when mashup text is heavily parameterized or indirect, or when field mappings are ambiguous. Some rename patterns in mashup text resolve more reliably than others; see [Troubleshoot Power BI Lineage When Columns Are Renamed][] for detail.

For measures you define in DAX, field lineage in Catalog shows relationships in the graph when paths resolve. Read or edit the full measure formula in Power BI Desktop or your standard authoring tools. Catalog field lineage and lineage detail panels focus on the dependency graph rather than reproducing that formula text. For measure expectations and troubleshooting, see Measures and DAX in [Troubleshoot Power BI Lineage When Columns Are Renamed][] and [Missing DAX measure definitions in Catalog][] in [Power BI troubleshooting in Catalog][].

## Lineage in the Catalog UI

Catalog shows lineage between reports, semantic data sets, and warehouse tables in the same lineage graph as other assets. When Dataflow-backed paths resolve, you typically see warehouse objects linked to the **semantic data set** or **report** that consumes the Dataflow, not a separate Dataflow page in Catalog.

Use this sequence when you want to inspect Dataflow-backed lineage:

1. Open the **semantic data set** or **report** that loads from the Dataflow in Catalog from **Dashboards** in the left navigation or from [Catalog search][]. Then select the **Lineage** tab on that asset.

    <figure><figcaption>On a report or dashboard, Lineage shows preview cards for Dashboard Lineage and Field Lineage before you open the full graph.</figcaption></figure>

2. On a report or dashboard, choose **Open Dashboard Lineage** to inspect object-level dependencies in the lineage canvas. On a semantic data set, the canvas opens from the same **Lineage** tab without those preview cards. Use **`+`** on the left to expand upstream sources and on the right to expand downstream usage, as described in [Lineage][]. When mashup patterns resolve, upstream warehouse tables appear behind the semantic data set or report.

    <figure><figcaption>Dashboard lineage with the report focused; upstream nodes include the semantic data set and warehouse objects.</figcaption></figure>

3. Select a warehouse table or view in the graph, or open that warehouse asset and its lineage, when you want to confirm how Catalog links warehouse objects into Power BI. Downstream nodes show which semantic data sets and reports consume the warehouse object.

    <figure><figcaption>Warehouse view focused with downstream semantic data set and report.</figcaption></figure>

4. For column-level paths, return to the asset **Lineage** tab and choose **Open Field Lineage**, or from a warehouse column use **Go To Column Lineage** when that control is available. Field lineage depends on how clearly mashup text maps fields. Dataflow-backed paths show here when table-level links support column resolution.

    <figure><figcaption>Field lineage for one column from the warehouse through the semantic data set into the report.</figcaption></figure>

Every time your sources sync into Catalog, lineage is recomputed. The lineage graph uses data from the last 30 days; if you expect a recent change, adjust the time range under the graph if links look stale.

## Supported Mashup Patterns and Limits

Catalog anchors Dataflow support on mashup text the Power BI admin APIs return and on pattern matching and heuristics against connector shapes Catalog tests. Catalog does not parse Power Query M. Treat the following as the supported surface for data set to Dataflow references:

- **Recognized connector entry points** - Mashup that navigates through **`PowerBI.Dataflows`** or **`PowerPlatform.Dataflows`** with recognizable **Dataflow ID** and **entity** values in the text.
- **Composite key** - Catalog ties a reference to a flow and entity when both identifiers appear in a supported structure. If either is missing or the mashup shape does not match a tested pattern, lineage through that Dataflow does not resolve until the model's mashup matches a supported shape and Power BI exposes matching admin metadata.

Linked entities inside a Dataflow, where one entity is built on another, are part of supported modeling when mashup patterns expose those internal links during lineage computation.

Plan for the following limits:

- **Limited pattern coverage** - Arbitrary, parameterized, or highly dynamic mashup logic can stay unresolved even when the model is valid in Power BI.
- **Table navigation versus embedded SQL** - Semantic data sets built through the visual table picker in Power Query often export less explicit SQL than models where you embed SQL with `Value.NativeQuery`. Warehouse lineage is strongest on the embedded SQL path. Table selection is sometimes required, for example for incremental refresh. See [Lineage When Semantic Models Use Table Navigation Versus SQL][] in [Power BI troubleshooting in Catalog][].
- **Ambiguous or multi-source entities** - When one entity resolves to multiple warehouse table paths in a way Catalog cannot reduce to a single path, column-level lineage can be incomplete even when table-level links exist.
- **Dynamic or complex mashup** - Parameters, indirection, or unusual connector shapes can weaken lineage until mashup text aligns with what extraction returns.
- **Warehouse gaps** - If the warehouse tables your Dataflow reads are not in Catalog, the graph stops where Catalog has no upstream object, regardless of Power BI extraction.

Fabric Dataflow behavior follows the same Power BI integration, admin API output, and mashup patterns described in [Supported Mashup Patterns and Limits][]. Use those references when you design validations for Fabric-backed flows.

## Performance and Freshness

Use this section to set expectations for how often Dataflow-backed metadata and lineage update in Catalog. Treat extraction duration, Microsoft API behavior, and lineage recomputation timing as driven by your integration schedules and successful sync outcomes rather than by fixed timing promises in this documentation.

For the first load, Catalog-managed Power BI ingestion can take up to 48 hours for the first sync, as described in [Power BI setup][]. Treat any semantic data set or report as potentially missing upstream lineage until that first pass completes successfully.

For ongoing Power BI metadata, after the first sync, Catalog-managed environments follow the schedule you coordinate with Catalog operations. Client-managed environments follow the schedule you configure for `castor-extract-powerbi` and upload, as in [Power BI setup][]. On the client-managed path after your trial, schedule extraction at your desired frequency, up to once per day, so Dashboard sections stay current, as described in [Data visualization integrations][].

For warehouse metadata, warehouse integrations typically sync once per day after the first sync, as described in the [Catalog onboarding guide][]. Lineage through a Dataflow still requires warehouse tables and columns to exist in Catalog, so warehouse freshness and Power BI freshness both matter.

When models or admin settings change, use Power BI to refresh or republish affected data sets, then allow a full Catalog extraction cycle before you judge lineage. Power BI sometimes serves updated mashup and admin metadata shortly after your change, but Catalog only reflects it after the next successful extraction.

## How to Validate Dataflow Lineage

Follow these steps when you want to confirm that lineage crosses a Dataflow path. Start from the **semantic data set** or **report** that consumes the Dataflow, not the Dataflow name in Catalog search.

1. **Confirm Power BI admin settings**  
   In the [Power BI Admin portal][], verify the same **Admin API Settings** called out in [Power BI setup][] remain enabled for your Catalog service principal's security group.

2. **Refresh or republish affected data sets**  
   Follow refresh and republish guidance for data sets in [Power BI setup][] so lineage-related metadata is current in Power BI before the next Catalog extraction.

3. **Confirm the warehouse side is in Catalog**  
   Open the warehouse integration documentation for your platform, such as [Snowflake][], and ensure the databases and objects your Dataflow reads are in scope for sync. If a data set still shows no upstream tables, add or extend the warehouse source that backs those tables, then let Catalog run another sync.

4. **Wait for the next scheduled extraction**  
   Catalog-managed environments run on a schedule you coordinate with Catalog operations. Client-managed environments use your own schedule for `castor-extract-powerbi` and upload. See [Power BI setup][].

If lineage for Dataflow-backed tables was empty in the past and prerequisites are fixed now, run through the same steps again and allow a full extraction cycle before you contact [Coalesce Support][].

## Troubleshooting

For incomplete Dataflow-backed lineage, stale metadata after Admin API changes, unrecognized mashup patterns, and client-managed extraction issues, use [Power BI troubleshooting in Catalog][]. That guide collects step-by-step checks in one place so this article stays focused on how Dataflows affect lineage computation in Catalog.

## What's Next?

- Complete or review Power BI credentials and admin steps in [Power BI setup][].
- Resolve lineage, search, and extraction problems in [Power BI troubleshooting in Catalog][].
- Fix or understand field lineage after column renames in [Troubleshoot Power BI Lineage When Columns Are Renamed][].
- Review quick and advanced search behavior for visualization models in [Catalog search][].
- Connect and sync the warehouses your Dataflows read in [Data warehouse integrations][].
- Practice expanding upstream and downstream lineage in [Lineage][].

[Catalog onboarding guide]: /docs/catalog/catalog-onboarding-guide
[Catalog search]: /docs/catalog/navigate/search
[Data visualization integrations]: /docs/catalog/integrations/data-viz
[Data warehouse integrations]: /docs/catalog/integrations/data-warehouses
[Introduction to Dataflows]: https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-introduction-self-service
[Lineage When Semantic Models Use Table Navigation Versus SQL]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#lineage-when-semantic-models-use-table-navigation-versus-sql
[Lineage]: /docs/catalog/navigate/lineage
[Missing DAX measure definitions in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#missing-dax-measure-definitions-in-catalog
[Coalesce Support]: mailto:support@coalesce.io
[Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi
[Power BI Admin portal]: https://app.fabric.microsoft.com/admin-portal/tenantSettings
[Power BI troubleshooting in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting
[Snowflake]: /docs/catalog/integrations/data-warehouses/snowflake
[How Catalog Uses Dataflow Metadata for Lineage]: #how-catalog-uses-dataflow-metadata-for-lineage
[Supported Mashup Patterns and Limits]: #supported-mashup-patterns-and-limits
[Troubleshoot Power BI Lineage When Columns Are Renamed]: /docs/catalog/integrations/data-viz/power-bi/power-bi-lineage-column-rename

---

## Troubleshoot Power BI Lineage When Columns Are Renamed

Use this guide when column or field lineage in Catalog looks incorrect after you rename columns in Power BI Power Query. You'll learn typical symptoms, why renames affect lineage, which rename patterns Catalog resolves best, how to validate integrations and timing when something still looks off, and where to look for issues unrelated to column renames, such as measure-level detail.

## Symptoms You Might See

You might notice one or more of these behaviors:

- **Field lineage stops after a rename step** - Table-level lineage still shows upstream warehouse tables, but field lineage doesn't continue through the rename.
- **Some columns trace end to end** - Others stop at the semantic data set after heavy merging, splitting, or renaming.
- **Lineage was incomplete before you changed the model** - You simplified rename steps and want to know what to expect after Catalog finishes another extraction cycle.

Lineage is computed using recent activity. Confirm that the asset has been refreshed during the period Catalog uses for lineage, as described in [Lineage troubleshooting][].

## Why Renames Affect Lineage

Catalog links Power BI fields to upstream warehouse columns by pattern matching exported mashup text and related admin API output against the warehouse graph. Catalog does not parse Power Query M. When you rename a column, the field name in the model can differ from the name the warehouse integration ingested. Catalog needs a recognizable mapping in mashup text and metadata to connect the new name back to the source column.

When that mapping is explicit and stable, column-level lineage can run end to end. When mashup builds rename logic indirectly or at runtime, the same logical rename can be harder to trace automatically, so column-level lineage can stay partial even when table-level links exist.

## What Catalog Resolves for Renames

Catalog resolves column-level lineage more reliably when exported mashup matches tested patterns for direct `Table.RenameColumns` usage in Power Query, including a single mapping and multiple rename pairs in one step. In practice, that means the rename list is visible in mashup as a static table of old and new names Catalog can match, not only the outcome after arbitrary logic runs. When generated mashup shows plain old-name and new-name pairs for those steps, lineage mapping back to warehouse columns is strongest.

If you use other ways to relabel fields, for example by only changing display names without a clear rename step in mashup, or by using rename shapes that pattern matching does not treat as direct `Table.RenameColumns` mappings, field lineage can remain weaker until the model uses a pattern Catalog can match.

## Dynamic or Indirect Rename Logic

Some models build the rename list at query time, for example by zipping two lists, reading from parameters, or computing which columns to rename in a step that doesn't leave a fixed pair list in the same form as a direct `Table.RenameColumns` call. In those cases, column-level lineage can stay incomplete because the relationship between old and new names isn't expressed the same way in the metadata Catalog ingests.

When you can change the model, use this sequence:

1. **Simplify to explicit renames** for the fields you need in lineage, using `Table.RenameColumns` with a direct mapping in mashup where possible.
2. **Reduce indirection** in the same query chain where Catalog must tie fields to warehouse columns.
3. After changes, follow **How to Validate Lineage After Changing Renames** in this guide.

Custom wrapper functions or tenant-specific mashup helpers that hide `Table.RenameColumns` inside abstractions can fall outside the mashup patterns Catalog recognizes. If lineage is still wrong after admin settings, refresh, and a full extraction cycle, contact [Coalesce Support][] with a short description of the mashup pattern. Do not include credentials.

## How to Validate Lineage After Changing Renames

Work through these steps in order when you have fixed or simplified rename steps and want Catalog to show updated field lineage.

1. **Confirm Power BI admin settings**  
   In the [Power BI Admin portal][], keep the same **Admin API Settings** that [Power BI setup][] requires for your Catalog service principal, including options that surface detailed metadata and mashup expressions when your organization uses them for lineage.

2. **Refresh or republish affected data sets**  
   Refresh or republish in Power BI so the service returns updated mashup text and metadata that match your changes. This matters for data sets that refresh rarely or that use DirectQuery, as described in [Power BI setup][].

3. **Confirm warehouse tables and columns are in Catalog**  
   The warehouse objects referenced in mashup must be in scope for your warehouse integration. If Catalog doesn't have a table or column, field lineage can't attach to it. Use your warehouse integration documentation, for example [Snowflake][], to verify scope and sync.

4. **Allow a full Catalog extraction cycle**  
   Catalog only reflects updated mashup text after the next successful Power BI extraction. For Catalog-managed environments, that follows the schedule you coordinate with Catalog operations. For client-managed environments, that follows your `castor-extract-powerbi` schedule and upload, as in [Power BI setup][]. The first Power BI sync can take up to 48 hours. After that, wait for at least one successful extraction on your schedule before you judge field lineage.

5. **Re-check lineage after both sides have synced**  
   If table-level lineage looks right but some fields still don't trace, compare those columns to the rename patterns in this guide. If everything matches supported patterns and extraction succeeded, contact [Coalesce Support][] with timestamps and asset names.

After configuration is correct and extraction runs successfully, you can see more column links than before for the same model, especially where you replaced ambiguous rename logic with direct `Table.RenameColumns` mappings. Large tenants still follow the same timing expectations as other Power BI ingestion.

## When Lineage Stays Partial for Other Reasons

Column renaming is one reason field lineage can stop early. These situations overlap with rename issues in the UI:

- **Multiple warehouse sources in one logical table** - Catalog can't always reduce paths to a single resolved column.
- **Heavy merge, join, or append steps** - These steps can obscure which input column feeds an output field.
- **Parameterized or highly dynamic connection logic** - Extraction may infer less. See [Power BI Dataflows in Catalog][] and [Power BI troubleshooting in Catalog][] for broader troubleshooting steps.

Table-level lineage can be present while column-level lineage ends at an ambiguous transform. That is expected until mashup text and metadata give a clear enough mapping.

## Measures and DAX

Column rename behavior in Power Query M is separate from how Catalog treats DAX measures in lineage. Field lineage shows how warehouse columns, model fields, and measures connect in the graph when paths resolve. It does not reproduce the full DAX measure definition from Power BI Desktop inside lineage or asset detail views.

That gap is expected when admin API settings are correct and extraction succeeds. For a comparison table, setup requirements, and a symptom-based FAQ, see [DAX, Mashup Expressions, and Field Lineage in Catalog][] on [Power BI setup][] and [Missing DAX measure definitions in Catalog][] in [Power BI troubleshooting in Catalog][].

## What's Next?

- Complete or review credentials and **Admin API Settings** in [Power BI setup][].
- Read broader Power BI Dataflow patterns and limits in [Power BI Dataflows in Catalog][] and step through Power BI issues in [Power BI troubleshooting in Catalog][].
- Use the general lineage guide for timeouts, temporary tables, and other topics in [Lineage troubleshooting][].
- Practice expanding upstream and downstream graphs in [Lineage][].

[DAX, Mashup Expressions, and Field Lineage in Catalog]: /docs/catalog/integrations/data-viz/power-bi/powerbi#dax-mashup-expressions-and-field-lineage-in-catalog
[Coalesce Support]: mailto:support@coalesce.io
[Lineage]: /docs/catalog/navigate/lineage
[Lineage troubleshooting]: /docs/catalog/navigate/lineage/lineage-troubleshooting
[Power BI Admin portal]: https://app.fabric.microsoft.com/admin-portal/tenantSettings
[Power BI Dataflows in Catalog]: /docs/catalog/integrations/data-viz/power-bi
[Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi
[Power BI troubleshooting in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting
[Snowflake]: /docs/catalog/integrations/data-warehouses/snowflake
[Missing DAX measure definitions in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#missing-dax-measure-definitions-in-catalog

---

## Power BI Troubleshooting in Catalog

Use this guide when lineage through Power BI Dataflows looks incomplete or stale, when you expected to find a Dataflow as a Catalog asset, when metadata in Catalog still looks wrong after you fix tenant settings, when credentials expire, or when Power BI ingestion never completes.

Work through the sections that match your situation.

## Ingestion Fails or Metadata Never Appears

Use this section when a Catalog-managed Power BI integration stays empty after the first sync window, client-managed uploads do not refresh Catalog, or your Catalog contact reports that ingestion failed because pull extraction is not supported for Power BI.

Native Power BI integrations expect the setup in [Power BI setup][]. Catalog-managed extraction uses Entra credentials in **Settings > Integrations**. Client-managed extraction uses `castor-extract-powerbi` and `castor-upload`. See [Ingestion architecture][] on the setup page for how those paths differ from [BI Importer][].

### Symptoms

You may see one or more of the following:

- A Catalog-managed integration shows no dashboards or data sets after 48 hours or longer, even when tenant admin settings match [Power BI setup][].
- `castor-extract-powerbi` finishes locally but Catalog stays stale after upload.
- Support or your Catalog team confirms ingestion failed with an error that pull extraction is not supported for Power BI.
- Ingestion errors reference an expired client secret. See [Integration Paused After Expired Credentials][].

### Likely Cause

The Power BI source is configured on Catalog's generic BI Importer ingestion path instead of the native Power BI integration path. That mismatch often follows an earlier trial upload, a custom MVP integration, or a source that was never moved to standard Entra-based setup.

When credentials are the issue instead, the Entra client secret expired or the JSON in Catalog does not match the active secret in Azure.

### What To Do

Work through these steps in order:

1. Confirm you follow [Power BI setup][] for Catalog-managed or client-managed extraction, not BI Importer CSV templates unless your Catalog team explicitly scoped a custom BI Importer integration.
2. For Catalog-managed sources, verify credentials in **Settings > Integrations** and allow up to 48 hours for the first cycle after a clean configuration. If errors mention an expired secret, follow [Integration Paused After Expired Credentials][].
3. For client-managed sources, confirm `castor-upload` uses the Source Id and token Catalog provided, that upload files match a recent `castor-extract-powerbi` run, and that your scheduler runs extract before upload.
4. If ingestion still fails immediately or your Catalog team confirms the generic path error, contact [Coalesce Support][] with your integration name and the approximate time of the failed ingestion run.

## Integration Paused After Expired Credentials

Use this section when Catalog-managed Power BI ingestion stopped after an Entra client secret expired, or when Support notified you that the integration was paused until credentials are updated.

You may see an email or Support message about an expired client secret, a paused Power BI source under **Settings > Integrations**, or search and lineage that stop updating for Power BI assets while warehouse sync continues.

Work through these steps in order:

1. In the [Azure portal][], open the Catalog app registration and create a new client secret under **Manage** > **Certificates & secrets**. Copy the new **Value** before you leave the page.
2. In Catalog, go to **Settings > Integrations**, open the Power BI source, and choose **Edit credentials**. Update the JSON with the current `clientId`, `secret`, and `tenantId`.
3. Reply to Support or contact [Coalesce Support][] if the integration remains paused after you save valid credentials. Catalog operations may need to re-enable the source and run ingestion again.
4. Allow one full extraction cycle before you judge search or lineage freshness.

For proactive renewal cadence and the first-sync rotation warning, see [Credential lifecycle][] on [Power BI setup][].

## Missing DAX Measure Definitions in Catalog

Use this section when field lineage shows column and measure relationships but not the complete DAX measure definition you see in Power BI Desktop.

Typical signs include field lineage that lists measure names and upstream columns but not the full DAX text from the formula bar, or asset detail and search that surface names and descriptions without the measure definition body. Admin API settings for DAX and mashup expressions can be enabled and extraction can succeed while lineage panels still omit formula text.

Catalog field lineage is built around dependency relationships in the metadata graph. Enabling **Enhance admin APIs responses with DAX and mashup expressions** is required for Power BI integration and helps lineage computation, but Catalog does not reproduce the full measure editor experience inside lineage or asset detail views. That behavior is expected when extraction is healthy. It is not the same signal as a broken integration.

To validate and work around the gap:

1. Confirm extraction completed successfully after your last model change.
2. Use **Open Field Lineage** on the semantic data set or report when you need to inspect how columns connect into measures in the graph.
3. For the full DAX definition, open the measure in Power BI Desktop or your standard authoring tools.
4. For Power Query rename behavior that affects field paths, see [Troubleshoot Power BI Lineage When Columns Are Renamed][].

For the comparison table across Power BI Desktop, field lineage, and asset detail, see [DAX, Mashup Expressions, and Field Lineage in Catalog][] on [Power BI setup][].

## Lineage or Metadata Still Looks Wrong After Admin API Changes

When **Admin API Settings** are correct but graphs or upstream paths still look empty or old, the gap is usually timing or scope rather than the toggle itself.

1. **Refresh or republish** the affected semantic data sets in Power BI so mashup and related metadata match what you expect in the service.
2. **Wait for Power BI** to show consistent metadata for those assets in the admin experience you use outside Catalog, then plan for one full Catalog extraction afterward.
3. **Confirm warehouse objects** for the same logical environment, for example DEV compared to prod, are actually in Catalog and match what the Dataflow queries.
4. **Re-run validation** only after both warehouse sync and Power BI extraction have completed successfully for that cycle.

## Lineage When Semantic Models Use Table Navigation Versus SQL

Use this section when a semantic data set appears in Catalog but shows no upstream warehouse tables, or when lineage works for one model but not another that points at the same Databricks tables.

Catalog derives Power BI to warehouse links by pattern matching exported mashup text and related metadata from Microsoft's admin APIs against tested connector and transform shapes. Catalog does not parse Power Query M. Lineage is strongest when mashup exposes an explicit SQL statement, for example through `Value.NativeQuery`, that Catalog can match to tables already synced from your warehouse integration.

### How Authoring Mode Affects Lineage

| Authoring approach in Power BI | Typical exported mashup shape | Lineage in Catalog |
| ------------------------------ | ----------------- | ------------------ |
| **Embedded or pasted SQL** in Power Query | `Value.NativeQuery` with a SQL string that references catalog, schema, and table names | Strongest path to Databricks, Snowflake, and other warehouse tables when those objects exist in Catalog |
| **Visual table navigation**, browse and select tables in the connector UI | Connector navigation steps without an explicit SQL string in exported mashup | Weaker or missing warehouse links even when the model is valid in Power BI |
| **DirectQuery or import with mixed steps** | Combination of navigation and transforms | Depends on whether exported mashup still contains explicit SQL or table paths Catalog can match |

Embedded SQL is not always practical. Some models need visual table selection, for example when you configure incremental refresh in Power BI. In those cases, warehouse lineage may be thinner or missing until metadata exposes a path Catalog can resolve.

### Validation Checklist

Work through these steps before you open a Support ticket:

1. **Confirm the warehouse side is in Catalog**  
   Open the integration for your warehouse platform, for example [Databricks][], and verify the catalog, schema, and tables the semantic model reads are in scope and synced. Lineage only connects to objects Catalog already knows about.

2. **Confirm Power BI admin settings**  
   Keep the **Admin API Settings** from [Power BI setup][] enabled for your Catalog service principal's security group, including detailed metadata and mashup expressions when your organization uses them for lineage.

3. **Refresh or republish the semantic data set**  
   Update the model in Power BI so mashup metadata matches what you expect in the service, then wait for one full Catalog extraction cycle.

4. **Compare authoring modes on a test model**  
   When policy allows, republish a copy of the model with embedded SQL that references the same warehouse tables. If lineage appears for the SQL version but not the table-navigation version, the gap is tied to exported mashup text rather than warehouse sync or credentials.

5. **Allow extraction to finish**  
   For Catalog-managed Power BI, the first sync can take up to 48 hours. After model changes, wait for at least one successful extraction before you judge lineage.

### When To Contact Support

If the checklist passes and warehouse lineage is still missing for a model that uses table navigation, contact [Coalesce Support][] with the semantic data set name, the warehouse platform, and screenshots of the Power Query steps in Power BI Desktop. Do not include credentials or connection secrets.

For a short summary on the setup page, see [Troubleshooting in Power BI setup][].

## No Upstream Lineage From a Data Set That Uses Dataflows

Check these common causes when a Dataflow-backed data set shows no upstream objects in Catalog.

- **Admin API metadata is off or scoped wrong** - If DAX and mashup expressions or detailed metadata are disabled or the security group for your Catalog service principal is missing, Catalog doesn't receive enough information to resolve Dataflow-backed tables. Fix the settings, then refresh data sets and wait for the next extraction.

- **Warehouse not in Catalog** - Lineage only connects assets Catalog already knows about. Add or expand the warehouse integration for the database your Dataflow queries, then sync again.

- **Stale Power BI metadata** - When admin settings are already correct, follow the refresh, wait, and extraction steps in [Lineage or Metadata Still Looks Wrong After Admin API Changes][] for the affected data sets.

- **Mashup pattern not matched** - If the data set's mashup does not expose both Dataflow ID and entity in a supported Power BI Dataflows or Power Platform Dataflows pattern, Catalog does not resolve lineage through that Dataflow. Compare the model's mashup to the supported patterns in [Power BI Dataflows in Catalog][], republish after changes, and wait for extraction.

For a step-by-step validation flow after you fix prerequisites, see [How to Validate Dataflow Lineage][] on [Power BI Dataflows in Catalog][].

## Lineage Is Partial or Stops at Some Columns

Column-level lineage depends on how clearly mashup text maps fields to upstream columns. Heavy merging or indirection inside the Dataflow or data set can limit column-level links even when table-level lineage appears. For column renames and `Table.RenameColumns` behavior in Power Query, see [Troubleshoot Power BI Lineage When Columns Are Renamed][]. When one entity effectively draws from multiple warehouse sources in parallel, column-level lineage can stop where Catalog cannot pick a single resolved path.

## Parameterized or Dynamic Mashup Logic

Organizations often use M parameters for environment or database names. Resolution depends on what Catalog can match in exported mashup text and metadata. If you use complex parameter wiring or highly dynamic connection logic and lineage stays incomplete after you try the other guidance in this article, contact [Coalesce Support][] with a short description of the pattern. Do not include credentials.

## Client-Managed Extraction

If you run `castor-extract-powerbi` yourself, compare your log output and output files with a recent successful run on the current extractor. If extraction completes but lineage doesn't improve, gather the run timestamp and contact [Coalesce Support][] so the team can trace ingestion with you.

## When You Cannot Tell Which Workspace an Asset Belongs To

Depending on how names are shown in lineage, you might not see the Power BI workspace at every step in the graph. When many workspaces share similar content, use consistent data set and report naming that includes the environment so you can tell assets apart.

## Dataflows Do Not Appear as Catalog Assets

Power BI Dataflows are not published as semantic models, searchable visualization models, or semantic representations in Catalog. That is expected. Catalog ingests Dataflow metadata during extraction and uses it only to compute lineage for **semantic data sets** and **reports** that load from Dataflows.

If you search for a Dataflow name under **Dashboards** or in [Catalog search][] and find no match, validate lineage from the consuming asset instead:

1. **Open the semantic data set or report** that loads from the Dataflow in Power BI, then find that same asset in Catalog under **Dashboards** or [Catalog search][].
2. **Select the Lineage tab** on that asset and expand upstream warehouse tables, as described in [Lineage in the Catalog UI][] on [Power BI Dataflows in Catalog][].
3. **Confirm the Power BI integration is healthy** in **Settings > Integrations**. A paused integration or expired secret stops lineage updates until you fix the connection. See [Integration Paused After Expired Credentials][].
4. **Allow a full extraction cycle** after admin setting or model changes. For Catalog-managed Power BI, the first ingestion can take up to 48 hours, as noted in [Ingestion Fails or Metadata Never Appears][].
5. **Work through Dataflow-backed lineage checks** in [No Upstream Lineage From a Data Set That Uses Dataflows][] and [How to Validate Dataflow Lineage][] when upstream tables are still missing.

## Users See Power BI Workspaces They Should Not

When a new user sees every Power BI workspace in Catalog, or a new workspace appears with company-wide visibility after extraction, the cause is usually default-open dashboard folder access rather than a sync or credentials problem.

Work through [Power BI workspaces and Catalog visibility][], then the Power BI items under [Assets Access Control troubleshooting][]: [New User Sees All Power BI Workspaces][] and [New Power BI Workspace Appears With Broad Access][]. If the workspace should not appear in Catalog at all, see [Scope Power BI ingestion][]. Confirm the behavior with a non-admin account in the same teams as the affected user.

## Microsoft Teams and Search Deep Links for Power BI

Use this section when users open Catalog Power BI results from Microsoft Teams, Microsoft Search, or the Catalog AI Assistant and land on a workspace report or dashboard in Power BI instead of your organization's published Power BI App.

Typical signs include one or more of the following:

- Federated search or Teams results link to a Power BI workspace report or tile dashboard URL.
- Users expected the published Power BI App entry point your organization uses for audience controls.
- App-level permissions in Power BI do not apply the way they do when users open content from the app directly.

### What Catalog Ingests and Links Today

Catalog's standard Power BI integration ingests reports, dashboards, and semantic data sets from Microsoft admin scan APIs, and reads Dataflow metadata during extraction to compute lineage. Dashboards here are tile collections Power BI exposes through admin metadata. Catalog does not ingest published Power BI Apps as a separate asset type, and Dataflows are not published as searchable Catalog assets.

For each ingested report or dashboard, Catalog stores the `webUrl` Microsoft returns in admin metadata when available. When that URL is missing, Catalog falls back to a default workspace path under `app.powerbi.com`. Catalog search, the AI Assistant, and federated Microsoft surfaces pass that stored URL through as the outbound link to Power BI. Catalog does not rewrite links to a published app URL.

:::tip[Published Power BI App vs Microsoft Entra app]

A **published Power BI App** is Microsoft's distribution package for curated reports and dashboards. The **Microsoft Entra app** you register for Catalog extraction is a separate concept used only for API access. See [Supported asset types][] on [Power BI setup][].

:::

### What You Can Do Today

1. **Limit what Catalog ingests**  
   Use [Scope Power BI ingestion][] to exclude workspaces or folders whose raw reports or dashboards should not appear in Catalog search. That removes them from federated results indexed from Catalog, not from Power BI itself.

2. **Limit who discovers assets in Catalog**  
   **Manage access** on dashboard folders restricts which teams see ingested Power BI content in Catalog browse and search. See [Assets Access Control][]. That control does not change Power BI App audience rules when someone follows an outbound link into the Power BI service.

3. **Contact Support for product requests**  
   If you need federated search to open published app URLs or to suppress individual dashboards while keeping other Power BI assets indexed, contact [Coalesce Support][]. Catalog does not expose a customer setting for app-level deep links today.

For workspace visibility defaults after ingestion, see [Users See Power BI Workspaces They Should Not][] earlier in this guide.

## What's Next?

- Review extraction modes and native versus BI Importer paths in [Ingestion architecture][].
- Walk through setup, credential lifecycle, admin settings, first sync, and table navigation versus embedded SQL in [Power BI setup][] and [Troubleshooting in Power BI setup][].
- Return to concepts, UI paths, and supported mashup patterns in [Power BI Dataflows in Catalog][].
- Fix or understand field lineage after column renames in [Troubleshoot Power BI Lineage When Columns Are Renamed][].
- Review quick and advanced search behavior in [Catalog search][].
- For general lineage behavior that is not specific to Power BI, see [Lineage troubleshooting][].

[Assets Access Control]: /docs/catalog/administration/assets-access-control
[Assets Access Control troubleshooting]: /docs/catalog/administration/assets-access-control#troubleshooting
[Azure portal]: https://portal.azure.com/
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[Credential lifecycle]: /docs/catalog/integrations/data-viz/power-bi/powerbi#credential-lifecycle
[Databricks]: /docs/catalog/integrations/data-warehouses/databricks
[DAX, Mashup Expressions, and Field Lineage in Catalog]: /docs/catalog/integrations/data-viz/power-bi/powerbi#dax-mashup-expressions-and-field-lineage-in-catalog
[Lineage or Metadata Still Looks Wrong After Admin API Changes]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#lineage-or-metadata-still-looks-wrong-after-admin-api-changes
[New Power BI Workspace Appears With Broad Access]: /docs/catalog/administration/assets-access-control#new-power-bi-workspace-appears-with-broad-access
[New User Sees All Power BI Workspaces]: /docs/catalog/administration/assets-access-control#new-user-sees-all-power-bi-workspaces
[Power BI workspaces and Catalog visibility]: /docs/catalog/administration/assets-access-control#power-bi-workspaces-and-catalog-visibility
[Scope Power BI ingestion]: /docs/catalog/integrations/data-viz/power-bi/powerbi#scope-power-bi-ingestion
[Troubleshooting in Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi#troubleshooting
[How to Validate Dataflow Lineage]: /docs/catalog/integrations/data-viz/power-bi#how-to-validate-dataflow-lineage
[Lineage in the Catalog UI]: /docs/catalog/integrations/data-viz/power-bi#lineage-in-the-catalog-ui
[No Upstream Lineage From a Data Set That Uses Dataflows]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#no-upstream-lineage-from-a-data-set-that-uses-dataflows
[Ingestion Fails or Metadata Never Appears]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#ingestion-fails-or-metadata-never-appears
[Integration Paused After Expired Credentials]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#integration-paused-after-expired-credentials
[Catalog search]: /docs/catalog/navigate/search
[Coalesce Support]: mailto:support@coalesce.io
[Ingestion architecture]: /docs/catalog/integrations/data-viz/power-bi/powerbi#ingestion-architecture
[Lineage troubleshooting]: /docs/catalog/navigate/lineage/lineage-troubleshooting
[Power BI Dataflows in Catalog]: /docs/catalog/integrations/data-viz/power-bi
[Power BI setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi
[Troubleshoot Power BI Lineage When Columns Are Renamed]: /docs/catalog/integrations/data-viz/power-bi/power-bi-lineage-column-rename
[Supported asset types]: /docs/catalog/integrations/data-viz/power-bi/powerbi#supported-asset-types
[Users See Power BI Workspaces They Should Not]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#users-see-power-bi-workspaces-they-should-not

---

## Power BI Setup

Connect Power BI to Coalesce Catalog with Microsoft Entra, Fabric tenant settings, and either Catalog-managed or client-managed extraction. This page covers optional IP allowlisting, app registration, security groups, Admin API settings, credentials, extractor scheduling, and how to narrow Power BI ingestion when you have many workspaces or strict production boundaries.

## Related Guides

Use these companion pages alongside setup:

- [Power BI Dataflows in Catalog][] for lineage through Dataflows, supported mashup patterns, and validation from consuming semantic data sets
- [Power BI troubleshooting in Catalog][] when ingestion fails, credentials expire, search misses assets, or lineage looks stale
- [Troubleshoot Power BI Lineage When Columns Are Renamed][] for Power Query rename behavior and measure expectations in field lineage

## Supported Asset Types

Catalog ties Power BI assets to warehouse objects your integrations have already synced. Lineage depth varies by asset type: semantic data sets and standard reports usually trace farthest. Catalog ingests Dataflow metadata during extraction to compute lineage only; Dataflows are not searchable semantic models, Catalog assets, or semantic representations. Paginated reports typically show lighter field-level detail.

For a full comparison table by asset type, including paginated reports and Dataflows, see [Which Power BI Assets Connect to Warehouse Lineage][] in [Power BI Dataflows in Catalog][].

Catalog's standard integration does not ingest published Power BI Apps as a distinct asset type. It ingests the underlying reports, dashboards, and semantic data sets Microsoft exposes through admin scan metadata. Outbound links from Catalog to Power BI use those workspace-level URLs from Microsoft metadata. That is separate from the Microsoft Entra app you register in [Before You Begin][] for API credentials. For Teams and Microsoft Search link behavior, see [Microsoft Teams and Search Deep Links for Power BI][] in [Power BI troubleshooting in Catalog][].

## Before You Begin

Work through this page in order: optional IP allowlisting, then the Microsoft Entra app, security group, and Power BI Admin portal settings. Confirm the following first:

- You need Cloud Application Administrator or Application Administrator access in Microsoft Entra ID and Fabric Administrator access in Power BI to complete this setup.
- You have a warehouse-type integration configured if you want Catalog to relate Power BI assets to warehouse tables for lineage.
- If Catalog runs ingestion for you, the first sync can take up to 48 hours. Plan to validate in Catalog after that finishes, and again after you change tenant settings or the model so a new sync can run.

For data sets that rarely refresh or that use only DirectQuery, plan to refresh or republish when you need full lineage detail from the Power BI APIs. See [Refresh and Republish Data Sets][].

### 1. IP Allowlisting

This only applies if you need a VPN to connect to Power BI. If you allowlist by IP, map your Catalog hostname to the fixed address:

- `34.42.92.72` for instances at [app.us.castordoc.com][]
- `35.246.176.138` for instances at [app.castordoc.com][]

### 2. Create a Microsoft Entra App for Catalog in the Azure Portal

Log in to the [Azure portal][] and search for Microsoft Entra ID. In it, create the new App with the following parameters and then click Register:

- name: `Catalog`
- Supported account types: `Accounts in this organizational directory only`

On the homepage of your newly created application, from the **Overview** screen, copy the values for the following fields and store them in a secure location for later:

- Application (client) ID
- Directory (tenant) ID

From the left menu, navigate to **Manage** > **API permissions** and add the following two Microsoft Graph permissions, both of type **Application permissions**:

- `GroupMember.Read.All`
- `User.ReadBasic.All`

Once added, make sure **Admin consent is granted** for both permissions.

:::warning[Avoid extra Power BI permissions on the app]

Make sure the app does not have any admin-consent required permissions for Power BI set on it. They're never used and can cause errors that are hard to troubleshoot. See how to [check whether your app has admin-consent required permissions][Power BI admin-consent permissions].

:::

Navigate to **Manage** > **Certificates & secrets** and create a new client secret with the description and expiration date of your choosing. Then, for the newly created client secret, click the clipboard icon to copy the **Value** and store it in a secure location for later.

For more details, see [Create an Azure AD app for Power BI embedded][].

### 3. Create a Microsoft Entra Security Group

In the left menu of the **Microsoft Entra ID** page, under the **Manage** section, click **Groups**.

Then create a new group with the following configuration:

- Set the **Group type** to Security.
- Enter `API AD` as **Group name** and, if you want, a **Group description**.
- Under **Members**, search for the application registration created above and add it to the list.

For more details, see [Create an Azure AD security group][].

### 4. Enable the Power BI Service Admin Settings

Go to the [Power BI Admin portal][] tenant settings. See [how to get to the Admin portal][] for navigation help. For Microsoft guidance, see [Enable the Power BI service admin settings][] and [Metadata scanning setup][].

- In the **Developer Settings** section, enable:
  - Enable `Service principals can use Fabric APIs`
- Add the group **API AD** to it.

- In the **Admin API Settings** section, enable:
  - Enable `Service principals can access read-only admin APIs`
  - Enable `Enhance admin APIs responses with detailed metadata`
  - Enable `Enhance admin APIs responses with DAX and mashup expressions`
- Add the group **API AD** to all of them.

### Refresh and Republish Data Sets

This step is recommended. To get lineage information from the Power BI API:

- Refresh or republish data sets, especially those that are not scheduled.
- Republish data sets that contain only DirectQuery tables.

## Ingestion Architecture

Power BI connects to Catalog through a native integration built for Power BI admin APIs and `castor-extract-powerbi` output. That path is separate from the generic [BI Importer][] workflow used for BI tools Catalog does not integrate with directly.

Who runs extraction defines how metadata reaches Catalog:

- **Catalog-managed extraction** - You supply Entra app credentials in **Settings > Integrations**. Catalog runs scheduled extraction against Power BI on your behalf. Choose this path when your tenant allows stored service principal credentials and you want Catalog to own the schedule.
- **Client-managed extraction** - You install `castor-extractor[powerbi]`, run `castor-extract-powerbi` on infrastructure you control, and upload artifacts with `castor-upload` on a schedule you define. Catalog ingests when upload files are present. Choose this path when you prefer not to store Power BI secrets in Catalog or when you already operate extract-and-upload jobs.

Extraction modes describe how metadata enters Catalog. They are not the same as sync back features that push descriptions from Catalog into a BI tool. Sync back is documented separately under [Sync back integrations][].

## DAX, Mashup Expressions, and Field Lineage in Catalog

When **Enhance admin APIs responses with DAX and mashup expressions** is enabled, Microsoft's admin APIs can return measures, DAX expressions, and mashup text. Catalog uses that output while ingesting Power BI metadata and building lineage. Catalog does not parse Power Query M; it applies pattern matching and heuristics against exported mashup for the connector and transform shapes Catalog tests. Microsoft's documentation for Fabric and Power BI describes how tenant administrators expose that level of detail for admin and scanning scenarios.

In Catalog, field lineage shows how warehouse columns, semantic model tables and columns, and measures connect in the lineage graph when those paths resolve. That view centers on relationships and dependencies, not on reproducing every authoring surface from Power BI Desktop.

Use this table to compare what each surface shows for measures:

| Surface | What Catalog shows | What Catalog does not show |
| ------- | ------------------ | -------------------------- |
| **Power BI Desktop** | Full DAX measure definition in the formula bar or measure dialog | N/A, authoring environment |
| **Catalog field lineage** | Dependency paths among warehouse columns, model fields, and measures when the graph resolves | Full DAX formula text inside the lineage canvas |
| **Catalog asset detail and search** | Names, descriptions, and lineage relationships from ingested metadata | Complete measure definition text as in Power BI Desktop |

Enabling admin API settings for DAX and mashup expressions is required for Power BI integration and lineage computation. It does not add the complete measure formula to lineage panels or asset detail views.

If your tenant matches this setup page and extraction completes successfully, missing formula text in the lineage UI reflects how Catalog presents lineage today, not a failed or incorrectly configured integration. For a symptom-based FAQ and what to use instead, see [Missing DAX measure definitions in Catalog][] in [Power BI troubleshooting in Catalog][].

For scenarios that compare measure lineage to Power Query rename behavior, see [Troubleshoot Power BI Lineage When Columns Are Renamed][].

## Catalog Managed

Send your Microsoft Entra app credentials in **Settings > Integrations** so Catalog can run scheduled extraction against Power BI on your behalf.

Send the following:

- `Tenant (Directory) ID`: Your Power BI instance tenant identifier
- `Client (Application) ID`: the ID of the Catalog application for Power BI
- `Secret Value`: the value of the secret associated to the Catalog App

Input your credentials directly in the [Catalog App integration settings][] under the following format:

```json
{
    "clientId": "****",
    "secret": "****",
    "tenantId": "****"
}
```

Your first sync can take up to 48 hours. Your Catalog team will notify you when it completes.

If you prefer not to store Power BI credentials in Catalog, continue to [Client Managed][].

### Credential Lifecycle

Microsoft Entra client secrets expire on the schedule you chose when you created the secret. Plan renewals before expiry so Catalog-managed extraction does not stop.

Typical symptoms when a secret expires:

- Ingestion errors that reference an expired client secret for your Catalog app registration
- The Power BI integration shows as paused in **Settings > Integrations**
- Search and lineage stop updating even though tenant admin settings are unchanged

To renew credentials:

1. In the [Azure portal][], open your Catalog app registration and create a new client secret under **Manage** > **Certificates & secrets**. Copy the new **Value** before you leave the page.
2. In Catalog, go to **Settings > Integrations**, open your Power BI source, and choose **Edit credentials**. Paste the updated `clientId`, `secret`, and `tenantId` JSON.
3. If Support or your Catalog team paused the integration after the failure, notify [Coalesce Support][] after you save valid credentials so they can re-enable the source and run ingestion again when needed.
4. Allow one full extraction cycle before you judge search or lineage freshness.

:::warning[First sync window]

Avoid rotating the client secret while the first 48-hour sync is still running unless Microsoft has already invalidated the old secret. A mid-sync rotation can extend the time before Catalog shows a complete graph.

:::

For step-by-step recovery when the integration is already paused, see [Integration Paused After Expired Credentials][] in [Power BI troubleshooting in Catalog][].

## Client Managed

Client-managed extraction runs on your infrastructure. You install the extractor, schedule runs, and push artifacts to Catalog with the upload tooling your Catalog team provides.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[powerbi]
```

For further details, see the [castor-extractor PyPI page][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-powerbi [arguments]
```

The script will run and display logs as follows:

```text
INFO - Starting extraction of PowerBiAsset.ACTIVITY_EVENTS
INFO - Wrote output file: ./files/1708021983-activity_events.json
INFO - Starting extraction of PowerBiAsset.DASHBOARDS
INFO - Wrote output file: ./files/1708021983-dashboards.json
INFO - Starting extraction of PowerBiAsset.DATASETS
INFO - Wrote output file: ./files/1708021983-datasets.json
INFO - Starting extraction of PowerBiAsset.METADATA
INFO - scan bbe1669a-8d4b-4598-a3a1-8763ea2babe7 ready
INFO - Wrote output file: ./files/1708021983-metadata.json
INFO - Starting extraction of PowerBiAsset.REPORTS
INFO - Wrote output file: ./files/1708021983-reports.json
INFO - Wrote output file: /tmp/catalog/1649078755-summary.json
```

#### Arguments

- `-t`: Tenant ID, your Power BI instance tenant identifier
- `-c`: Client (Application) ID, the ID of the Catalog application for Power BI
- `-s`: Secret Value, the value of the secret associated to the Catalog App
- `-o`, `--output`: Target folder to store the extracted files

#### Optional Arguments

- `-sc`, `--scopes` : Power BI Scopes to be used, optional
- `-l`, `--login_url` : Login URL of your Microsoft Entra server, optional
- `-a`, `--api_base`: Power BI REST API base URL, optional

Run any extractor command with `--help` to print the full argument list.

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Power BI content in Catalog. Here is how to do it:

The Catalog team will provide you with:

1. `Source Id`. Catalog uses this identifier to match your uploaded files to your Catalog instance.
2. `Catalog Token`, the API token for upload.

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Upload Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: Source ID provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are { `DBT` | `VIZ` | `WAREHOUSE` }

#### Target Files

To specify the target files, use exactly one of the following:

- `-f`, `--file_path`: to push a single file.
- `-d`, `--directory_path`: to push several files at once from one directory.

When you use `--directory_path`, the upload tool sends every file in that directory. Confirm the directory contains only the extracted artifacts before you push.

Then you will need to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

## Scope Power BI Ingestion

Large tenants, frequent deployments, and CI/CD pipelines that create many Power BI workspaces can push more metadata into Catalog than you want to browse or link in lineage. Catalog can limit which Power BI visualization assets are ingested by applying path-based glob rules to each asset's folder path in the integration pipeline. Your Catalog team can also layer Premium capacity or Fabric capacity boundaries when that matches how you separate production from experimentation. This section explains how that scope works and how to plan changes with your Catalog team.

You connect the Power BI app and credentials in **Settings > Integrations**. Folder allow lists, block lists, and optional capacity-backed scope are applied during ingestion on the Catalog side. Coordinate path and capacity rules with your Catalog point of contact or [Coalesce Support][] so they match your next extraction or refresh cycle.

### What the Rules Target

Catalog stores a normalized folder path for each visualization asset. Your allow list or block list patterns match those paths using glob-style rules, so you can include or exclude segments that stay stable in your tenant, such as a production workspace label or a shared root folder.

Workspace names and other labels from Power BI often appear as segments in the paths Catalog evaluates. The exact layout depends on your tenant and how assets are organized, so you should confirm real path examples with your Catalog contact when you design patterns. Teams that standardize prefixes or brackets in workspace names, for example to mark production, often get simpler patterns and fewer surprises after deployment.

### Allow List or Block List

You configure either an allow list of path patterns or a block list of path patterns for the Power BI visualization integration, not both at the same time.

- **Allow list** - Catalog ingests an asset only when its path matches at least one active pattern. The typical starting point when you only need to omit a bounded set of development, test, or sandbox paths.
- **Block list** - Catalog ingests every asset whose path does not match any active pattern. Good for teams that prefer to send the definitive list of folders or workspaces to keep instead of maintaining a growing exclude list whenever CI/CD creates new spaces.

If both modes are present in the integration configuration, ingestion fails with a clear configuration error until your Catalog team removes one of the modes.

When you have a very large dashboard count and workspaces or folders change often, fixed lists of workspace names can be hard to maintain. Ask your Catalog contact for a scope model that fits high churn, which can include capacity-based rules, broader path patterns, or a staged plan across several extraction cycles.

### Scope by Premium Capacity or Deployment Workspaces

Some organizations group workspaces by Microsoft Fabric capacity or by deployment pipeline stages such as development, test, and production Premium capacities. In those layouts you can ask your Catalog team to include only the capacities that host approved workspaces or to exclude specific capacities you treat as non-production, for example reserved personal capacities you never want in Catalog.

Bring the capacity display names or object identifiers from the Microsoft Fabric admin center and describe which capacities should remain in scope. Your Catalog team maps that intent onto the ingestion configuration the same way they manage path rules.

### Multiple Power BI Integrations in One Tenant

Some organizations run more than one Catalog Power BI integration against the same tenant when different product lines or warehouses need separate graphs. Give each integration its own path allow or block rules so the same published artifact does not appear twice under two sources unless that duplication is intentional. Your Catalog contact can help mirror exclusion logic across integrations so each integration stays aligned with its warehouse or domain.

### Narrow Discovery Without Removing Assets From Ingestion

Path and capacity rules remove assets from Catalog for that integration, which shrinks search and lineage graphs. Ingestion scope controls what Catalog pulls from Power BI. **Manage access** on dashboard folders controls who can see what was ingested. Use both when you need to separate production from non-production workspaces and limit discovery by team.

When you still need the metadata in Catalog but want to steer readers away from legacy or non-production dashboards, combine ingestion scope with Catalog governance instead of relying on ingestion alone:

- **Manage access** on dashboard folders limits which teams can browse and search those folders while admins retain full visibility. Power BI workspaces map to dashboard folders; new workspace folders are company-wide visible until you restrict them. See [Power BI workspaces and Catalog visibility][] and [Assets Access Control][].
- Deprecation marks a dashboard as outdated but keeps it discoverable for admins and in lineage with a clear visual signal. It does not hide the asset the way an ingestion block list does. See **Deprecated** under [Dashboards][].

### How to Request a Change

Work with your Catalog point of contact or [Coalesce Support][] to update ingestion scope. Share the workspace or folder boundaries you want, whether you need an allow list or a block list, capacity names or IDs if you use capacity-based scope, and any naming conventions your CI/CD system uses so paths stay predictable.

After your Catalog team applies an update, expect the change on the next successful extraction or refresh cycle for that Power BI source.

When you narrow an allow list or add block patterns, dashboards, reports, or semantic models can disappear from Catalog on the next run, which removes their downstream lineage until you broaden scope and ingest them again. After each change, spot check critical assets in Catalog. If you switch from a block list to an allow list but the set of ingested assets intentionally stays the same, lineage should stay aligned, but you should still confirm a sample of dashboards and models because any accidental exclusion drops links that depended on those assets.

If production reporting workspaces are omitted from an allow list, those assets stop appearing in Catalog and no longer participate in lineage graphs until you fix the patterns and complete another successful ingestion.

## Troubleshooting

These topics come up often while onboarding Power BI or validating lineage in Catalog. For ingestion failures, expired credentials, Dataflow lineage validation, DAX expectations, semantic model authoring modes, and longer diagnosis flows, see [Power BI troubleshooting in Catalog][].

### Lineage Differs When You Build a Semantic Model From the Table Browser Versus Embedded SQL

Links to warehouse tables are strongest when mashup text exposes explicit SQL, for example through `Value.NativeQuery`, that Catalog can match to objects already synced from your warehouse integration. Models built only through the visual table picker often export less explicit SQL, which limits what Catalog can resolve.

For symptoms, a validation checklist that includes Databricks and other warehouses, and when to contact Support, see [Lineage When Semantic Models Use Table Navigation Versus SQL][] in [Power BI troubleshooting in Catalog][].

### First Sync Still Running After a Day

Large tenants or metadata-heavy workspaces extend first ingestion toward the upper end of the documented window.

To resolve this:

1. Confirm **Admin API Settings** and **Developer Settings** match [Enable the Power BI service admin settings][] on this page.
2. Avoid concurrent credential rotations until the first sync completes.
3. If progress stalls beyond two days while settings are correct, contact [Coalesce Support][].

### Admin Portal APIs Return Errors for the Service Principal

Missing security group attachment or disabled Fabric API toggles block Catalog's read-only calls.

To resolve this:

1. Confirm the Microsoft Entra security group contains the Catalog app registration and appears in **Service principals can use Fabric APIs**.
2. Verify **Service principals can access read-only admin APIs** and the detailed metadata toggles stay enabled.
3. Retry ingestion after fifteen minutes so Admin API caches clear.

### Lineage Looks Thin for Rarely Refreshed or DirectQuery-Only Data Sets

Catalog reads data set metadata that reflects your refresh and modeling patterns. To resolve this:

1. Refresh data sets or republish when you need full lineage expansion, as noted in [Refresh and Republish Data Sets][].
2. Schedule refreshes that align with how analysts consume the semantic models.

### Credentials Saved but Catalog Cannot Pull Workspaces

Wrong tenant ID, expired client secret, or restricted service principal membership produces auth failures that surface as integration errors.

To resolve this:

1. Follow [Credential lifecycle][] on this page to rotate the client secret in Azure and update **Settings > Integrations** immediately after Microsoft invalidates the old secret.
2. Confirm the Catalog app remains in the Entra security group tied to Fabric tenant settings.
3. If the integration is paused, see [Integration Paused After Expired Credentials][] in [Power BI troubleshooting in Catalog][].

[Assets Access Control]: /docs/catalog/administration/assets-access-control
[Power BI workspaces and Catalog visibility]: /docs/catalog/administration/assets-access-control#power-bi-workspaces-and-catalog-visibility
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[Azure portal]: https://portal.azure.com/
[app.castordoc.com]: https://app.castordoc.com/
[app.us.castordoc.com]: https://app.us.castordoc.com/
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install
[Catalog App integration settings]: https://app.castordoc.com/settings/integrations
[Client Managed]: /docs/catalog/integrations/data-viz/power-bi/powerbi#client-managed
[Coalesce Support]: mailto:support@coalesce.io
[Create an Azure AD app for Power BI embedded]: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#step-1---create-an-azure-ad-app
[Create an Azure AD security group]: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#step-2---create-an-azure-ad-security-group
[Credential lifecycle]: /docs/catalog/integrations/data-viz/power-bi/powerbi#credential-lifecycle
[Dashboards]: /docs/catalog/assets/dashboards#details-panel
[Enable the Power BI service admin settings]: https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal#step-3---enable-the-power-bi-service-admin-settings
[how to get to the Admin portal]: https://learn.microsoft.com/en-us/fabric/admin/admin-center#how-to-get-to-the-admin-portal
[Integration Paused After Expired Credentials]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#integration-paused-after-expired-credentials
[Lineage When Semantic Models Use Table Navigation Versus SQL]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#lineage-when-semantic-models-use-table-navigation-versus-sql
[Metadata scanning setup]: https://learn.microsoft.com/en-us/fabric/admin/metadata-scanning-setup
[Power BI Admin portal]: https://app.fabric.microsoft.com/admin-portal/tenantSettings
[Power BI Dataflows in Catalog]: /docs/catalog/integrations/data-viz/power-bi
[Power BI troubleshooting in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting
[Power BI admin-consent permissions]: https://learn.microsoft.com/en-us/power-bi/enterprise/read-only-apis-service-principal-authentication#how-to-check-if-your-app-has-admin-consent-required-permissions
[Sync back integrations]: /docs/catalog/integrations/sync-back
[Refresh and Republish Data Sets]: /docs/catalog/integrations/data-viz/power-bi/powerbi#refresh-and-republish-data-sets
[Which Power BI Assets Connect to Warehouse Lineage]: /docs/catalog/integrations/data-viz/power-bi#which-power-bi-assets-connect-to-warehouse-lineage
[Troubleshoot Power BI Lineage When Columns Are Renamed]: /docs/catalog/integrations/data-viz/power-bi/power-bi-lineage-column-rename
[Missing DAX measure definitions in Catalog]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#missing-dax-measure-definitions-in-catalog
[Microsoft Teams and Search Deep Links for Power BI]: /docs/catalog/integrations/data-viz/power-bi/power-bi-troubleshooting#microsoft-teams-and-search-deep-links-for-power-bi
[Before You Begin]: /docs/catalog/integrations/data-viz/power-bi/powerbi#before-you-begin

---

## Qlik Sense

## Requirements

:::warning

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

To get things started with Qlik in Catalog, you will need:

* Ability to use (or create) a user with access to needed spaces
* Ability to create an API for that user. Follow the steps in [Setting up the API key][].
* This user must have at least the "Developer" role

Related pages:

<!-- <DocCardList /> -->

## Catalog Managed

Please:

* Input your credentials directly in the [Catalog App integration settings][] with the following format:

```json
{
    "apiKey": "*****",
    "baseUrl": "https://xxxx.xx.qlikcloud.com"
}
```

For your first sync, it will take up to 48 h and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, please continue to [Client Managed](#client-managed).

## Client Managed

### Doing a One Shot Extract

For your trial, you can give us a one-shot view of your BI tools.

To get things working quickly, here is a [Google Colab notebook][] to run the package.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[qlik]
```

For further details, see the [castor-extractor PyPI page][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-qlik [arguments]
```

The script will run and display logs as following:

```bash
INFO - Extracting SPACES from REST API
INFO - Wrote output file: /tmp/catalog/qlik/1655814192-spaces.json
INFO - Extracting USERS from REST API
INFO - Wrote output file: /tmp/catalog/qlik/1655814192-users.json
INFO - Extracting APPS from REST API
INFO - Wrote output file: /tmp/catalog/qlik/1655814192-apps.json

...

INFO - Wrote output file: /tmp/catalog/qlik/1655814192-summary.json
```

#### Arguments

* `-b`, `--base-url`: Qlik base URL, your API endpoint, usually your Qlik URL homepage
* `-a`, `--api-key`: the generated `API_KEY` from the Profile page settings
* `-o`, `--output`: target folder to store the extracted files

#### Optional Arguments

* `-e`, `--except-http-error-statuses`: list of HTTP statuses for which to catch errors and log as warning instead. Helpful to continue script execution when missing rights on some assets.

:::info

You can also get help with argument `--help`

:::

## Use ENV Variables

If you don't want to specify arguments every time, you can set the following ENV in your `.bashrc`:

```text
CASTOR_QLIK_BASE_URL
CASTOR_QLIK_API_KEY

CASTOR_OUTPUT_DIRECTORY
```

Then the script can be executed without any arguments:

```bash
castor-extract-qlik
```

It can also be executed with partial arguments (the script looks in your `ENV` as a fallback):

```bash
castor-extract-qlik --output /tmp/catalog
```

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Qlik content in Catalog. Here is how to do it:

The Catalog team will provide you with

1. `Catalog Identifier` (an id for us to match your Qlik files with your Catalog instance)
2. `Catalog Token` An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Upload Arguments

* `-k`, `--token`: Token provided by Catalog
* `-s`, `--source_id`: account id provided by Catalog
* `-t`, `--file_type`: source type to upload. Currently supported are { `DBT` | `VIZ` | `WAREHOUSE` }

#### Target Files

To specify the target files, provide **one** of the following:

* `-f`, `--file_path`: to push a single file

 or

* `-d`, `--directory_path`: to push several files at once (\*)

:::warning

(\*) The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will need to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

## Python Usage

The extraction script mentioned in the previous section works for basic usage. The following section allows custom usage of the library.

### Using Explicit Arguments

```python
from castor_extractor.visualization.qlik import extract_all

# run extraction
extract_all(
    base_url="https://<your-id>.<region>.qlikcloud.com",
    api_key="********",
    output="/tmp/catalog",
    except_http_error_statuses=[403],
)
```

### Using ENV

```python
from castor_extractor.visualization.qlik import extract_all

# run extraction
extract_all()
```

If you want to manipulate a specific asset:

```python
from castor_extractor.visualization.qlik import QlikClient, QlikAsset

client = QlikClient(
    server_url="https://<your-id>.<region>.qlikcloud.com", 
    api_key="********",
    except_http_error_statuses=[403],
)

# fetch spaces
spaces = client.fetch(QlikAsset.SPACES)

# fetch users
users = client.fetch(QlikAsset.USERS)

# fetch apps
apps = client.fetch(QlikAsset.APPS)

# fetch lineage - scoped on apps
lineage = client.fetch(QlikAsset.LINEAGE, apps=apps)

# fetch measures - scoped on apps
measures = client.fetch(QlikAsset.MEASURES, apps=apps)
```

[Setting up the API key]: https://docs.coalesce.io/docs/catalog/integrations/data-viz/qlik-sense/setting-up-the-api-key
[Catalog App integration settings]: https://app.castordoc.com/settings/integrations
[Google Colab notebook]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install

---

## Setting Up the API Key

Configure API key access so Catalog can connect to your Qlik Sense instance. You can create a dedicated Catalog user or add a key to an existing user.

## Requirements

- Ability to use (or create) a user with access to needed spaces
- Ability to create an API key for that user
- The user must have at least the "Developer" role

## 1. Get a Qlik API Key

### Option A: Create a Catalog Qlik User

You can invite **`onboarding@castordoc.com`** to your Qlik instance. The Catalog team will then create the API key using the Qlik UI.

1. Invite a new user using **`onboarding@castordoc.com`**.
2. Allocate a Professional license to this user.

### Option B: Add a Key to an Existing User

If you have an account with sufficient privileges, you can create an API key linked to your account.

1. Go to your profile settings and click **API Keys**.
2. Create an API key.
3. Input the API key in the [App](https://app.castordoc.com/settings/integrations) with the following format:

```json
{
    "apiKey": "*****",
    "baseUrl": "https://xxxx.xx.qlikcloud.com"
}
```

## 2. Add This User to the Wanted Spaces

- Add the user chosen above to each space you want to see in Catalog. The "Can Manage" right is required.
- For lineage purposes, make sure to also add Apps that are used for data transformation. Catalog cannot compute lineage on what it cannot see.
- Space access grants access to all space apps, except when using [Section Access](https://help.qlik.com/en-US/sense/August2022/Subsystems/Hub/Content/Sense_Hub/Scripting/Security/manage-security-with-section-access.htm). For those apps, add the Catalog user one by one (or using a script).

---

## Redash

Integrate Redash with the Catalog to sync dashboards and queries.

## Requirements

* You must have an Admin User API key.

## Catalog Managed

### 1. Get Credentials

To get things started with Redash in the Catalog, you will need to be a Redash admin and provide us with:

* Your `User API key`
* Your `Base URL`

To retrieve your `User API key` go to "Settings > Account", select the API Key tab and copy the value from there.

As for your `Base URL`, we need the full domain of your Redash account. The expected format for it is `https://app.redash.io/myorg/` or `https://redash.myorg.com/`

For more information, see the [Redash API documentation][].

### 2. Add Credentials in Catalog App

You can now enter the newly created credentials in the [Catalog App integrations settings][].

* Go to "Settings > Integrations"
* Click on "Redash Add"
* Select "Managed by Catalog"
* Choose a name for this integration in Catalog
* Add the credentials as JSON
* Click on "Save"

The expected JSON format is:

```json
{
    "baseUrl": "<str>",
    "apiKey": "<str>"
}
```

[Redash API documentation]: https://redash.io/help/user-guide/integrations-and-api/api
[Catalog App integrations settings]: https://app.castordoc.com/settings/integrations

---

## Sigma

Connect Catalog to Sigma to sync workbooks, data sets, Data Models, and related metadata. You can use Catalog-managed credentials or run the extraction package yourself. Sigma's APIs control which objects and lineage details Catalog can return, so some graphs look complete while others stay object-level only.

## Requirements

Before you connect Sigma to Catalog, confirm that you meet the Sigma access requirements and warehouse prerequisites in this section.

:::warning[Warehouse integration required]

A warehouse integration must already be configured to complete the first ingestion of this integration.

:::

You also need the following:

- You must be an Org Admin to create a Sigma API token and Client ID.
- Anyone with access to an active Sigma API token and Client ID can authenticate with Sigma's API at the access level associated with the token.

## Catalog-Managed

To get started with Sigma in Catalog, you need to be a Sigma admin and provide:

- A Sigma API token from Sigma. Treat it like any other secret credential.
- Your Client ID
- The Host URL of your instance

For more information on how to retrieve the API token and Client ID, see [Get an API Token and Client ID][].

To get the correct Host URL of your instance, see [Identify your API request URL][].

Input your credentials directly into your Catalog account in the following format:

```json
{
    "apiToken": "*****",
    "clientId": "*****",
    "host": "https://<your_provider_url>.sigmacomputing.com"
}
```

:::info[Sigma API Details]

For further details on the Sigma API, see [Get Started with Sigma's API][].

:::

For your first sync, it will take up to 48 hours and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, continue to [Client-Managed][].

### Client-Managed

#### Doing a Time Extract

For your trial, you can give us a one time view of your BI tool.

To get things working quickly, use this [Google Colab][] to run the package.

#### Running the Extraction Package

##### Install the PyPI Package

For further details on the Catalog Extractor PyPI package, see [castor-extractor on PyPI][].

##### Running the PyPI Package

Once the package has been installed, run the following command in your terminal:

```bash
castor-extract-sigma [arguments]
```

The script will run and display logs similar to:

```bash
INFO - Extracting users from Sigma API
INFO - POST(https://cloud.sigma.com/api/4.0/login)
INFO - GET(https://cloud.sigma.com/api/4.0/users/search)
INFO - Fetched page 1 / 7 results
INFO - GET(https://catalog.cloud.sigma.com/api/4.0/users/search)
INFO - Fetched page 2 / 0 results

...

INFO - Wrote output file: /tmp/catalog/1649079699-projects.json
INFO - Wrote output file: /tmp/catalog/1649079699-summary.json
```

##### Extraction Arguments

- `-H`, `--host`: Sigma host
- `-c`, `--client-id`: Sigma client ID
- `-a`, `--api-token`: Generated API key
- `-o`, `--output`: Directory to write to

#### Scheduling and Push to Catalog

When moving out of trial, you will want to refresh your Sigma content in Catalog. Here is how to do it:

The Catalog team will provide you with:

1. A Catalog Identifier we use to match your Sigma files with your Catalog instance.
2. A Catalog Token, an API token you use when uploading to Catalog.

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

##### Upload Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: Account ID provided by Catalog
- `-t`, `--file_type`: Source type to upload. Currently supported: `DBT`, `VIZ`, or `WAREHOUSE`

##### Target Files

To specify the target files, provide exactly one of the following:

- `-f`, `--file_path`: To push a single file

or

- `-d`, `--directory_path`: To push several files at once

The tool will upload every file included in the given directory. Make sure it contains only the extracted files before pushing.

Then schedule the script run and the push to Catalog using your preferred scheduler.

## Known Limitations

Catalog reads Sigma metadata and lineage through Sigma's public APIs. What you see in Catalog therefore depends on what those APIs expose, their rate limits, and how Sigma models dashboards, fields, and dependencies for your tenant.

Use this section to interpret gaps that are tied to Sigma's API rather than Catalog configuration alone:

- **Dashboard coverage** - Some dashboards or nested dashboard content might not extract with the same depth as Sigma workbooks and data sets you see elsewhere in Catalog.
- **Field-level lineage** - Column-to-field lineage between warehouse columns and Sigma fields can be incomplete or absent when Sigma does not expose the relationships Catalog needs at field granularity. Asset-level lineage is still the most reliable view for many tenants. It reflects which Sigma objects connect to which warehouse tables or other Sigma assets.
- **API limits** - Aggressive extraction schedules or very large Sigma estates can encounter throttling responses from Sigma. Catalog retries within normal extraction runs, but a sync can take longer or need another run before every object appears.

If extraction finishes successfully and credentials are correct, missing pieces in lineage or dashboards are often consistent with Sigma's API surface rather than an error in Catalog. For how asset-level lineage differs from field-level lineage inside Catalog, see [Lineage][].

## Sigma Data Models

Sigma Data Models are a Sigma asset type that Sigma treats as part of its semantic modeling layer so teams can reuse and govern modeled data. Catalog's Sigma connector treats Data Models as first-class Sigma assets alongside workbooks and data sets during extraction.

You'll see Data Models reflected in Catalog's Sigma visualization content the same way as other synced Sigma assets, including when you search Catalog or inspect lineage graphs that involve those models.

Setup stays the same: configure your warehouse integration, supply a valid Sigma API token, Client ID, and Host URL using Catalog-managed or client-managed extraction, then run sync on your usual cadence. You do not use a separate integration or credential block just for Data Models.

For lineage specifically:

- Expect asset-level relationships wherever Sigma's APIs describe how a Data Model connects to upstream data sets, modeled tables, or warehouse objects Catalog already knows about through your integrations.
- **Field-level lineage** builds only from details Sigma publishes for each metric and dimension. Dimensions and metrics on a Data Model can still lack column-to-field edges when those details are unavailable, consistent with [Known Limitations][]. Catalog records what Sigma returns.

Together, workbook, data set, and Data Model metadata give you governance and reuse context inside Catalog even when finer-grained field edges are sparse.

## What's Next?

- Browse other integrations on the [BI Tools][] overview.
- Learn how Catalog represents asset-level versus field-level lineage in [Lineage][].

---

[BI Tools]: /docs/catalog/integrations/data-viz
[Known Limitations]: #known-limitations
[Client-Managed]: #client-managed
[Lineage]: /docs/catalog/navigate/lineage
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor on PyPI]: https://pypi.org/project/castor-extractor/#pip-install
[Get an API Token and Client ID]: https://help.sigmacomputing.com/hc/en-us/articles/4408555307027-Get-an-API-Token-and-Client-ID
[Identify your API request URL]: https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url
[Get Started with Sigma's API]: https://help.sigmacomputing.com/hc/en-us/articles/4408835546003-Get-Started-with-Sigma-s-API#h_01FJAKR4N3XMBA3V70MVABJHR9

---

## Sisense

Connect Catalog to Sisense to sync dashboards, widgets, and data models. You can use Catalog-managed credentials or prepare uploads yourself using the BI Importer format. Catalog-managed Sisense sync is still maturing, so this page describes credential JSON, how to choose a correct base URL, and how Sisense objects map into Catalog.

## Requirements

You need a Sisense administrator account or another role that can authorize Catalog to read Sisense metadata with the credentials you provide.

Confirm whether your deployment is Sisense Cloud or self-hosted. The base URL you enter must match the hostname where users open Sisense in the browser, including any custom domain your organization configured.

## Catalog Managed

Use this section when Catalog stores your Sisense credentials and runs scheduled sync for your Workspace.

:::info[Integration in progress]

Catalog-managed Sisense onboarding is still maturing. Use this section when Catalog operations enable credential-based sync for your tenant. For a self-contained path you fully control today, use [Client Managed][] with BI Importer.

:::

To connect Sisense with Catalog-managed credentials, supply:

- Username
- Password
- Base URL for your Sisense instance

Enter credentials in Catalog using this JSON shape:

```json
{
    "username": "*****",
    "password": "*****",
    "baseUrl": "https://your-company.sisense.com/"
}
```

Use these patterns when you choose `baseUrl`:

- **Sisense Cloud** - Start from the URL your users open to reach Sisense. It usually looks like `https://<tenant>.sisense.com/` or a vanity hostname your administrator configured. Include `https://` in the URL and add a trailing slash only when Catalog's integration form shows that format.
- **Self-hosted** - Use the internal or external HTTPS hostname where the Sisense web application listens, for example `https://analytics.mycompany.com/`. If Sisense sits behind a reverse proxy or custom port, mirror exactly what browsers use.

Replace placeholder hosts such as `https://www.XXXX.com/` with a real reachable URL before saving credentials.

For your first sync, allow up to 48 hours. Catalog notifies you when the initial sync completes.

If you prefer not to share credentials with Catalog operations, continue to [Client Managed][].

## Client Managed

### Doing a One-Time Extract

For a trial or one-time snapshot, assemble Sisense metadata into [BI Importer expected format][], then upload using the channel your Catalog contact provides.

The Catalog extractor package published on PyPI includes dedicated command-line entry points for several BI platforms. There is not yet a published `castor-extract-sisense` command in that package. Until a Sisense-specific command ships, rely on BI Importer formatting and upload flows. You can use the shared [Google Colab][] notebook to explore extraction patterns for other tools while you build Sisense files manually or with custom scripts.

### Sisense to Catalog Translation

:::info[Sisense element mapping]

Sisense elements map to Catalog as follows:

- **Sisense dashboard** - Dashboard in Catalog
- **Sisense widget** - Tile in Catalog
- **Data Models** - Live models, ElastiCube Models, and Hybrid models map to visualization models in Catalog

:::

## Common Issues

If credentials fail or dashboards look incomplete in Catalog, start with these checks:

- **401 or connection errors after saving credentials** - Double-check `baseUrl` for typos, missing `https://`, or use of an internal-only hostname that Catalog cannot reach. Align with the URL analysts use in the browser.
- **Partial dashboards** - Sisense security rules and user scope can hide widgets from the service account. Confirm the account sees every asset you expect in the Sisense UI.

## What's Next?

- Browse related setup patterns on [BI Tools][].
- Review file layout and columns for uploads in [BI Importer][].

---

[BI Tools]: /docs/catalog/integrations/data-viz
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[BI Importer expected format]: /docs/catalog/developer/catalog-apis/bi-importer#file-expected-content
[Client Managed]: #client-managed
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing

---

## MicroStrategy

Connect Catalog to MicroStrategy to sync projects, reports, and dossiers. Create a dedicated MicroStrategy user with the appropriate permissions and provide credentials to Catalog.

## Requirements

:::warning

A warehouse integration must already be configured to complete the first ingestion of this integration.

:::

You will need your [MicroStrategy administrator](https://www2.microstrategy.com/producthelp/Current/Workstation/en-us/Content/workstation_create_users_and_groups.htm) to complete these steps.

## If Needed, Whitelist Catalog IP

Here are our fixed IPs:

- For instances on [app.us.castordoc.com](https://app.us.castordoc.com/): `34.42.92.72`
- For instances on [app.castordoc.com](https://app.castordoc.com/): `35.246.176.138`

## Catalog Managed

To get started with MicroStrategy in Catalog, you need your MicroStrategy admin to create a dedicated user with the appropriate access:

### Create User in MicroStrategy

Create a new user in your MicroStrategy Workstation and assign minimum permissions for integrating with Catalog. See [create a new user](https://www2.microstrategy.com/producthelp/Current/Workstation/en-us/Content/workstation_create_users_and_groups.htm).

1. Open the [Workstation window](https://www2.microstrategy.com/producthelp/Current/Workstation/en-us/Content/navigating_Workstation.htm) with the navigation pane in smart mode.
2. From the left navigation menu, click **Users and Groups**.
3. In the upper left of the _Users and Groups_ page, click the **Select an Environment** dropdown and select your environment.
4. In the left menu of your selected environment, next to _All Users_, click the **+** button to create a new user.
5. In the _Create New User_ dialog:
   1. For _Account and Credentials_, enter the following details:
      1. For _Full Name_, enter a meaningful name for the new user.
      2. For _Email Address_, enter an email address for the new user.
      3. For _Username (Login)_, enter a username for the new user.
      4. For _Password_, create a password for the new user and confirm it in the next step.
      5. To disallow the new user from changing the password, check the **User cannot change password** box.
      6. At the bottom left of the form, check the **Active User** box.
   2. For _User Groups_, all users are automatically members of the [_Everyone_](https://www2.microstrategy.com/producthelp/Current/SystemAdmin/WebHelp/Lang_1033/Content/About_MicroStrategy_user_groups.htm) group, which typically has read permission for most objects. To assign any permissions not inherited from the default group to the new user:
      1. In the top right of _User Groups_, click **Manage User Group** to add a new user group.
      2. Click **Update** to confirm your selections.
   3. To assign [user privileges](https://community.microstrategy.com/s/article/KB30634-List-of-user-privileges-their-descriptions-and-their?language=en_US), in the left menu, click **Privileges** and check the following boxes:
      - **Use Architect Editors**: for fetching attribute, fact, and table definitions
      - **Use Library Web**: for fetching project metadata
      - **Web Report SQL**: for fetching SQL statements
      - **Web use Metric Editor**: for fetching metric definitions
      - **Web run Document**: for fetching document definitions
      - **Web run Dossier**: for fetching dossier definitions
   4. Click **Save** to complete setup.

### Provide Credentials

Input your credentials directly into your Catalog account in the following format:

```json
{
    "baseUrl": "https://www.XXXX.com/",
    "username": "*****",
    "password": "*****",
    "projectIds": "project_id_1, project_id_2"
}
```

- `baseUrl`: Enter the hostname of your MicroStrategy instance
- `username`: Enter the username you created for the instance
- `password`: Enter the password for the username
- `projectIds` (optional): Enter a string of the comma-separated project IDs you would like to import in Catalog

For your first sync, it will take up to 48 h and we will let you know when it is complete.

---

## Superset

Integrate Apache Superset with Catalog to sync dashboards, charts, and related metadata. You can use Catalog-managed credentials or prepare uploads yourself using the BI Importer format. Catalog-managed Superset sync is still maturing; this page describes prerequisites, credential JSON, how Superset objects map today, and where to extend payloads when your ingestion includes more than charts and dashboards.

## Requirements

You need to be a Superset Admin for the Superset deployment you connect to Catalog. Admin access is required so Catalog can read metadata across your Superset deployment that standard viewer roles cannot reach.

## Catalog Managed

Use this section when Catalog stores your Superset credentials and runs scheduled sync for your Workspace.

:::info[Integration in progress]

Catalog-managed Superset onboarding is still maturing. The admin requirement above applies to both Catalog-managed sync and any manual Superset steps you run yourself. If you need a fully self-run pipeline today, use [Client Managed][] with BI Importer.

:::

Provide Catalog with Superset connection details in this JSON shape:

```json
{
    "host_url": "<Superset Host URL>",
    "username": "<Superset admin username>",
    "password": "<Superset admin password>"
}
```

`host_url` should be the root URL analysts use to open Superset in a browser. Include `https://` in the value and adjust for reverse proxies or custom ports exactly as your network exposes Superset.

Enter credentials directly in Catalog.

For your first sync, allow up to 48 hours. Catalog notifies you when the initial sync completes.

If you prefer not to share credentials with Catalog operations, continue to [Client Managed][].

## Client Managed

### Doing a One-Time Extract

For a trial or one-time snapshot, assemble Superset metadata into [BI Importer expected format][], then upload using the channel your Catalog contact provides.

The Catalog extractor package published on PyPI includes dedicated command-line entry points for several BI platforms. There is not yet a published `castor-extract-superset` command in that package. Until an Apache Superset-specific command ships, rely on BI Importer formatting and upload flows. You can use the shared [Google Colab][] notebook to explore extraction patterns for other tools while you build Superset exports manually or with custom scripts.

### Superset to Catalog Translation

These mappings reflect what Catalog documents today for Superset ingestion:

- **Chart** - Tile
- **Dashboard** - Dashboard

Superset also contains data sets, databases, SQL Lab saved queries, security roles, and row-level security objects. Those objects appear in Catalog only when your connector or BI Importer payload includes them and Catalog ingestion supports that shape for your tenant. When you expand beyond charts and dashboards, work with your Catalog contact to confirm which Superset objects are in scope for your project.

## Common Issues

If sync fails or dashboards look incomplete in Catalog, start with these checks:

- **403 responses during sync** - Confirm the account is still an Admin and that Superset did not expire the password or API-enabled policy you rely on.
- **Missing charts on dashboards** - Chart-level permissions or data source access might block the service account even when dashboards list correctly for admins.

## What's Next?

- Browse related setup patterns on [BI Tools][].
- Review file layout and columns for uploads in [BI Importer][].

---

[BI Tools]: /docs/catalog/integrations/data-viz
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[BI Importer expected format]: /docs/catalog/developer/catalog-apis/bi-importer#file-expected-content
[Client Managed]: #client-managed
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing

---

## Tableau(Data-viz)

Integrate Tableau with the Catalog to sync dashboards, data sources, and metadata.

## Requirements

Before you connect Tableau, make sure the following are in place. You need a Warehouse type integration configured to complete the first ingestion of this integration.

For all Tableau clients:

* Tableau admin credentials

For Tableau Server clients only:

* Have the Metadata API enabled. View how to [enable the Tableau Metadata API][].
* Have the API gateway. By default it is the URL you use to access Tableau.
* Use Tableau Server version 2019.3 or later. Older versions won't have lineage to tables available.

For Tableau on-premises, both Catalog managed and client managed work.

If you have MFA enabled for your account, you'll need to use the Tableau personal access token [PAT][] option.

## Allowlist Catalog IP

Add the following IPs to your allowlist:

* For instances on [app.us.castordoc.com][]: `34.42.92.72`
* For instances on [app.castordoc.com][]: `35.246.176.138`

:::warning[Self-Hosted Access]

Self-hosted repositories must be accessible from our IP over the public internet for Catalog managed integrations.

:::

## Catalog Managed

You can upload your credentials directly in the Coalesce App when creating your Tableau integration.

We need a Tableau [PAT][]:

```json
{
  "serverUrl": "https://something.tableau.com/",
  "siteId": "something",
  "tokenName": "catalog",
  "token": "abcdefgh"
}
```

`serverUrl`: Tableau base URL, your API endpoint, usually your Tableau URL homepage. For example: `<https://eu-west-1a.online.tableau.com>`
`siteId`: The Tableau Server site you're authenticating with. For example, in the site URL `http://MyServer/#/site/MarketingTeam/projects`, the site name is `MarketingTeam`. To connect with the Default site on the server, set `siteId` to `"Default"`.

For your first sync, it'll take up to 48 hours and we'll let you know when it's complete.

If you aren't comfortable giving us access to your credentials, continue to [Client managed](#client-managed) below.

## Client Managed

If you prefer to manage extraction yourself, you can run the Catalog extraction package on your own infrastructure.

### Doing a One Shot Extract

For your trial, you can give us a one shot view of your BI tool.

To get things working quickly, here's a [Google Colab][] to run our package.

### Running the Extraction Package

Install the extractor, run it against your Tableau instance, and then upload the results to Catalog.

#### Install the PyPI Package

```bash
pip install castor-extractor[tableau]
```

For further details, see the [castor-extractor PyPI page][].

#### Run the PyPI Package

Once the package has been installed, you can run the following command in your terminal:

```bash
castor-extract-tableau [arguments]
```

The script will run and display logs as follows:

```bash
INFO - Logging in using user and password authentication
INFO - Signed into https://eu-west-1a.online.tableau.com as user with id ****
INFO - Extracting USER from Tableau API
INFO - Fetching USER
INFO - Querying all users on site

...

INFO - Wrote output file: /tmp/catalog/1649078755-custom_sql_queries.json
INFO - Wrote output file: /tmp/catalog/1649078755-summary.json
```

#### Credentials

You can sign in using the following method:

* **Tableau personal access token, [PAT][]**
  * `-n`, `--token-name`: Tableau token name
  * `-t`, `--token`: Tableau token

#### Other Arguments

* `-b`, `--server-url`: Tableau base URL, your API endpoint, usually your Tableau URL homepage. For example: `<https://eu-west-1a.online.tableau.com>`
* `-i`, `--site-id`: Tableau Site ID, is empty if your site is the default one
* `-o`, `--output`: target folder to store the extracted files

You can also get help with `--help`.

### Scheduling and Push to Catalog

When moving out of trial, you'll want to refresh your Tableau content in the Catalog. Here is how to do it:

* Your source ID provided by Catalog, referred to as `source_id` in the code examples
* Your Catalog Token given by Catalog

We recommend using the `castor-upload` command:

```bash
castor-upload [arguments]
```

The Catalog team will provide you with:

1. **`Catalog Identifier`:** an ID to match your Tableau files with your Catalog instance
2. **`Catalog Token`:** an API token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

* `-k`, `--token`: Token provided by Catalog
* `-s`, `--source_id`: account id provided by Catalog
* `-t`, `--file_type`: source type to upload. Currently supported are { `DBT` | `VIZ` | `WAREHOUSE` }

#### Target Files

To specify the target files, provide one of the following:

* `-f`, `--file_path`: to push a single file
* `-d`, `--directory_path`: to push several files at once

The tool uploads all files in the given directory. Make sure it contains only the extracted files before pushing.

Then you'll have to schedule the script run and the push to the Catalog. Use your preferred scheduler to create this job.

## Troubleshooting

These sections cover common issues with Tableau sign-in, lineage, and extraction.

### Sign-In or Authentication Errors

The Catalog Tableau ingestion signs in to your Tableau Server or Tableau Cloud instance using the Tableau REST API. If credentials, site details, or network access are wrong, extraction can fail before any metadata is collected. Logs from the extraction path might reference the Tableau Server Client for Python and show a sign-in failure such as `FailedSignInError` with `401001: Signin Error`. Treat that pattern as a Tableau authentication or connectivity problem for this integration, then work through the checks below.

#### Match Errors to the Right System

Match the failing integration and product in the log line. Numeric codes alone are not enough to tell which system returned an error.

Use this flow when sign-in fails for Catalog-managed or client-managed extraction:

1. **Confirm credentials outside Catalog** - Sign in to Tableau with the same account, PAT name, and token value your integration uses, or run a small REST sign-in test your security team approves. If sign-in fails there, fix the account or token in Tableau before you change anything in Catalog.
2. **Refresh or replace PAT values** - If you use a [PAT][], create a new token in Tableau when the old one expires, is revoked, or was rotated without updating Catalog. Update `tokenName` and `token` together so they match the new token in Tableau.
3. **Check `serverUrl` and `siteId`** - `serverUrl` must be the HTTPS base URL you use to reach Tableau, with the correct hostname and region for Tableau Cloud, or the correct server host for Tableau Server. Avoid typos, mixing `http` and `https`, and extra characters at the end of the URL unless your Tableau admin documents a specific format. `siteId` must be the site name from the Tableau URL. For the default site, use `Default` as described in [Catalog Managed](#catalog-managed).
4. **Verify allowlisted egress** - Confirm the Catalog egress IPs in [Allowlist Catalog IP](#allowlist-catalog-ip) are allowlisted on your side if traffic is restricted, so Catalog can reach your Tableau sign-in endpoint.
5. **Client-managed extractor** - If you run `castor-extract-tableau` on your infrastructure, PAT, `serverUrl`, and `siteId` rules are the same. Password-based sign-in can fail after a password change until you update stored credentials or environment variables. See the [castor-extractor Tableau package documentation][] for flags and environment variables.
6. **Retry ingestion** - After you update credentials or network rules, run or schedule extraction again.

If sign-in still fails after these steps, contact your Catalog point of contact with the exact error text, including the sign-in failure line, and the time stamp of the failed run. For REST error reference material from Tableau, see [Tableau REST API][].

### Lineage Is Missing or Inconsistent After Metadata Refresh

After a Tableau metadata refresh, dashboard metadata and field metadata may update correctly, while lineage is incomplete or inconsistent for assets in the same workbook or site. Metadata extraction and lineage processing happen in separate steps, so they don't always finish at the same time.

If lineage is missing for some assets after a refresh:

1. Wait for the next scheduled sync cycle to complete. Lineage processing can lag behind metadata extraction.
2. Make sure the Metadata API is enabled on your Tableau Server. If you haven't set it up, follow the steps to [enable the Tableau Metadata API][].
3. Confirm that your Tableau Server version is 2019.3 or later. Older versions don't support table-level lineage.
4. Check that the `serverUrl` and `siteId` in your credentials match the site where the affected assets are published.
5. If the issue persists after a full sync cycle, reach out to your Catalog point of contact with the affected workbook URLs and the time stamp of the last successful sync.

### Client-Managed Extraction Timeouts

Client-managed Tableau extractions can time out when processing a large number of assets or when the Tableau server responds slowly. Symptoms include DAG failures, incomplete extraction logs, or missing metadata in Catalog after a scheduled run.

To resolve extraction timeouts:

1. Configure separate extraction runs for different databases or schemas instead of extracting everything in a single run.
2. Use `--skip-columns` and `--skip-fields` to reduce the extraction payload if you don't need column-level or field-level metadata.
3. Stagger extraction schedules so multiple integrations don't run at the same time.
4. Check the extractor logs for specific error messages and time stamps.
5. Confirm that your [PAT][] is valid and that the Metadata API is healthy on your Tableau Server.
6. If timeouts continue, reach out to your Catalog point of contact with the failure time stamps and error messages from the extractor logs.

---

[enable the Tableau Metadata API]: https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#enable-the-tableau-metadata-api-for-tableau-server
[PAT]: https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor PyPI page]: https://pypi.org/project/castor-extractor/#pip-install
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[castor-extractor Tableau package documentation]: /docs/catalog/developer/castor-extractor/bi-tools/tableau
[Tableau REST API]: https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api.htm

---

## ThoughtSpot

## Requirements

:::warning[Warehouse Integration Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

For all ThoughtSpot clients:

- **ThoughtSpot admin credentials**

You must create a user in your ThoughtSpot instance that has "Can administer ThoughtSpot" privilege.

## If Needed, Whitelist Catalog IP

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

## Catalog Managed

You can upload your credentials directly in the App when creating your ThoughtSpot integration.

We need:

```json
{
    "base_url": "https://*****.thoughtspot.cloud",
    "username": "Catalog",
    "password": "*****"
}
```

For your first sync, it will take up to 48 h and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, please continue to [Client Managed](#client-managed).

## Client Managed

### Doing a One Shot Extract

For your trial, you can simply give us a one shot view of your BI tool.

To get things working quickly, here is a [Google Colab][] to run our package swiftly.

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[thoughtspot]
```

For further details, see the [castor-extractor installation instructions][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-thoughtspot [arguments]
```

Example:

```bash
castor-extract-thoughtspot -b "https://*****.thoughtspot.cloud" -u "Catalog" -p "*****" -o "/output/directory"
```

#### Credentials

**User and password:**

- `-u`, `--username`: ThoughtSpot username
- `-p`, `--password`: ThoughtSpot password

#### Other Arguments

- `-b`, `--server-url`: ThoughtSpot base URL, your API endpoint, usually your ThoughtSpot URL homepage (for example, `https://*****.thoughtspot.cloud`)
- `-o`, `--output`: target folder to store the extracted files

:::info[Help]

You can also get help with the argument `--help`.

:::

### Scheduling and Push to Catalog

When moving out of trial, you will want to refresh your ThoughtSpot content in Catalog. Here is how to do it:

- Your source id provided by Catalog, referred as `source_id` in the code examples
- Your Catalog Token given by Catalog

We recommend using the `castor-upload` command:

```bash
castor-upload [arguments]
```

The Catalog team will provide you with:

1. `Catalog Identifier` (an id for us to match your ThoughtSpot files with your Catalog instance)
2. `Catalog Token` - An API Token

#### Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: source id provided by Catalog
- `-t`, `--file_type`: source type to upload (`VIZ`)

#### Target Files

To specify the target files, provide **one** of the following:

- `-f`, `--file_path`: to push a single file

or

- `-d`, `--directory_path`: to push several files at once

:::warning[Directory Contents]

The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will have to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

You are done.

[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Zoho

Integrate Zoho Analytics with Catalog to sync dashboards, reports, and semantic metadata. You can use Catalog-managed OAuth credentials or prepare uploads yourself using the BI Importer format. Catalog-managed Zoho onboarding is still maturing and involves OAuth codes that expire quickly, so this page walks through credential JSON, regional server hosts, handoff options, and frequent OAuth failures.

## Requirements

You must be a Zoho Analytics **Account Administrator** to create OAuth 2.0 credentials such as Client ID and Client Secret for the APIs Catalog calls.

## Catalog Managed

Use this section when Catalog stores your Zoho credentials and runs scheduled sync for your Workspace.

:::info[Integration in progress]

Catalog-managed Zoho onboarding is still maturing. Plan credential handoff quickly after you generate OAuth codes because Self Client authorization codes expire within minutes. For a self-contained path you fully control today, use [Client Managed][] with BI Importer.

:::

### Regional Server URI Examples

Pick the `server_uri` host from the same region where your Zoho Analytics account lives. Zoho publishes the authoritative list in [Zoho Server URI documentation][]. Common examples:

| Region or deployment | Example `server_uri` host |
| -------------------- | ------------------------- |
| United States | `analyticsapi.zoho.com` |
| Europe | `analyticsapi.zoho.eu` |
| India | `analyticsapi.zoho.in` |
| Australia | `analyticsapi.zoho.com.au` |
| Japan | `analyticsapi.zoho.jp` |
| Canada | `analyticsapi.zohoanalytics.ca` |

Always confirm the latest host names in Zoho's documentation before go-live, especially if Zoho adds regions or changes endpoints.

### Prepare OAuth Values

Gather the following before you paste JSON into Catalog or Safenote:

- **Server URI** - Pick the host name from Regional Server URI Examples that corresponds to your account region.
- **Client ID** and **Client Secret** from your Self Client registration.
- **Authorization code** from the Self Client consent flow ([Zoho self client documentation][]).

Self Client settings commonly include:

- **Client Type** - Self Client
- **Scope** - `ZohoAnalytics.metadata.read`
- **Time Duration** - 3 hours, the maximum Self Client duration Zoho documents for this flow

See [Zoho Analytics API Documentation][] for the latest OAuth steps.

### Send Credentials to Catalog

You can send credential bundles through [Safenote][] over Slack or email when your onboarding asks for it. Use JSON shaped like this, replacing placeholders with your real values:

```json
{
    "server_uri": "analyticsapi.zoho.com",
    "client_id": "1234",
    "client_secret": "1234",
    "code": "1234"
}
```

Use the host name that matches your Zoho data center, not the placeholder `analyticsapi.zoho.XXX` literal string.

For your first sync, allow up to 48 hours after Catalog confirms credentials. Catalog notifies you when the initial sync completes.

If you prefer not to share credentials with Catalog operations, continue to [Client Managed][].

## Client Managed

### Doing a One-Time Extract

For a trial or one-time snapshot, assemble Zoho Analytics metadata into [BI Importer expected format][], then upload using the channel your Catalog contact provides.

The Catalog extractor package published on PyPI includes dedicated command-line entry points for several BI platforms. There is not yet a published `castor-extract-zoho` command in that package. Until a Zoho-specific command ships, rely on BI Importer formatting and upload flows. You can use the shared [Google Colab][] notebook to explore extraction patterns for other tools while you build Zoho exports manually or with custom scripts.

### Zoho to Catalog Translation

:::info[Zoho element mapping]

Zoho elements map to Catalog `dashboard_type` values as follows:

- **Zoho Table** - `VIZ_MODEL`
- **Zoho Dashboard** - `DASHBOARD`
- **Zoho Report** - `TILE`

:::

## Common Issues

For redirect URI mismatches, scope errors, and the general flow to update credentials in Catalog after a vendor-side OAuth change, see [Integration OAuth troubleshooting][]. Then use the Zoho-specific checks below when OAuth handoff fails or metadata looks wrong in Catalog:

- **Invalid or expired OAuth code** - Self Client authorization codes expire within minutes. Generate a fresh code in Zoho, then paste it into Safenote or Catalog immediately. If your team cannot complete handoff the same day, regenerate the code right before you send it.
- **Wrong data center** - Symptoms include DNS failures or authentication errors against `analyticsapi.zoho.*`. Match `server_uri` to the region where your Zoho Analytics account was created using [Zoho Server URI documentation][].
- **Insufficient scope** - Confirm the Self Client includes `ZohoAnalytics.metadata.read` so Catalog can read analytics metadata.

## What's Next?

- Browse related setup patterns on [BI Tools][].
- Review file layout and columns for uploads in [BI Importer][].

---

[Integration OAuth troubleshooting]: /docs/catalog/integrations/oauth-troubleshooting
[BI Tools]: /docs/catalog/integrations/data-viz
[BI Importer]: /docs/catalog/developer/catalog-apis/bi-importer
[BI Importer expected format]: /docs/catalog/developer/catalog-apis/bi-importer#file-expected-content
[Client Managed]: #client-managed
[Google Colab]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis?usp=sharing
[Safenote]: https://safenote.co/
[Zoho Analytics API Documentation]: https://www.zoho.com/analytics/api/v2/#oauth
[Zoho self client documentation]: https://www.zoho.com/analytics/api/v2/#self-client
[Zoho Server URI documentation]: https://www.zoho.com/analytics/api/v2/#server-uri

---

## Amazon Athena and Glue

Connect Coalesce Catalog to metadata in the AWS Glue Data Catalog and, when you want richer analytics in Catalog, to optional Amazon Athena and AWS CloudTrail signals.

## Before You Begin

You need the following:

- Permission to create IAM policies and an IAM user in AWS, or another programmatic credential source your organization approves
- The AWS account ID and Region where Glue and, if used, Athena are configured
- Access to your Catalog instance to enter connection details

## How Catalog Uses AWS

Catalog reads warehouse metadata through the Glue API so it can list databases, tables, and columns. That path is the foundation for documenting tables and columns in Catalog.

When you grant additional permissions, Catalog can also use Athena APIs that list and describe query executions and work groups. That supports features such as query popularity, lineage, and other SQL-oriented signals that depend on Athena query metadata. If those permissions are missing, Catalog can still ingest Glue metadata while query-related depth may be reduced or absent.

When you allow CloudTrail `LookupEvents`, Catalog can enrich query activity with information about who ran queries. If CloudTrail lookup isn't allowed, metadata ingestion and many query-listing flows can still work. User-centric enrichment may be limited or omitted.

For query observation, Catalog uses read-oriented Athena APIs. You don't need to grant `athena:StartQueryExecution` or `athena:GetQueryResults` for Catalog to read execution metadata using that design.

:::tip[Glue action coverage]

AWS resource types such as partitions, views, and user-defined functions can require specific Glue read actions. The policies below keep the same Glue read actions that appeared in earlier Coalesce guides. If onboarding fails for uncommon objects, compare your resources against [AWS Glue IAM documentation][] and adjust with your security team.

:::

## Create a Catalog User

Create an IAM identity for programmatic access before you attach the JSON policy you build in [Choose an IAM Scope][].

1. Create an IAM user for Catalog using [AWS IAM user creation instructions][].
2. After you build an IAM policy in [Choose an IAM Scope][], attach that policy to this user.
3. Create an access key for programmatic use and store the secret securely.

## If Needed, Allowlist Catalog IP

Allow outbound connections from Catalog to your AWS APIs from the fixed IP that matches your Catalog host:

- For instances on [app.us.castordoc.com][], use `34.42.92.72`
- For instances on [app.castordoc.com][], use `35.246.176.138`

## Choose an IAM Scope

Use this section to decide how much access to grant:

- **Glue metadata only** - You want tables and columns documented in Catalog with minimal AWS scope. Use the policy in [Glue metadata only][].
- **Glue metadata and Athena query signals** - You also want query popularity, lineage, or similar features that rely on Athena query history. Add the statements in [Add Athena query activity][] to your Glue policy (or use the combined example in [Glue with Athena query activity][]).
- **Glue, Athena, and named principals** - You want user-style attribution on warehouse activity. Add the statement in [Add CloudTrail lookup][] on top of the Athena-enabled policy (or use [Glue with Athena query activity and CloudTrail][]).

Optional scopes change how deep analytics go. They don't necessarily change whether basic tables and columns appear when Glue read access succeeds.

### Glue Metadata Only

This JSON grants read access to Glue Data Catalog objects. Replace `<region>` and `<account_id>` with your values. Set `"Version"` to `2012-10-17`, which is the standard value for IAM policy documents.

:::warning[Replace placeholders]

Replace every `<region>` and `<account_id>` placeholder before you save the policy.

:::

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:BatchGetPartition",
                "glue:SearchTables",
                "glue:GetTableVersions",
                "glue:GetTableVersion",
                "glue:GetUserDefinedFunctions",
                "glue:GetUserDefinedFunction"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account_id>:tableVersion/*/*/*",
                "arn:aws:glue:<region>:<account_id>:table/*/*",
                "arn:aws:glue:<region>:<account_id>:catalog",
                "arn:aws:glue:<region>:<account_id>:database/*"
            ]
        }
    ]
}
```

### Add Athena Query Activity

Append these statements to the policy you use for Glue metadata, or merge them into a single document with your Glue statement. They don't start queries or fetch result rows in S3; they allow listing and describing executions and work groups.

```json
{
    "Effect": "Allow",
    "Action": [
        "athena:ListWorkGroups"
    ],
    "Resource": "*"
},
{
    "Effect": "Allow",
    "Action": [
        "athena:GetWorkGroup",
        "athena:ListQueryExecutions",
        "athena:GetQueryExecution",
        "athena:BatchGetQueryExecution",
        "athena:ListDatabases",
        "athena:GetDatabase",
        "athena:GetDataCatalog",
        "athena:ListTableMetadata",
        "athena:GetTableMetadata"
    ],
    "Resource": [
        "arn:aws:athena:<region>:<account_id>:datacatalog/*",
        "arn:aws:athena:<region>:<account_id>:workgroup/*"
    ]
}
```

Remember to substitute `<region>` and `<account_id>` wherever those placeholders appear in the policy.

### Add CloudTrail Lookup

Append this statement to allow Catalog to call `LookupEvents` for principal enrichment:

```json
{
    "Effect": "Allow",
    "Action": [
        "cloudtrail:LookupEvents"
    ],
    "Resource": [
        "*"
    ]
}
```

## Full Policy Examples

The following examples combine the pieces above so you can copy a single policy when you know your target scope.

### Glue With Athena Query Activity

This example keeps Glue catalog reads and Athena query observation in one policy you can paste as a single document.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:BatchGetPartition",
                "glue:SearchTables",
                "glue:GetTableVersions",
                "glue:GetTableVersion",
                "glue:GetUserDefinedFunctions",
                "glue:GetUserDefinedFunction"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account_id>:tableVersion/*/*/*",
                "arn:aws:glue:<region>:<account_id>:table/*/*",
                "arn:aws:glue:<region>:<account_id>:catalog",
                "arn:aws:glue:<region>:<account_id>:database/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:ListWorkGroups"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:GetWorkGroup",
                "athena:ListQueryExecutions",
                "athena:GetQueryExecution",
                "athena:BatchGetQueryExecution",
                "athena:ListDatabases",
                "athena:GetDatabase",
                "athena:GetDataCatalog",
                "athena:ListTableMetadata",
                "athena:GetTableMetadata"
            ],
            "Resource": [
                "arn:aws:athena:<region>:<account_id>:datacatalog/*",
                "arn:aws:athena:<region>:<account_id>:workgroup/*"
            ]
        }
    ]
}
```

### Glue With Athena Query Activity and CloudTrail

This example adds CloudTrail lookup for principal enrichment on top of Glue metadata and Athena query observation.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:BatchGetPartition",
                "glue:SearchTables",
                "glue:GetTableVersions",
                "glue:GetTableVersion",
                "glue:GetUserDefinedFunctions",
                "glue:GetUserDefinedFunction"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account_id>:tableVersion/*/*/*",
                "arn:aws:glue:<region>:<account_id>:table/*/*",
                "arn:aws:glue:<region>:<account_id>:catalog",
                "arn:aws:glue:<region>:<account_id>:database/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:ListWorkGroups"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:GetWorkGroup",
                "athena:ListQueryExecutions",
                "athena:GetQueryExecution",
                "athena:BatchGetQueryExecution",
                "athena:ListDatabases",
                "athena:GetDatabase",
                "athena:GetDataCatalog",
                "athena:ListTableMetadata",
                "athena:GetTableMetadata"
            ],
            "Resource": [
                "arn:aws:athena:<region>:<account_id>:datacatalog/*",
                "arn:aws:athena:<region>:<account_id>:workgroup/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudtrail:LookupEvents"
            ],
            "Resource": "*"
        }
    ]
}
```

## Add Your Connection Details on Catalog

On the Catalog integration screen for Amazon Athena and AWS Glue, enter credentials in this format:

```json
{
    "aws_region": "<your_region>",
    "aws_account_id": "<your_account>",
    "access_key_id": "<your_key_id>",
    "access_key_secret": "<your_secret>"
}
```

Replace each placeholder with your real Region, account ID, and key material.

## What's Next?

- Return to the [Integrations][] hub to browse other Catalog connectors.
- Review [AWS Glue IAM documentation][] and [Amazon Athena IAM documentation][] when you tighten resource patterns in your policies.

---

[Integrations]: /docs/catalog/integrations
[Choose an IAM Scope]: #choose-an-iam-scope
[Glue metadata only]: #glue-metadata-only
[Add Athena query activity]: #add-athena-query-activity
[Add CloudTrail lookup]: #add-cloudtrail-lookup
[Glue with Athena query activity]: #glue-with-athena-query-activity
[Glue with Athena query activity and CloudTrail]: #glue-with-athena-query-activity-and-cloudtrail
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[AWS Glue IAM documentation]: https://docs.aws.amazon.com/glue/latest/dg/set-up-iam.html
[Amazon Athena IAM documentation]: https://docs.aws.amazon.com/athena/latest/ug/managed-policies.html
[AWS IAM user creation instructions]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html

---

## BigQuery Set Up

Set up BigQuery with the Catalog. Create a Google Service Account, grant the required roles, and add credentials to the Catalog App.

## Which Rights Do You Need To Complete the Onboarding?

In order to create the Google Service account, you need to have at minimum one of the following roles:

* Service Account Admin
* Editor Basic

## What Will Catalog Do With This Access?

These roles let Catalog read your metadata by querying BigQuery system metadata. Catalog can see the data model, the queries, and the users in that metadata. This access does not let Catalog read your table data.

## Need Some More Info?

For more on Google Cloud service accounts and IAM roles, see:

* [Service account role documentation][]
* [Roles and permission documentation][]

## 1. Whitelist Catalog's IP on BigQuery

Needed only if you've set up network policies, meaning your BigQuery instance only accepts connections coming from specific IPs.

To allow Catalog to connect to BigQuery, you will first need to whitelist Catalog's IP address.

Here are our fixed IPs:

* For instances on [app.us.castordoc.com][]: `34.42.92.72`
* For instances on [app.castordoc.com][]: `35.246.176.138`

BigQuery's documentation on [configuring public IP connectivity][] can be found in the Google Cloud docs.

## 2. Create Catalog User on BigQuery

### 2.1 Create Google Service Account for Catalog

Create a service account for Catalog from the [Google Console][]. See how to [create and manage service accounts][].

Make sure to create and download a JSON key for that service account. See how to [create service account keys][].

### 2.2 Grant Roles to Google Service Account

Grant the needed access to this new service account with the following roles. See how to [grant and change access][].

You can find the documentation on the different roles in the [BigQuery access control docs][].

:::warning[Add Roles for All Projects]

Make sure to add roles for all projects you would like to see in the Catalog App.

:::

* **BigQuery Read Session User**

    The `BigQuery Read Session User` role allows Catalog to create and use read sessions. It allows read capabilities via SQL yet does not grant by itself any data access.

* **BigQuery Metadata Viewer**

    The `metadataViewer` role allows Catalog to fetch the schemas of your data.

* **BigQuery Job User**

    The `jobUser` role is used by Catalog to parse queries and compute lineage and usage stats.

* **BigQuery Resource Viewer**

    The `resourceViewer` is also used by Catalog to parse queries and compute lineage and usage stats.

## 3. Add New Credentials in Catalog App

:::info[Catalog Admin and Service Account Project]

You must be a Catalog admin to add credentials.

If you use a separate Google Cloud project to host service accounts, update the JSON key. Overwrite the `project_id` with a project from which the service user can run queries.

:::

You can enter the newly created credentials in the [Catalog App integrations settings][].

1. Go to **Settings > Integrations**.
2. Click **BigQuery Add**.
3. Add the credentials.

<figure><figcaption>BigQuery integration fields in the Catalog settings.</figcaption></figure>

## Troubleshooting

This section covers common issues and how to resolve them.

### First Ingestion Stays In Progress Longer Than Expected

Large projects, many data sets, or busy query history can extend the first ingestion window. Catalog may still be scanning metadata and query jobs while the UI shows a first-run state.

**Resolution:**

1. Allow at least one full business day before assuming a stall, then confirm the service account still has **BigQuery Job User**, **BigQuery Metadata Viewer**, **BigQuery Read Session User**, and **BigQuery Resource Viewer** on **every** GCP project you want in Catalog.
2. Verify the JSON key `project_id` matches a project the service account can run jobs from (see the note under **Add New Credentials in Catalog App**).
3. If status does not change after roles and project IDs are correct, contact [Coalesce Support][] with your Catalog region and approximate start time of the integration.

### Cannot Connect After Saving Credentials

Network policies or VPC restrictions may block Catalog's outbound IPs from reaching BigQuery.

**Resolution:**

1. Confirm you followed [Whitelist Catalog's IP on BigQuery][] and allowed both Catalog fixed IPs for your hostname, whether you use `app.castordoc.com` or `app.us.castordoc.com`.
2. Retry credentials in **Settings > Integrations** after the network team confirms the allowlist.
3. If errors persist, capture the error text from the integration screen and contact [Coalesce Support][].

### Lineage or Usage Looks Empty Right After Setup

Query parsing and usage statistics depend on successful job metadata reads in BigQuery.

**Resolution:**

1. Keep **BigQuery Job User** and **BigQuery Resource Viewer** on the projects where analysts run queries.
2. Wait for ingestion to finish a full cycle so jobs from the last day can appear in Catalog's query sampling windows.
3. For ongoing gaps on specific tables, verify table-level metadata is visible to the Catalog service account in the BigQuery console.

### Wrong Projects or Data Sets Appear in Catalog

Roles might be granted on only one project while analysts use several.

**Resolution:**

1. Revisit **Grant Roles to Google Service Account** and apply the same role bundle to each GCP project that should appear.
2. Remove or narrow scope only after confirming with your governance team so you do not hide production assets unexpectedly.

### Credentials Rejected After Rotating the Service Account Key

Catalog stores the JSON key you uploaded; a rotated key in Google Cloud alone does not update Catalog.

**Resolution:**

1. Download the new JSON key from Google Cloud and replace the full credential payload in **Settings > Integrations > BigQuery**.
2. Save and trigger a connection test if your UI offers one.
3. Revoke the old key in Google Cloud after Catalog validates successfully.

---

## Appendix

### Queries Run by the Catalog User

* Databases

    ```sql
    WITH catalogs AS (
        SELECT DISTINCT catalog_name
        FROM `{project}.region-{region}.INFORMATION_SCHEMA.SCHEMATA`
    )

    SELECT
        catalog_name AS database_id,
        catalog_name AS database_name
    FROM catalogs
    ```

* Schemas

    ```sql
    SELECT
        catalog_name AS database_id,
        catalog_name AS database_name,
        schema_name,
        schema_owner,
        creation_time,
        last_modified_time,
        location,
        CONCAT(catalog_name, '.', schema_name) AS schema_id
    FROM `{project}.region-{region}.INFORMATION_SCHEMA.SCHEMATA`
    ```

* Tables

    ```sql
    WITH d AS (
        SELECT
            table_catalog,
            table_schema,
            table_name,
            option_value AS `comment`
        FROM
            `{project}.region-{region}.INFORMATION_SCHEMA.TABLE_OPTIONS`
        WHERE TRUE
            AND option_name = 'description'
            AND option_value IS NOT NULL
            AND option_value != ''
            AND option_value != '""'
    ),
    t AS
    (
        SELECT
            table_catalog,
            table_schema,
            table_name,
            option_value AS tags
        FROM
            `{project}.region-{region}.INFORMATION_SCHEMA.TABLE_OPTIONS`
        WHERE TRUE
            AND option_name = 'labels'
            AND option_value IS NOT NULL
            AND option_value != ''
            AND option_value != '""'
    ),

    dt AS
    (
        SELECT
            catalog_name,
            schema_name,
            option_value AS tags
        FROM
            `{project}.region-{region}.INFORMATION_SCHEMA.SCHEMATA_OPTIONS`
        WHERE TRUE
            AND option_name = 'labels'
            AND option_value IS NOT NULL
            AND option_value != ''
            AND option_value != '""'
    )

    SELECT
        i.table_catalog AS database_name,
        i.table_catalog AS database_id,
        i.table_schema AS `schema_name`,
        i.table_name AS table_name,
        i.table_type,
        i.is_insertable_into,
        i.is_typed,
        i.creation_time,
        d.comment,
        CONCAT(
            COALESCE(t.tags, ""),
            COALESCE(dt.tags, "")
        ) AS tags,
        CONCAT(i.table_catalog, '.', i.table_schema) AS schema_id,
        CONCAT(i.table_catalog, '.', i.table_schema, '.', i.table_name) AS table_id
    FROM
        `{project}.region-{region}.INFORMATION_SCHEMA.TABLES` AS i
    LEFT JOIN d ON i.table_catalog = d.table_catalog
                       AND i.table_schema = d.table_schema
                       AND i.table_name = d.table_name
    LEFT JOIN t ON i.table_catalog = t.table_catalog
                    AND i.table_schema = t.table_schema
                    AND i.table_name = t.table_name
    LEFT JOIN dt ON i.table_catalog = dt.catalog_name
                    AND i.table_schema = dt.schema_name
    ```

* Columns

    ```sql
    WITH field_path AS (
        SELECT
            table_catalog,
            table_schema,
            table_name,
            column_name,
            description,
            field_path,
            data_type
        FROM
            `{project}.region-{region}.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS`
    )

    SELECT
        c.table_catalog AS database_id,
        c.table_catalog AS database_name,
        c.table_schema AS `schema_name`,
        c.table_name AS table_name,
        f.field_path AS column_name,
        c.ordinal_position,
        c.is_nullable,
        f.data_type,
        c.is_generated,
        c.generation_expression,
        c.is_stored,
        c.is_hidden,
        c.is_updatable,
        c.is_system_defined,
        c.is_partitioning_column,
        c.clustering_ordinal_position,
        f.description AS `comment`,
        CONCAT(c.table_catalog, '.', c.table_schema) AS schema_id,
        CONCAT(c.table_catalog, '.', c.table_schema, '.', c.table_name) AS table_id,
        CONCAT(c.table_catalog, '.', c.table_schema, '.', c.table_name, '.', f.field_path) AS column_id
    FROM
        `{{project}}.region-{{region}}.INFORMATION_SCHEMA.COLUMNS` AS c
        LEFT JOIN field_path AS f ON
                c.table_catalog = f.table_catalog
                AND c.table_schema = f.table_schema
                AND c.table_name = f.table_name
                AND c.column_name = f.column_name
    WHERE TRUE
        AND c.column_name != '_PARTITIONTIME'
    ```

* Users

    ```sql
    SELECT DISTINCT user_email AS user_email
    FROM `{project}.region-{region}.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
    WHERE TRUE
        AND DATE(creation_time) >= DATE_ADD(CURRENT_DATE, INTERVAL -30 DAY)
    ```

* Queries

    ```sql
    SELECT
        job_id AS query_id,
        creation_time,
        project_id AS database_name,
        user_email AS user_name,
        user_email AS user_id,
        job_type,
        statement_type,
        priority,
        start_time,
        end_time,
        query AS query_text,
        state,
        reservation_id,
        total_bytes_processed,
        total_bytes_billed,
        total_slot_ms,
        error_result,
        cache_hit,
        destination_table,
        referenced_tables,
        labels,
        parent_job_id
    FROM `{project}.region-{region}.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
    WHERE TRUE
        AND DATE(creation_time) = DATE_ADD(CURRENT_DATE, INTERVAL -1 DAY)
        AND EXTRACT(hour FROM creation_time) BETWEEN 0 AND 23
        AND job_type = 'QUERY'
    ```

* View DDL

    ```sql
    SELECT
        table_catalog AS database_name,
        table_schema AS schema_name,
        table_name AS view_name,
        '{{"project_id": "' || table_catalog || '", "dataset_id": "' || table_schema || '", "table_id": "' || table_name || '"}}' as destination_table,
        view_definition
     FROM `{project}.{dataset}.INFORMATION_SCHEMA.VIEWS`
    ```

### References

Here's a list of all Google Cloud documentation referenced above:

* [configuring public IP connectivity][]
* [Service account role documentation][]
* [Roles and permission documentation][]
* [grant and change access][]

[Service account role documentation]: https://cloud.google.com/iam/docs/understanding-roles#service-accounts-roles
[Roles and permission documentation]: https://cloud.google.com/iam/docs/understanding-roles
[configuring public IP connectivity]: https://cloud.google.com/sql/docs/mysql/configure-ip
[Google Console]: https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts?supportedpurview=project
[create and manage service accounts]: https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating
[create service account keys]: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys
[grant and change access]: https://cloud.google.com/iam/docs/granting-changing-revoking-access#access-control-via-console
[BigQuery access control docs]: https://cloud.google.com/bigquery/docs/access-control
[Catalog App integrations settings]: https://app.castordoc.com/settings/integrations
[Coalesce Support]: mailto:support@coalesce.io
[Whitelist Catalog's IP on BigQuery]: #1-whitelist-catalogs-ip-on-bigquery
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/

---

## BigQuery(Bigquery)

## Setup

See the BigQuery setup and extraction guides in this section.

## What Catalog Extracts From BigQuery

Catalog extracts and displays in the app:

- All your data assets from Database to Tables and their Columns
  - Tables and Columns with their descriptions
- Lineage between Tables and Dashboards
- Lineage between Columns and Dashboards
- Table tags

---

## Databricks(Data-warehouses)

:::info[Unity Catalog and Hive]

This integration covers both Unity Catalog and non-Unity Catalog (Hive) Databricks.

:::

:::info[Per Workspace]

This integration is to be replicated for each workspace you want to integrate with Catalog.

:::

## Requirements

- You must be a Databricks administrator and Metastore Admin of the workspace to integrate.
- OAuth M2M authentication requires a service principal with appropriate permissions.

To create a service principal, follow the instructions for your cloud provider:

- [Amazon Web Services][]
- [Google Cloud Platform][]
- [Microsoft Azure][]

## 1. Generate OAuth Credentials

Once you have created a service principal:

1. Click on the service principal.
2. Navigate to the **OAuth secrets** tab.
3. Click **Generate secret**.
4. **Important**: Copy both the **Client ID** and **Client Secret** immediately - the secret will only be shown once.

## 2. Retrieve Your `host`

Your `host` or instance name can be found in your Databricks URL: `https://<instance-name>.cloud.databricks.com` or `https://<instance-name>/`.

## 3. Retrieve Your HTTP path

Your `http_path` identifies the compute resource that Coalesce will use to query metadata. The location differs between SQL warehouses and clusters. For more details, see the [Databricks compute details documentation][].

**For Clusters:**

1. Log in to your Databricks workspace.
2. In the sidebar, click **Compute**.
3. Select your cluster.
4. On the **Configuration** tab, expand **Advanced options**.
5. Click the **JDBC/ODBC** tab and copy the **HTTP path**.

**For SQL Warehouses:**

1. Log in to your Databricks workspace.
2. In the sidebar, click **SQL Warehouses**.
3. Select your SQL warehouse.
4. On the **Connection details** tab, copy the **HTTP path**.

## 4. Enable System Tables

To enable `system` tables, follow the [Databricks system tables documentation][].

After enabling system tables, grant your service principal access to the `system` catalog:

1. In your Databricks workspace, go to **Catalog** and navigate to the `system` catalog.
2. Click **Permissions** tab → **Grant**.
3. Select your service principal and grant: **USE CATALOG**, **USE SCHEMA**, and **SELECT**.

## 5. Add Connection to Coalesce Catalog

Now that you have your OAuth credentials, add the Databricks integration to Coalesce:

1. Go to your [integration page][] in Coalesce.
2. Click **Add Integration** and select **Databricks**.

   <figure><figcaption></figcaption></figure>

3. Select **Catalog Managed** and name your integration.
4. Fill in the credential fields:
   - **Host**: Your Databricks workspace hostname (for example, `https://dbc-abc12345-6789.cloud.databricks.com`)
   - **HTTP Path**: Your SQL warehouse path (for example, `/sql/1.0/warehouses/xxxxx`)
   - **Client ID**: The OAuth client ID from your service principal
   - **Client Secret**: The OAuth client secret from your service principal

<figure><figcaption></figcaption></figure>

For your first sync, it will take up to 48 hours and we will notify you when it is complete.

[Amazon Web Services]: https://docs.databricks.com/aws/en/admin/users-groups/service-principals
[Google Cloud Platform]: https://docs.databricks.com/gcp/en/admin/users-groups/service-principals
[Microsoft Azure]: https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/service-principals
[Databricks compute details documentation]: https://docs.databricks.com/aws/en/integrations/compute-details
[Databricks system tables documentation]: https://docs.databricks.com/en/admin/system-tables/index.html#enable
[integration page]: https://app.castordoc.com/settings/integrations

---

## Domo Data

## Requirements

For all Domo clients:

- Domo admin credentials
- Domo Developer Token
- Communication around the use of Domo as a warehouse

## If Needed, Whitelist Catalog IP

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

## Catalog Managed

To get things started with Domo in Catalog, you will need to be a Domo admin and provide us with:

- A `Client ID` - Domo [documentation][]
- A Domo `API token` (Client Secret) - Domo [documentation][]
- The `Base URL` of your Domo instance (`https://XXXX.domo.com/`)
- A `Developer Token` - Domo [developer token documentation][]
- The `Cloud Id` of your external Warehouse (if applicable, remove otherwise)

For more information on how to retrieve those values, see the [Domo API quickstart][] or reach out to the team.

```json
{
    "apiToken": "*****",
    "clientId": "*****",
    "baseUrl": "https://XXXX.domo.com/",
    "developerToken": "*****",

    "cloudId": "my-external-warehouse-cloud-id"
}
```

:::warning[SSO and Direct Sign-On]

The Catalog account needs to have direct sign on enabled, if SSO is activated on your Domo instance.

:::

- `clientId`: id of the Catalog account
- `baseUrl`: Domo base URL, your API endpoint, usually your Domo URL homepage
- `developerToken`: Token to access the Domo private API
- `cloudId`: Id of the external warehouse used within Domo (if applicable, remove otherwise)

:::danger[Dashboard Access]

The account used to log into the developer portal to generate the client id and secret needs to have access to all the dashboards you would like to have in Catalog.

You can go in the Admin > Dashboards section in Domo and share the needed dashboards with that user.

:::

For your first sync, it will take up to 48 h and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, please continue to [Client Managed](#client-managed).

## Client Managed

### Doing a One Shot Extract

For your trial, you can simply give us a one shot view of your BI tools.

### Running the Catalog Package

#### Install the PyPI Package

```bash
pip install castor-extractor
```

For further details, see the [castor-extractor installation instructions][].

#### Run the Package

Once the package has been installed, you should be able to run the following command in your terminal:

```bash
castor-extract-domo [arguments]
```

#### Arguments

- `-c`, `--client-id`: Domo client id
- `-a`, `--api-token`: Domo access token
- `-d`, `--developer-token`: Domo developer token
- `-b`, `--base-url`: Domo base URL
- `-o`, `--output`: target folder to store the extracted files

#### Optional Arguments

- `-C`, `--cloud-id`: Mandatory if you use an external warehouse (and not Domo as a warehouse)

### Scheduling and Push to Catalog

When moving out of trial, you will want to refresh your Domo content in Catalog. Here is how to do it:

The Catalog team will provide you with:

1. `Catalog Identifier` (an id for us to match your Domo files with your Catalog instance)
2. `Catalog Token` - An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Castor-Upload Arguments

- `-k`, `--token`: Token provided by Catalog
- `-s`, `--source_id`: account id provided by Catalog
- `-t`, `--file_type`: source type to upload. Currently supported are `DBT`, `VIZ`, or `WAREHOUSE`

#### Target Files

To specify the target files, provide **one** of the following:

- `-f`, `--file_path`: to push a single file

or

- `-d`, `--directory_path`: to push several files at once

:::warning[Directory Contents]

The tool will upload **all files** included in the given directory.

Make sure it contains only the extracted files before pushing.

:::

Then you will have to schedule the script run and the push to Catalog. Use your preferred scheduler to create this job.

You are done.

[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[documentation]: https://developer.domo.com/portal/1845fc11bbe5d-api-authentication#step-1-create-client-id-and-secret
[developer token documentation]: https://domo-support.domo.com/s/article/360042934494?language=en_US
[Domo API quickstart]: https://developer.domo.com/portal/8ba9aedad3679-ap-is#quickstart
[castor-extractor installation instructions]: https://pypi.org/project/castor-extractor/#pip-install

---

## Warehouses

To integrate your warehouse, create a dedicated user (with specific rights to enable metadata access without accessing your data) and share the credentials in the [Catalog App integration settings][].

## Supported Technologies

<DocCardList />

## What If a Warehouse Technology Is Not Covered

If your warehouse technology is not listed as one of the integrations, you can use the Warehouse Importer to provide the necessary metadata:

<ButtonLink to="/docs/catalog/developer/catalog-apis/warehouse-importer">
  Warehouse Importer
</ButtonLink>

We provide a list of the necessary metadata we need for content to show up in Catalog. Explore the metadata you can access and format it into our templates.

If you are feeling overwhelmed by the magnitude of the task, try filling in a couple of examples by hand. This way, you can test Catalog's main functionalities.

Please reach out to the sales or ops team for more details.

[Catalog App integration settings]: https://app.castordoc.com/settings/integrations

---

## MySQL(Data-warehouses)

Connect Catalog to MySQL to sync metadata from your databases. You can use Catalog-managed credentials or extract metadata for client-managed upload.

## Catalog Managed

Use Catalog-managed connectivity when Catalog holds the credentials and connects to MySQL directly.

### Requirements

You should have:

- Super-user privileges so you can run administrative commands on MySQL.
- MySQL version 5.7 or higher.

### 1. Whitelist Catalog's IP on MySQL

To allow Catalog to connect to MySQL, you will first need to whitelist Catalog's IP address.

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

Catalog connects directly to your MySQL databases. As such, it needs to be public. Often, clients give Catalog access to an exact replica of their database.

### 2. Create a Catalog User on MySQL

#### Create the Catalog User

Use a safe password from a password manager.

```sql
CREATE USER 'catalog'@'%' IDENTIFIED BY '<password>';
```

#### Grant Rights to the Catalog User

```sql
GRANT SELECT,SHOW VIEW,EXECUTE ON *.* TO 'catalog'@'%';
```

On MySQL 5.7 you need `EXECUTE` for certain metadata paths. Newer releases may not require it.

### 3. Add New Credentials in Catalog App

:::info[Catalog admin required]

You must be a Catalog admin to add credentials.

:::

You can now enter the newly created credentials in the [Catalog App][].

- Go to **Settings > Integrations**.
- Click **MySQL Add**.
- Add the credentials using the following format:

```json
{
    "host": "host",
    "port": "port",
    "user": "user",
    "password": "password"
}
```

- `host`: MySQL Host
- `port`: MySQL Port
- `user`: MySQL User
- `password`: MySQL Password

## Client Managed

Use client-managed extraction when you generate CSV extracts and upload them yourself.

### One-Time Extraction

For your trial, you can give us a one-shot view of your MySQL metadata. Follow the [Warehouse importer][] specification for the expected format and upload steps.

Use the following queries to extract the necessary information for each file:

- `database.csv`

```sql
SELECT 
  CATALOG_NAME AS `id`,
  CATALOG_NAME AS `database_name`
FROM INFORMATION_SCHEMA.SCHEMATA
GROUP BY 1, 2
```

- `schema.csv`

```sql
SELECT 
  CONCAT(CATALOG_NAME, ".", SCHEMA_NAME) AS `id`,
  CATALOG_NAME AS `database_id`,
  SCHEMA_NAME AS `schema_name`,
  "" AS `description`,
  "" AS `tags`
FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME NOT IN ("INFORMATION_SCHEMA")
```

- `table.csv`

```sql
SELECT 
  CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME) AS `id`,
  CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA) AS `schema_id`,
  TABLE_NAME AS `table_name`,
  TABLE_COMMENT AS `description`,
  "" AS `tags`,
  CASE 
    WHEN TABLE_TYPE = "BASE TABLE" THEN "TABLE"
    WHEN TABLE_TYPE = "SYSTEM VIEW" THEN "VIEW"
    ELSE TABLE_TYPE
  END AS `type`
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA NOT IN ("INFORMATION_SCHEMA")
```

- `column.csv`

```sql
SELECT 
  CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME, ".", COLUMN_NAME) AS `id`,
  CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME) AS `table_id`,
  COLUMN_NAME AS `column_name`,
  COLUMN_COMMENT AS `description`,
  CASE
    WHEN DATA_TYPE IN ("char", "varchar") THEN "STRING"
    WHEN DATA_TYPE IN ("binary", "varbinary", "blob", "tinyblob", "mediumblob", "longblob") THEN "BINARY"
    WHEN DATA_TYPE IN ("text", "tinytext", "mediumtext", "longtext") THEN "TEXT"
    WHEN DATA_TYPE IN ("enum", "set") THEN "ARRAY"
    WHEN DATA_TYPE IN ("bit", "tinyint", "smallint", "mediumint", "int", "bigint", "integer", "long") THEN "INTEGER"
    WHEN DATA_TYPE IN ("bool") THEN "BOOLEAN"
    WHEN DATA_TYPE IN ("float", "real") THEN "FLOAT"
    WHEN DATA_TYPE IN ("double", "double precision", "decimal", "dec", "numeric", "fixed") THEN "DECIMAL"
    WHEN DATA_TYPE IN ("date", "datetime", "year") THEN "DATE"
    WHEN DATA_TYPE IN ("timestamp") THEN "TIMESTAMP"
    WHEN DATA_TYPE IN ("time") THEN "TIME"
    WHEN DATA_TYPE IN ("json") THEN "OBJECT"
    ELSE UPPER(DATA_TYPE)
  END AS `data_type`,
  ORDINAL_POSITION AS `ordinal_position`
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA NOT IN ("INFORMATION_SCHEMA")
```

- `query.csv`

Use the following file. We currently cannot retrieve your queries for this integration:

<ButtonLink to="/downloads/query.csv">
  Query
</ButtonLink>

- `view_ddl.csv`

```sql
SELECT 
  TABLE_CATALOG AS `database_name`,
  TABLE_SCHEMA AS `schema_name`,
  TABLE_NAME AS `view_name`,
  VIEW_DEFINITION AS `definition`
FROM INFORMATION_SCHEMA.VIEWS
```

- `user.csv`

For MySQL 8.0 and higher

```sql
SELECT
  USER AS `id`,
  CONCAT(USER, "@", HOST) AS `email`
FROM INFORMATION_SCHEMA.USER_ATTRIBUTES
GROUP BY 1, 2
```

For other versions of MySQL

```sql
SELECT
  GRANTEE AS `id`,
  GRANTEE AS `email`
FROM INFORMATION_SCHEMA.USER_PRIVILEGES
GROUP BY 1, 2
```

You can upload the resulting files into the [Catalog App][] directly.

### After Your Trial

Generate the 7 files above and push them to our endpoint using the [Catalog Uploader][].

We will provide you with an API secret and a `source_id`.

:::danger[All Files Required]

All 7 files are mandatory and data must be consistent and coherent.

:::

## Troubleshooting

This section covers common issues and how to resolve them.

### Client-Managed Upload Fails Validation or Ingestion Stays Stale

Files produced outside the documented CSV shapes do not match what the warehouse importer expects, which leads to rejected uploads or incomplete refreshes even when extraction finishes.

**Resolution:**

1. Compare every generated file to the shapes described in **Client Managed** on this page and to the [Warehouse importer][] specification for column order and required fields.
2. Regenerate all seven files in one batch so identifiers stay consistent across `table.csv`, `column.csv`, and related extracts.
3. Re-upload through the [Catalog Uploader][] after fixing formats.

### Ingestion Errors Reference Parsing or Schema Drift

MySQL versions differ on catalog tables available for user extracts.

**Resolution:**

1. Run the query for the user CSV file that matches your MySQL major version, as listed earlier on this page.
2. Confirm the query CSV extract follows the placeholder guidance when query logging is not available from MySQL directly.

### Catalog-Managed Connection Fails Immediately

Host allowlisting, wrong host or port, or insufficient grants block login before ingestion starts.

**Resolution:**

1. Verify **Whitelist Catalog's IP on MySQL** includes both fixed IPs for your Catalog hostname.
2. Confirm the Catalog user still has `SELECT`, `SHOW VIEW`, and `EXECUTE` when required after privilege reviews.

### Trial Upload Worked but Production Schedules Fail

Scheduler credentials or paths often differ between trial laptops and production runners.

**Resolution:**

1. Confirm the production job uses the same `castor-upload` token, `source_id`, and file paths documented in **After Your Trial**.
2. Ensure the upload directory contains **only** the seven expected files before each push.

### Mixed JSON Credentials or Special Characters in Passwords

Passwords with quotes or backslashes can break JSON payloads copied into the Catalog App.

**Resolution:**

1. Paste credentials through a password manager export or escape characters according to JSON rules.
2. Rotate to a temporary alphanumeric password to isolate quoting issues, then restore your preferred policy.

---

[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[Catalog App]: https://app.castordoc.com/settings/integrations
[Warehouse importer]: /docs/catalog/developer/catalog-apis/warehouse-importer
[Catalog Uploader]: /docs/catalog/developer/castor-extractor

---

## PostgreSQL

Grant access to your PostgreSQL metadata to Catalog and run a few tests to verify the setup.

## Before You Begin

You need to be a PostgreSQL superuser with the ability to impersonate all schema owners. Your Postgres version should be 11 or later.

When new schemas are created, part of the process will need to be repeated.

Catalog will only be granted USAGE on all schemas and REFERENCES on all tables. That is the minimum role for Catalog to read your metadata without accessing your data.

### More Information

See the PostgreSQL documentation:

- [Privileges and access documentation][]
- [User creation and access to system tables][]

## 1. Whitelist Catalog's IP on Postgres

To allow Catalog to connect to Postgres, you will first need to whitelist Catalog's IP address.

Here are our fixed IPs:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

Catalog connects directly to your Postgres databases. As such, it needs to be public. It is frequent that our clients provide Catalog an access to an exact replica of their database.

## 2. Create Catalog User on Postgres

You are asked to enter credentials in the Catalog App. Here is a step-by-step guide to create them. You can run the SQL code below.

[View the full SQL procedure][]

### 2.1 Create Procedure To Grant Rights on Current Content

- The procedure grants USAGE, CREATE, REFERENCES rights to a user on existing content
- CREATE rights are required to allow REFERENCES to work
- REFERENCES are required for Catalog to see column names

Catalog will not be creating schemas or referencing foreign keys with these rights.

```sql
CREATE OR REPLACE PROCEDURE grant_rights_schema_today(user_name VARCHAR)
AS $$
DECLARE
    row record;
    sch text;
BEGIN
    FOR row IN (
        SELECT nspname
        FROM pg_namespace
        WHERE nspname NOT LIKE 'pg_%'
            AND nspname != 'information_schema'
    ) LOOP
        sch := row.nspname; 
        EXECUTE 'GRANT USAGE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        EXECUTE 'GRANT CREATE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        EXECUTE 'GRANT REFERENCES ON ALL TABLES IN SCHEMA ' || sch || ' TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;
```

### 2.2 Create Procedure To Grant Default Privileges on Future Tables

The procedure sets default REFERENCES on new tables created in schemas that already exist when you run it. It does not grant schema-level USAGE and does not cover schemas you create later. For each schema you'll impersonate the schema owner as only he can execute such procedure

```sql
CREATE OR REPLACE PROCEDURE grant_rights_schema_future(user_name VARCHAR)
AS $$
DECLARE
    row record;
    usr text;
    sch text;
BEGIN
    FOR row IN (
        SELECT distinct tableowner,schemaname
        FROM pg_tables 
        WHERE schemaname not LIKE 'pg_%'
            AND schemaname != 'information_schema'
            AND tableowner != 'rdsadmin'
    ) LOOP
        sch := row.schemaname;
        usr := row.tableowner;
        EXECUTE 'ALTER DEFAULT PRIVILEGES FOR USER "' || usr || '" IN SCHEMA ' || sch 
            || ' GRANT REFERENCES ON TABLES TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;
```

Here is an example of a statement of the loop, for schema `tesla` owned by user `elon`: `ALTER DEFAULT PRIVILEGES FOR USER elon IN SCHEMA tesla GRANT REFERENCES ON TABLES TO catalog`

### 2.3 Create Catalog User

Use a safe password, from a password manager for example.

**Password should not include `@` or `:`.**

```text
CREATE USER "catalog"
    PASSWORD '<password>';
```

### 2.4 Grant Rights to Catalog User

You can finally grant rights to Catalog user with procedures defined in 2.1 and 2.2

You will have to run these commands every time a new schema is created.

```text
CALL grant_rights_schema_today('catalog');
CALL grant_rights_schema_future('catalog');
```

## 3. Test the Catalog User

The Catalog User should now be correctly set up. Before you give Catalog the credentials, let's first run a test to make sure everything works well.

1. Connect to your Postgres cluster using the `catalog` credentials you've just created
2. Run the following checks:

### 3.1 Check Schemas the Catalog User Can See

```sql
SELECT
    db.oid AS database_id,
    db.datname AS database_name,
    n.oid::TEXT AS schema_id,
    n.nspname AS schema_name,
    u.usename AS schema_owner,
    u.usesysid AS schema_owner_id,
    sd.description AS comment
FROM pg_catalog.pg_namespace AS n
CROSS JOIN pg_catalog.pg_database AS db
JOIN pg_catalog.pg_user u ON u.usesysid = n.nspowner
LEFT JOIN pg_catalog.pg_description sd ON sd.classoid = n.oid
WHERE TRUE
    AND db.datname = CURRENT_DATABASE()
    AND n.nspname NOT LIKE 'pg_%%'
    AND n.nspname NOT IN ('catalog_history', 'information_schema')
```

You should get all of your Postgres schemas.

## 4. Add New Credentials in Catalog App

:::info[Catalog Admin Required]

You must be a Catalog admin to do this.

:::

You can now enter the newly created credentials in the [Catalog App][].

- Go to **Settings > Integrations**
- Click **Postgres Add**
- Add the credentials

<figure><figcaption></figcaption></figure>

## Troubleshooting

This section covers common issues and how to resolve them.

### Asset Counts Dropped Sharply After a Run or Maintenance Window

Catalog reconciles what exists in Postgres against what it already indexed. Operations that remove schemas, drop tables, or recreate databases under the same name can look like large deletes until the next successful ingestion.

1. Confirm whether maintenance scripts dropped or renamed schemas that Catalog had indexed.
2. Re-run grants for future content if new schemas replace old ones so the Catalog user keeps REFERENCES visibility.
3. If counts still look wrong while Postgres still holds the data, contact [Coalesce Support][] with timestamps and one example schema name.

### Multiple Postgres Integration Entries Overlap the Same Data

Teams sometimes create second connections during migrations or tests. Overlapping sources can confuse ownership of assets until ingestion settles.

1. Prefer one canonical Postgres source per environment and retire trial credentials once production credentials work.
2. Align hostname, database name, and user so Catalog does not duplicate the same logical instance.

### Column Names Are Missing on Table Pages

REFERENCES on tables is required for Catalog to read column-level metadata.

1. Re-run the grant procedures in **Create Catalog User on Postgres** for schemas that were added after onboarding.
2. Confirm REFERENCES grants succeeded on tables in `information_schema` visibility tests.

### Connection Errors After Password Rotation or Host Change

Catalog stores the credentials you entered during onboarding.

1. Update **Settings > Integrations** with the current password, host, or port.
2. Repeat **Test the Catalog User** queries from this page before relying on new ingestion.

### Postgres Version Is Below 11

Catalog documents a minimum version for supported deployments.

1. Upgrade the database to a supported Postgres release or connect Catalog to a replica that meets the minimum version.
2. If you cannot upgrade yet, discuss options with [Coalesce Support][] before expanding scope.

---

## Appendix

### Full Code

```sql
CREATE OR REPLACE PROCEDURE grant_rights_schema_today(user_name VARCHAR)
AS $$
DECLARE
    row record;
    sch text;
BEGIN
    FOR row IN (
        SELECT nspname
        FROM pg_namespace
        WHERE nspname NOT LIKE 'pg_%'
            AND nspname != 'information_schema'
    ) LOOP
        sch := row.nspname; 
        EXECUTE 'GRANT USAGE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        -- Create is needed references to work
        EXECUTE 'GRANT CREATE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        -- References allow the catalog user to see columns names
        -- It also allows it to reference foreign keys, but we will never do that
        EXECUTE 'GRANT REFERENCES ON ALL TABLES IN SCHEMA ' || sch || ' TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- This second procedures ensure that the catalog 
-- user can get meta data on FUTURE content.
-- Such grant must be done by the future CREATOR
CREATE OR REPLACE PROCEDURE grant_rights_schema_future(user_name VARCHAR)
AS $$
DECLARE
    row record;
    usr text;
    sch text;
BEGIN
    FOR row IN (
        SELECT distinct tableowner,schemaname
        FROM pg_tables 
        WHERE schemaname not LIKE 'pg_%'
            AND schemaname != 'information_schema'
            AND tableowner != 'rdsadmin'
    ) LOOP
        sch := row.schemaname;
        usr := row.tableowner;
        EXECUTE 'ALTER DEFAULT PRIVILEGES FOR USER "' || usr || '" IN SCHEMA ' || sch 
            || ' GRANT REFERENCES ON TABLES TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Create user with syslog access
CREATE USER "catalog"
    PASSWORD '<password>'

-- Grant usage on all schemas
-- You will need to run this everytime a new a new schema is added
CALL grant_rights_schema_today('catalog');
CALL grant_rights_schema_future('catalog');
```

### Queries Run by Catalog

Information below concerns the queries Catalog will run on Postgres with the rights granted above. We only extract metadata for columns, schemas, tables, and roles.

- Databases

```sql
    SELECT
        db.oid AS database_id,
        db.datname AS database_name,
        de.description AS "comment"
    FROM pg_catalog.pg_database AS db
        LEFT JOIN pg_catalog.pg_description AS de ON de.classoid = db.oid
    WHERE db.datname = CURRENT_DATABASE()
```

- Schemas

```sql
    SELECT
        db.oid AS database_id,
        db.datname AS database_name,
        ns.oid::TEXT AS schema_id,
        ns.nspname AS schema_name,
        u.usename AS schema_owner,
        u.usesysid AS schema_owner_id,
        de.description AS "comment"
    FROM pg_catalog.pg_namespace AS ns
        CROSS JOIN pg_catalog.pg_database AS db
        JOIN pg_catalog.pg_user AS u ON u.usesysid = ns.nspowner
        LEFT JOIN pg_catalog.pg_description AS de ON de.classoid = ns.oid
    WHERE TRUE
        AND db.datname = CURRENT_DATABASE()
        AND ns.nspname NOT LIKE 'pg_%%'
        AND ns.nspname NOT IN ('catalog_history', 'information_schema')
```

- Tables

```sql
    WITH ids AS (
        SELECT
            t.oid AS table_id,
            t.relname AS table_name,
            n.nspname AS schema_name,
            n.oid AS schema_id,
            u.usename AS table_owner,
            t.relowner AS table_owner_id,
            td.description AS "comment",
            t.reltuples::BIGINT AS tuples
        FROM pg_class AS t
            JOIN pg_catalog.pg_namespace AS n ON n.oid = t.relnamespace
            LEFT JOIN pg_catalog.pg_description AS td ON td.objoid = t.oid AND td.objsubid = 0
            LEFT JOIN pg_catalog.pg_user AS u ON u.usesysid = t.relowner
        WHERE TRUE
            AND n.nspname NOT LIKE 'pg_%%'
            AND n.nspname NOT IN ('catalog_history', 'information_schema')
            AND t.relam IN (0, 2)
    ),

    meta AS (
        SELECT
            db.datname AS database_name,
            db.oid AS database_id,
            t.table_schema AS schema_name,
            t.table_name AS table_name,
            t.table_type
        FROM information_schema.tables AS t
            CROSS JOIN pg_catalog.pg_database AS db
        WHERE TRUE
            AND db.datname = CURRENT_DATABASE()
            AND t.table_schema NOT LIKE 'pg_%%'
            AND t.table_schema NOT IN ('catalog_history', 'information_schema')
    )

    SELECT
        m.database_name,
        m.database_id,
        m.schema_name,
        i.schema_id,
        m.table_name,
        i.table_id,
        m.table_type,
        i.table_owner,
        i.table_owner_id,
        i.tuples,
        i.comment
    FROM meta AS m
        JOIN ids AS i ON (i.table_name = m.table_name AND i.schema_name = m.schema_name)
```

- Columns

```sql
    WITH ids AS (
        SELECT
            d.datname AS database_name,
            d.oid AS database_id,
            t.oid AS table_id,
            t.relname AS table_name,
            n.nspname AS schema_name,
            n.oid AS schema_id
        FROM
            pg_class AS t
            JOIN pg_catalog.pg_namespace AS n ON n.oid = t.relnamespace
            CROSS JOIN pg_catalog.pg_database AS d
        WHERE
            TRUE
            AND d.datname = CURRENT_DATABASE()
            AND n.nspname NOT LIKE 'pg_%%'
            AND n.nspname NOT IN ('catalog_history', 'information_schema')
            AND t.relam IN (0, 2)
    ),

    columns AS (
        SELECT
            ids.database_name,
            ids.database_id,
            c.table_schema AS schema_name,
            ids.schema_id,
            c.table_name AS table_name,
            ids.table_id,
            c.column_name,
            ids.table_id || '.' || c.column_name AS column_id,
            c.data_type,
            c.ordinal_position,
            c.column_default,
            c.is_nullable,
            c.character_maximum_length,
            c.character_octet_length,
            c.numeric_precision,
            c.numeric_precision_radix,
            c.numeric_scale,
            c.datetime_precision,
            c.interval_precision,
            c.interval_type,
            c.interval_precision,
            d.description AS "comment"
        FROM
            information_schema.columns AS c
            JOIN ids ON c.table_schema = ids.schema_name
                        AND c.table_name = ids.table_name
            LEFT JOIN pg_catalog.pg_description AS d ON d.objoid = ids.table_id
                                                        AND d.objsubid = c.ordinal_position
    )

    SELECT
        *
    FROM
        columns
```

- Users

```sql
    SELECT
        usename AS user_name,
        usesysid AS user_id,
        usecreatedb AS has_create_db,
        usesuper AS is_super,
        valuntil AS valid_until
    FROM pg_catalog.pg_user
```

- Groups

```sql
    SELECT
        groname AS group_name,
        grosysid AS group_id,
        grosysid AS group_list
    FROM pg_group
```

### References

Here's a list of all the Postgres documentation referenced above:

- [Privileges and access documentation][]
- [User creation and access to system tables][]

---

[Privileges and access documentation]: https://www.postgresql.org/docs/13/ddl-priv.html
[User creation and access to system tables]: https://www.postgresql.org/docs/11/app-createuser.html
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[View the full SQL procedure]: /docs/catalog/integrations/data-warehouses/postgres#full-code
[Catalog App]: https://app.castordoc.com/settings/integrations
[Coalesce Support]: mailto:support@coalesce.io

---

## Redshift(Data-warehouses)

Grant access to your Redshift metadata to Catalog and run a few tests to verify the setup.

## Requirements for Onboarding

You need specific permissions to finish onboarding. You should be a super user who can impersonate all schema owners.

When new schemas are created after onboarding, re-run the grant procedures in section 2.4.

## What Catalog Does With This Access

- Catalog is granted USAGE on all schemas and REFERENCES on all tables. That is the minimum access Catalog needs to read your metadata without accessing your table data.
- Catalog is also granted unrestricted SYSLOG access. Catalog needs SYSLOG to view queries run against tables.

## More Information

For details on privileges, access control, and user creation, see the AWS documentation:

- [Privileges and access documentation][]
- [User creation and access to system tables][]
- [SYSLOG and visibility of data in system tables and views][]

## 1. Whitelist Catalog's IP on Redshift

To allow Catalog to connect to Redshift, whitelist Catalog's IP address.

Catalog uses these fixed IPs:

- For instances on [US Catalog app][], use `34.42.92.72`.
- For instances on [EU Catalog app][], use `35.246.176.138`.

### How To Whitelist an IP on Redshift

1. From the **Redshift Dashboard**, click **Clusters**.
2. In the list of clusters, choose your cluster.
3. In the Configuration tab of the clusters detail page, under the VPC Security Groups section, click the name of the security group.
4. In the security group view, select the Inbound tab on the bottom half of the page, then in that tab click Edit.
5. In the Edit inbound rules dialog, add the IP addresses that Catalog can use to access the cluster. To add a new rule, click **Add Rule** at the bottom of the list, and set the IP address for your Catalog hostname.

## 2. Create Catalog User on Redshift

You'll enter credentials in the Coalesce App. Follow these steps to create them, or run the SQL below. For the full script in one place, see [Full SQL procedure in the Appendix][].

### 2.1 Create Procedure To Grant Rights on Existing Schemas

The procedure grants USAGE, CREATE, and REFERENCES on all schemas that exist when you run it, plus REFERENCES on all tables in those schemas.

- CREATE rights are required so REFERENCES can work.
- REFERENCES rights are required for Catalog to see column names.

Catalog won't create schemas or reference foreign keys with these grants alone.

```sql
CREATE OR REPLACE PROCEDURE grant_rights_schema_today(user_name VARCHAR)
AS $$
DECLARE
    row record;
    sch text;
BEGIN
    FOR row IN (
        SELECT nspname
        FROM pg_namespace
        WHERE nspname NOT LIKE 'pg_%'
            AND nspname != 'information_schema'
    ) LOOP
        sch := row.nspname; 
        EXECUTE 'GRANT USAGE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        EXECUTE 'GRANT CREATE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        EXECUTE 'GRANT REFERENCES ON ALL TABLES IN SCHEMA ' || sch || ' TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;
```

### 2.2 Create Procedure To Grant Default Privileges on Future Tables

The procedure sets default REFERENCES on **new tables** created in schemas that already appear in `pg_tables` when you run it. `ALTER DEFAULT PRIVILEGES` applies to tables within a named schema. It does not grant schema-level USAGE and does not cover schemas you create later.

For each schema, the procedure runs as the schema owner. Only the schema owner can define default privileges for objects they create.

```sql
CREATE OR REPLACE PROCEDURE grant_rights_schema_future(user_name VARCHAR)
AS $$
DECLARE
    row record;
    usr text;
    sch text;
BEGIN
    FOR row IN (
        SELECT distinct tableowner,schemaname
        FROM pg_tables 
        WHERE schemaname not LIKE 'pg_%'
            AND schemaname != 'information_schema'
    ) LOOP
        sch := row.schemaname;
        usr := row.tableowner;
        EXECUTE 'ALTER DEFAULT PRIVILEGES FOR USER "' || usr || '" IN SCHEMA ' || sch 
            || ' GRANT REFERENCES ON TABLES TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;
```

Here is an example statement from the loop for schema `tesla` owned by user `elon`: `ALTER DEFAULT PRIVILEGES FOR USER elon IN SCHEMA tesla GRANT REFERENCES ON TABLES TO castor`

:::info[Schemas added after setup]

When tooling such as dbt or Hightouch creates a **new schema** after onboarding, re-run both procedures in section 2.4, or grant on each new schema manually.

This matters most for **late-binding views** (`CREATE VIEW ... WITH NO SCHEMA BINDING`, the default for dbt-redshift). Redshift resolves their columns live at query time. Catalog needs **USAGE on every schema the view reads from** to extract those columns. Without it, the view can appear in Catalog with **no columns**, even though table metadata still ingests normally.

dbt and other Catalog integrations **enrich** metadata already extracted from Redshift. If the warehouse extraction returns no columns, integration column descriptions won't appear in Catalog either.

:::

### 2.3 Create Catalog User

The Catalog user must have SYSLOG access. Use a safe password, from a password manager for example.

```text
CREATE USER "castor"
    PASSWORD '<password>'
    SYSLOG ACCESS UNRESTRICTED;
```

### 2.4 Grant Rights to Catalog User

Grant rights with the procedures in sections 2.1 and 2.2, then run the following commands.

Run both commands whenever a new schema is created if you want Catalog to ingest it.

```text
CALL grant_rights_schema_today('castor');
CALL grant_rights_schema_future('castor');
```

For a single new schema, you can grant explicitly instead of re-running the procedures across all schemas:

```sql
GRANT USAGE ON SCHEMA your_new_schema TO "castor";
GRANT CREATE ON SCHEMA your_new_schema TO "castor";
GRANT REFERENCES ON ALL TABLES IN SCHEMA your_new_schema TO "castor";
CALL grant_rights_schema_future('castor');
```

The future-tables procedure only includes schemas that have at least one table in `pg_tables`. If you create an empty schema first, re-run `grant_rights_schema_future` after the first table is created.

## 3. Test the Catalog User

The Catalog user should now be correctly set up. Before you give Catalog the credentials, run a test to make sure everything works.

1. Connect to your Redshift cluster using the `castor` credentials you created.
2. Run the following checks:

### 3.1 Check Schemas the Catalog User Can See

```sql
SELECT
    db.oid AS database_id,
    db.datname AS database_name,
    n.oid::TEXT AS schema_id,
    n.nspname AS schema_name,
    u.usename AS schema_owner,
    u.usesysid AS schema_owner_id,
    sd.description AS comment
FROM pg_catalog.pg_namespace AS n
CROSS JOIN pg_catalog.pg_database AS db
JOIN pg_catalog.pg_user u ON u.usesysid = n.nspowner
LEFT JOIN pg_catalog.pg_description sd ON sd.classoid = n.oid
WHERE TRUE
    AND db.datname = CURRENT_DATABASE()
    AND n.nspname NOT LIKE 'pg_%%'
    AND n.nspname NOT IN ('catalog_history', 'information_schema')
```

You should see all of your Redshift schemas.

### 3.2 Check Queries the Catalog User Can See

```sql
SELECT
    q.query::VARCHAR(128) AS query_id,
    q.querytxt::VARCHAR(20000) AS query_text,
    db.oid AS database_id,
    q.database AS database_name,
    q.pid AS process_id,
    q.aborted,
    q.starttime AS start_time,
    q.endtime AS end_time,
    q.userid AS user_id,
    q.label
FROM pg_catalog.stl_query AS q
JOIN pg_catalog.pg_database AS db ON db.datname  = q.database
WHERE q.starttime > CURRENT_TIMESTAMP - INTERVAL '6 hours'
LIMIT 1000
```

You should get queries from different users.

## 4. Add Credentials in the Coalesce App

:::info[Catalog admin required]

You must be a Catalog admin to add integration credentials.

:::

You can now enter the credentials you created in the [Coalesce App][].

1. Go to **Settings > Integrations**.
2. Click **Redshift**, then **Add**.
3. Enter the credentials.

<figure><figcaption></figcaption></figure>

## Troubleshooting

These scenarios cover common Redshift onboarding and ingestion questions for Catalog.

### Catalog Shows Fewer Tables or Columns Than Expected

The Catalog user needs USAGE on schemas and REFERENCES on tables for full metadata visibility. New schemas created after onboarding do not inherit grants automatically. Re-running only the future-tables procedure is not enough, because it does not grant schema-level USAGE.

**Resolution:**

1. Re-run **both** procedures from section 2.4 for schemas added after onboarding, or run the per-schema grants shown there.
2. If a **view appears with no columns**, check whether it's a late-binding view and whether the Catalog user has USAGE on every underlying schema.
3. Confirm USAGE, CREATE, and REFERENCES grants completed without errors in Redshift.
4. If specific schemas stay empty, verify the Catalog user can see them when logged into Redshift with the same credentials stored in Catalog.
5. Wait for the next Catalog ingestion cycle, or contact [Coalesce Support][] if metadata still looks incomplete.

### Query Text or Lineage Looks Incomplete

Catalog reads `STL_QUERY` and related system views for workload insight. Restricted SYSLOG visibility limits what Catalog can associate with tables.

**Resolution:**

1. Confirm the Catalog user still has the SYSLOG-related access described under **What Catalog Does With This Access**.
2. Check whether recent queries appear in `STL_QUERY` for the same database Catalog monitors.
3. For persistent gaps, contact [Coalesce Support][] with a sample table name and time window.

### You Moved or Duplicated Environments, For Example During Cloud Migration

Pinned assets, favorites, or warehouse naming may change when you migrate from one engine to another. Catalog ties assets to the identifiers it ingests from Redshift.

**Resolution:**

1. Align Catalog integration credentials with the cluster that now hosts the canonical schema names.
2. Plan a fresh ingestion cycle after go-live and validate counts of databases and schemas in Catalog against Redshift.
3. Work with [Coalesce Support][] when you need help remapping or reconciling large moves between warehouses.

### Connection Failures After an IP or Security Group Change

Catalog connects from fixed egress IPs. Cluster security groups must still allow those addresses.

**Resolution:**

1. Re-check the steps under **1. Whitelist Catalog's IP on Redshift** and add both IPs for your Catalog hostname if any rule was replaced during maintenance.
2. Test connectivity from a controlled environment if your team uses network traces.
3. Update stored credentials in Catalog if the cluster endpoint or port changed.

### New Cluster or Restored Snapshot

Restored clusters often use new endpoints, passwords, or parameter groups.

**Resolution:**

1. Update **Settings > Integrations** in Catalog with the new host, database, user, and password.
2. Re-run the **3. Test the Catalog User** SQL from this page to confirm metadata queries succeed.
3. Schedule a full ingestion refresh if the cluster ID changed materially.

---

## Appendix

### Full Code

```sql
-- Create procedure to grant rights on existing schemas
CREATE OR REPLACE PROCEDURE grant_rights_schema_today(user_name VARCHAR)
AS $$
DECLARE
    row record;
    sch text;
BEGIN
    FOR row IN (
        SELECT nspname
        FROM pg_namespace
        WHERE nspname NOT LIKE 'pg_%'
            AND nspname != 'information_schema'
    ) LOOP
        sch := row.nspname; 
        EXECUTE 'GRANT USAGE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        -- Create is needed references to work
        EXECUTE 'GRANT CREATE ON SCHEMA ' || sch || ' TO "' || user_name || '"';
        -- References allow the catalog user to see column names
        -- It also allows it to reference foreign keys, but we will never do that
        EXECUTE 'GRANT REFERENCES ON ALL TABLES IN SCHEMA ' || sch || ' TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Create procedure to grant default privileges on future tables in existing schemas
CREATE OR REPLACE PROCEDURE grant_rights_schema_future(user_name VARCHAR)
AS $$
DECLARE
    row record;
    usr text;
    sch text;
BEGIN
    FOR row IN (
        SELECT distinct tableowner,schemaname
        FROM pg_tables 
        WHERE schemaname not LIKE 'pg_%'
            AND schemaname != 'information_schema'
    ) LOOP
        sch := row.schemaname;
        usr := row.tableowner;
        EXECUTE 'ALTER DEFAULT PRIVILEGES FOR USER "' || usr || '" IN SCHEMA ' || sch 
            || ' GRANT REFERENCES ON TABLES TO "' || user_name || '"';
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Create Catalog User
CREATE USER "castor"
    PASSWORD '<password>'
    SYSLOG ACCESS UNRESTRICTED;

-- Grant rights to Castor user
CALL grant_rights_schema_today('castor');
CALL grant_rights_schema_future('castor');
```

### Queries Run by the Catalog User

- Databases

```sql
    SELECT
        db.oid AS database_id,
        db.datname AS database_name,
        dd.description AS "comment"
    FROM pg_catalog.pg_database AS db
        LEFT JOIN pg_catalog.pg_description AS dd ON dd.classoid = db.oid
    -- when cross database query goes public this will evolve
    WHERE db.datname = CURRENT_DATABASE()
```

- Schemas

```sql
    SELECT
        db.oid AS database_id,
        db.datname AS database_name,
        n.oid::TEXT AS schema_id,
        n.nspname AS schema_name,
        u.usename AS schema_owner,
        u.usesysid AS schema_owner_id,
        sd.description AS "comment"
    FROM pg_catalog.pg_namespace AS n
    CROSS JOIN pg_catalog.pg_database AS db
    JOIN pg_catalog.pg_user AS u ON u.usesysid = n.nspowner
    LEFT JOIN pg_catalog.pg_description AS sd ON sd.classoid = n.oid
    WHERE TRUE
        AND db.datname = CURRENT_DATABASE()
        AND n.nspname NOT LIKE 'pg_%%'
        AND n.nspname NOT IN ('catalog_history', 'information_schema')
```

- Tables

```sql
    WITH tables AS (
        SELECT
            t.oid AS table_id,
            t.relname AS table_name,
            n.nspname AS schema_name,
            n.oid AS schema_id,
            u.usename AS table_owner,
            t.relowner AS table_owner_id,
            td.description AS "comment",
            t.reltuples::BIGINT AS tuples
        FROM pg_class AS t
            CROSS JOIN parameters AS p
            JOIN pg_catalog.pg_namespace AS n ON n.oid = t.relnamespace
            LEFT JOIN pg_catalog.pg_description AS td ON td.objoid = t.oid AND td.objsubid = 0
            LEFT JOIN pg_catalog.pg_user AS u ON u.usesysid = t.relowner
        WHERE TRUE
            AND t.relam IN (0, 2) -- should not be an index
    ),

    database AS (
        SELECT
            db.datname AS database_name,
            db.oid AS database_id
        FROM pg_catalog.pg_database AS db
        WHERE TRUE
            AND db.datname = CURRENT_DATABASE()
        LIMIT 1
    ),

    meta AS (
        SELECT
            t.table_schema AS schema_name,
            t.table_name AS table_name,
            t.table_type
        FROM information_schema.tables AS t
            CROSS JOIN parameters AS p
        WHERE TRUE
    ),

    external_tables AS (
        SELECT
            db.datname AS database_name,
            db.oid::TEXT AS database_id,
            s.schemaname AS schema_name,
            s.esoid::TEXT AS schema_id,
            t.tablename AS table_name,
            db.datname || '.' || s.schemaname || '.' || t.tablename AS table_id,
            u.usename AS table_owner,
            s.esowner AS table_owner_id,
            NULL AS tuples,
            NULL AS "comment",
            'EXTERNAL' AS table_type
        FROM SVV_EXTERNAL_TABLES AS t
            JOIN SVV_EXTERNAL_SCHEMAS AS s ON t.schemaname = s.schemaname
            JOIN pg_catalog.pg_user AS u ON s.esowner = u.usesysid
            JOIN pg_catalog.pg_database AS db ON CURRENT_DATABASE() = db.datname
    )

    SELECT
        d.database_name,
        d.database_id::TEXT AS database_id,
        t.schema_name,
        t.schema_id::TEXT AS schema_id,
        t.table_name,
        t.table_id::TEXT AS table_id,
        t.table_owner,
        t.table_owner_id,
        t.tuples,
        t.comment,
        COALESCE(m.table_type, 'BASE TABLE') AS table_type
    FROM tables AS t
        CROSS JOIN database AS d
        LEFT JOIN meta AS m ON (t.table_name = m.table_name AND t.schema_name = m.schema_name)

    UNION DISTINCT

    SELECT * FROM external_tables
```

- Columns

```sql
    WITH ids AS (
        SELECT
            db.datname AS database_name,
            db.oid AS database_id,
            t.oid AS table_id,
            t.relname AS table_name,
            n.nspname AS schema_name,
            n.oid AS schema_id
        FROM pg_class AS t
            JOIN pg_catalog.pg_namespace AS n ON n.oid = t.relnamespace
            CROSS JOIN pg_catalog.pg_database AS db
        WHERE TRUE
            AND db.datname = CURRENT_DATABASE()
            AND n.nspname NOT LIKE 'pg_%%'
            AND n.nspname NOT IN ('catalog_history', 'information_schema')
            AND t.relam IN (0, 2) -- should not be an index
    ),

    information_tables AS (
        SELECT
            i.database_name,
            i.database_id,
            c.table_schema AS schema_name,
            i.schema_id,
            c.table_name AS table_name,
            i.table_id,
            c.column_name,
            i.table_id || '.' || c.column_name AS column_id,
            c.data_type,
            c.ordinal_position,
            c.column_default,
            c.is_nullable,
            c.character_maximum_length,
            c.character_octet_length,
            c.numeric_precision,
            c.numeric_precision_radix,
            c.numeric_scale,
            c.datetime_precision,
            c.interval_precision,
            c.interval_type,
            d.description AS "comment"
        FROM information_schema.columns AS c
            JOIN ids AS i
                ON c.table_schema = i.schema_name AND c.table_name = i.table_name
            LEFT JOIN pg_catalog.pg_description AS d
                ON d.objoid = i.table_id AND d.objsubid = c.ordinal_position
    ),

    raw_tables AS (
        -- some table might be missing from information_schema.columns
        -- this is a fallback fetching from lower level pg tables
        SELECT
            i.database_name,
            i.database_id,
            n.nspname AS schema_name,
            n.oid AS schema_id,
            c.relname AS table_name,
            c.oid AS table_id,
            a.attname AS column_name,
            c.oid::TEXT || '.' || a.attname AS column_id,
            a.attnum AS ordinal_position,
            ad.adsrc AS column_default,
            CASE
                WHEN t.typname = 'bpchar' THEN 'char'
                ELSE t.typname
            END AS data_type,
            CASE a.attnotnull WHEN TRUE THEN 'NO' ELSE 'YES' END AS is_nullable
        FROM pg_attribute AS a
            JOIN pg_type AS t ON t.oid = a.atttypid -- type
            JOIN pg_class AS c ON c.oid = a.attrelid -- table
            LEFT JOIN pg_attrdef AS ad ON ( ad.adrelid = c.oid AND ad.adnum = a.attnum ) -- default
            JOIN pg_namespace AS n ON n.oid = c.relnamespace -- schema
            JOIN ids AS i ON n.nspname = i.schema_name AND c.relname = i.table_name -- database
        WHERE TRUE
            AND c.relname NOT LIKE '%%pkey'
            AND a.attnum >= 0
            AND t.typname NOT IN ('xid', 'cid', 'oid', 'tid', 'name')
    ),

    tables AS (
        SELECT
            COALESCE(i.database_name, r.database_name) AS database_name,
            COALESCE(i.database_id, r.database_id)::TEXT AS database_id,
            COALESCE(i.schema_name, r.schema_name) AS schema_name,
            COALESCE(i.schema_id, r.schema_id)::TEXT AS schema_id,
            COALESCE(i.table_name, r.table_name) AS table_name,
            COALESCE(i.table_id, r.table_id)::TEXT AS table_id,
            COALESCE(i.column_name, r.column_name) AS column_name,
            COALESCE(i.column_id, r.column_id) AS column_id,
            COALESCE(i.data_type, r.data_type) AS data_type,
            COALESCE(i.ordinal_position, r.ordinal_position) AS ordinal_position,
            COALESCE(i.is_nullable, r.is_nullable) AS is_nullable,
            COALESCE(i.column_default, r.column_default) AS column_default,
            i.character_maximum_length::INT AS character_maximum_length,
            i.character_octet_length::INT AS character_octet_length,
            i.numeric_precision::INT AS numeric_precision,
            i.numeric_precision_radix::INT AS numeric_precision_radix,
            i.numeric_scale::INT AS numeric_scale,
            i.datetime_precision::INT AS datetime_precision,
            i.interval_precision::TEXT AS interval_precision,
            i.interval_type::TEXT AS interval_type,
            i.comment::TEXT AS "comment"
        FROM raw_tables AS r
            LEFT JOIN information_tables AS i ON (i.table_id = r.table_id AND i.column_name = r.column_name)
    ),

    views_late_binding AS (
        SELECT
            i.database_name,
            i.database_id::TEXT AS database_id,
            c.schema_name,
            i.schema_id::TEXT AS schema_id,
            c.table_name,
            i.table_id::TEXT AS table_id,
            c.column_name,
            i.table_id::TEXT || '.' || c.column_name AS column_id,
            c.data_type,
            c.ordinal_position,
            'YES' AS is_nullable,
            NULL::TEXT AS column_default,
            NULL::INT AS character_maximum_length,
            NULL::INT AS character_octet_length,
            NULL::INT AS numeric_precision,
            NULL::INT AS numeric_precision_radix,
            NULL::INT AS numeric_scale,
            NULL::INT AS datetime_precision,
            NULL::TEXT AS interval_precision,
            NULL::TEXT AS interval_type,
            NULL::TEXT AS "comment"
        FROM (
            SELECT
                schema_name,
                table_name,
                column_name,
                MIN(data_type) AS data_type,
                MIN(ordinal_position) AS ordinal_position
            FROM PG_GET_LATE_BINDING_VIEW_COLS()
            -- syntax specific to this redshift system table
            COLS(
                schema_name NAME,
                table_name NAME,
                column_name NAME,
                data_type VARCHAR,
                ordinal_position INT
            )
            GROUP BY 1, 2, 3
        ) AS c
        JOIN ids AS i
            ON c.schema_name = i.schema_name AND c.table_name = i.table_name
    ),

    external_columns AS (
        SELECT
            db.datname AS database_name,
            db.oid::TEXT AS database_id,
            s.schemaname AS schema_name,
            s.esoid::TEXT AS schema_id,
            c.tablename AS table_name,
            db.datname || '.' || s.schemaname || '.' || c.tablename AS table_id,
            c.columnname AS column_name,
            db.datname || '.' || s.schemaname || '.' || c.tablename || '.' || c.columnname AS column_id,
            c.external_type AS data_type,
            MIN(c.columnnum) AS ordinal_position,
            CASE c.is_nullable WHEN 'false' THEN 'NO' ELSE 'YES' END AS is_nullable,
            NULL AS column_default,
            NULL AS character_maximum_length,
            NULL AS character_octet_length,
            NULL AS numeric_precision,
            NULL AS numeric_precision_radix,
            NULL AS numeric_scale,
            NULL AS datetime_precision,
            NULL AS interval_precision,
            NULL AS interval_type,
            NULL AS "comment"
        FROM SVV_EXTERNAL_COLUMNS AS c
            JOIN SVV_EXTERNAL_SCHEMAS AS s ON c.schemaname = s.schemaname
            JOIN pg_catalog.pg_database AS db ON CURRENT_DATABASE() = db.datname

        -- To remove duplicate column names that can occur in external tables (no check on CSVs)
        GROUP BY database_name, database_id, schema_name, schema_id, table_name, table_id, column_name, column_id, data_type, is_nullable
    )

    SELECT * FROM tables

    UNION DISTINCT

    SELECT * FROM views_late_binding

    UNION DISTINCT

    SELECT * FROM external_columns

```

- Users

```sql
    SELECT
        usename AS user_name,
        usesysid AS user_id,
        usecreatedb AS has_create_db,
        usesuper AS is_super,
        valuntil AS valid_until
    FROM pg_catalog.pg_user
```

- Groups

```sql
    SELECT
        groname AS group_name,
        grosysid AS group_id,
        grosysid AS group_list
    FROM pg_group
```

- Queries

```sql
WITH parameters AS (
        SELECT
            DATEADD(DAY,-1,CURRENT_DATE) AS day_start,
            0 AS hour_min,
            23 AS hour_max
    ),

    queries_deduplicated AS (
        SELECT DISTINCT q.query
        FROM pg_catalog.stl_query AS q
            CROSS JOIN parameters AS p
        WHERE TRUE
            AND DATE(q.starttime) = p.day_start
            AND EXTRACT('hour' FROM q.starttime) BETWEEN p.hour_min AND p.hour_max
    ),

    query AS (
        SELECT
            q.query,
            qt.text,
            qt.sequence,
            COUNT(*) OVER(PARTITION BY q.query) AS sequence_count
        FROM queries_deduplicated AS q
            INNER JOIN pg_catalog.stl_querytext AS qt ON q.query = qt.query
    ),

    raw_query_text AS
    (
        SELECT
            q.query,
            LISTAGG(q.text, '') WITHIN GROUP (ORDER BY q.sequence) AS agg_text
        FROM query AS q
        WHERE TRUE
        -- LISTAGG raises an error when total length >= 65535
        -- each sequence contains 200 char max
            AND q.sequence_count < (65535 / 200)
        GROUP BY q.query
    ),

    query_text AS (
        SELECT
            query,
            CASE
                WHEN agg_text ILIKE 'INSERT INTO%%'
                    THEN REGEXP_REPLACE(agg_text, 'VALUES (.*)', 'DEFAULT VALUES')
                ELSE agg_text
            END AS agg_text
        FROM raw_query_text
    ),

    read_query AS (
        SELECT
            q.query::VARCHAR(256) AS query_id,
            qt.agg_text::VARCHAR(60000) AS query_text,
            db.oid AS database_id,
            q.database AS database_name,
            q.pid AS process_id,
            q.aborted,
            q.starttime AS start_time,
            q.endtime AS end_time,
            q.userid AS user_id,
            q.label
        FROM pg_catalog.stl_query AS q
            JOIN query_text AS qt ON q.query = qt.query
            JOIN pg_catalog.pg_database AS db ON db.datname = q.database
            CROSS JOIN parameters AS p
        WHERE TRUE
            AND DATE(q.starttime) = p.day_start
            AND EXTRACT('hour' FROM q.starttime) BETWEEN p.hour_min AND p.hour_max
    ),

    -- the DDL part is sensible to any change of JOIN and AGGREGATION: test in the field prior to merging
    ddl_query AS (
        SELECT
            (q.xid || '-' || q.query_part_rank)::VARCHAR(256) AS query_id,
            q.query_text::VARCHAR(20000) AS query_text,
            db.oid AS database_id,
            db.datname AS database_name,
            q.process_id,
            0 AS aborted,
            q.start_time,
            q.end_time,
            q.user_id,
            q.label
        FROM (
            SELECT
                q.userid AS user_id,
                q.pid AS process_id,
                q.xid,
                q.starttime AS start_time,
                MAX(q.endtime) AS end_time,
                MIN(q.label) AS "label",
                (LISTAGG(q.text, '') WITHIN GROUP (ORDER BY q.sequence)) AS query_text,
                RANK() OVER(PARTITION BY q.userid, q.pid, q.xid ORDER BY q.starttime) AS query_part_rank
            FROM pg_catalog.stl_ddltext AS q
                CROSS JOIN parameters AS p
            WHERE TRUE
                AND DATE(q.starttime) = p.day_start
                AND EXTRACT('hour' FROM q.starttime) BETWEEN p.hour_min AND p.hour_max
                -- LISTAGG raises an error when total length >= 64K
                AND q.sequence < (65535 / 200)
            GROUP BY q.userid, q.pid, q.xid, q.starttime
        ) AS q
        CROSS JOIN pg_catalog.pg_database AS db
        WHERE db.datname = CURRENT_DATABASE()
    ),

    merged AS (
        SELECT * FROM read_query

        UNION DISTINCT

        SELECT * FROM ddl_query
    )

    SELECT
        q.*,
        u.usename AS user_name
    FROM merged AS q
        JOIN pg_catalog.pg_user AS u ON u.usesysid = q.user_id
```

- Views DDL

```sql
    SELECT
        CURRENT_DATABASE() AS database_name,
        n.nspname AS schema_name,
        c.relname AS view_name,
        CASE
            WHEN c.relnatts > 0 THEN 'CREATE OR REPLACE VIEW ' + QUOTE_IDENT(n.nspname) + '.' + QUOTE_IDENT(c.relname) + ' AS\n' + COALESCE(pg_get_viewdef(c.oid, TRUE), '')
            ELSE COALESCE(pg_get_viewdef(c.oid, TRUE), '')
        END AS view_definition
    FROM
        pg_catalog.pg_class AS c
    INNER JOIN
        pg_catalog.pg_namespace AS n
        ON c.relnamespace = n.oid
    WHERE
        TRUE
        AND relkind = 'v'
        AND n.nspname NOT IN ('information_schema', 'pg_catalog');
```

### References

See these AWS topics for additional detail:

- [Privileges and access documentation][]
- [User creation and access to system tables][]

---

[Privileges and access documentation]: https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html
[User creation and access to system tables]: https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_USER.html
[SYSLOG and visibility of data in system tables and views]: https://docs.aws.amazon.com/redshift/latest/dg/c_visibility-of-data.html
[US Catalog app]: https://app.us.castordoc.com/
[EU Catalog app]: https://app.castordoc.com/
[Full SQL procedure in the Appendix]: /docs/catalog/integrations/data-warehouses/redshift#appendix
[Coalesce App]: https://app.castordoc.com/settings/integrations
[Coalesce Support]: mailto:support@coalesce.io

---

## S3

Connect S3 to the Catalog to sync metadata. This integration requires an active Catalog-managed Snowflake integration.

## Catalog Managed

### Requirement

You should have an active Catalog managed Snowflake integration. For more details, see the [Snowflake integration docs][].

### Add New Credentials in Catalog App

:::info[Admin Required]

You must be a Catalog admin to do this.

:::

You can now enter the newly created credentials in the [Catalog App integrations settings][].

* Go to "Settings > Integrations"
* Click on "S3 Add"
* Add the credential using the following format:

```json
{
  "source_name": "Snowflake Integration"
}
```

* `source_name`: Name of the related Snowflake integration in the App

[Snowflake integration docs]: /docs/catalog/integrations/data-warehouses/snowflake
[Catalog App integrations settings]: https://app.castordoc.com/settings/integrations

---

## Salesforce(Data-warehouses)

<!-- vale Coalesce.FirstPerson = NO -->

:::warning

You'll create an External app in this guide. Connected apps can no longer be created as of Spring 2026. See [New Connected Apps Can No Longer Be Created in Spring ‘26 for Salesforce][].
:::

This guide will take you through the steps to connecting Salesforce to Catalog. You can either choose to manage connections using [Catalog](#client-managed) for automatic updates(recommended) or [managed by you](#client-managed).

## Prerequisites

- A Salesforce System Administrator must complete these steps.

Follow these instructions for automatic data updates.

## Create a Catalog User (Optional)

We recommend creating a user and assigning them to Catalog app in order to keep a clear audit trail in Salesforce.

1. From **Setup**, in **Quick Find**, search for **Users > Users**.
2. Click **New User**.
3. Set the User License based on your subscription.
4. Set the profile to **Standard User.**
5. Fill in the rest of the information based on your company's administration rules.

## Set App Permissions

This is the user who will have access to the Salesforce app. For these next steps, you’ll need to clone a Permission Set.

1. From **Setup**, in **Quick Find**, search for **Users > Permissions Sets**.
2. **Clone** any permission set with the license, **Salesforce**. If the license doesn’t match the user assigned to the app, the connection will fail. In this example, Event Monitoring User was cloned.
3. Give your permission set a name and description.
4. Under System, click **System Permissions**. Then click **Edit**.
5. Assign the following permissions:
    1. Run Reports
    2. View Dashboards in Public Folders
    3. View Reports in Public Folders
    4. Manage All Private Reports and Dashboards
    5. View Roles and Role Hierarchy
    6. View Setup and Configuration
6. **Save**.
7. On the same Permission Sets page, click **Manage Assignments** near the top.
8. Click **Add Assignment**.
9. Assign the user who should have the permissions to use the Catalog app.

<DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>Permission Sets in Salesforce Setup highlighting the System Permissions section.</figcaption>
    </figure>

   <figure>
    
   <figcaption>System permission selection in a Salesforce permission set, highlighting Manage All Private Reports and Dashboards.</figcaption>
    </figure>
</DocCarousel>

## Create External Client App

1. From **Setup**, in **Quick Find**, search for **External Client Apps > External Client App Manager**.
2. Click **New External Client App.**
3. In **Basic Information**, enter the required information. Name the app `CatalogConnector`.
4. In **API**, check **Enable OAuth**.
    1. **Callback URL** - We don’t use this, so you can enter any domain such as `https://localhost.`
    2. **OAuth Scopes:**
        1. `Access Lightning applications (lightning)`
        2. `Manage user data via APIs (api)`
        3. `Perform requests at any time (refresh_token, offline_access)`
    3. **Flow Enablement -  In the same API section check the following:**
        1. `Enable Client Credentials Flow`
5. Other settings can be left at the default.
6. Click **Create**.

<DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>External Client App Manager in Salesforce Setup showing the CatalogConnector app’s Basic Information settings.</figcaption>
    </figure>
    <figure>
    
    <figcaption>External Client App Manager in Salesforce Setup showing the CatalogConnector app’s OAuth settings.</figcaption>
    </figure>

</DocCarousel>

### Enable OAuth Flows

1. After saving, it will take you back to the app home page.
2. Click on **OAuth Policies**.
3. Under **OAuth Flows and External Client App Enhancements** check **Enable Client Credentials Flow**.
4. Enter the Salesforce email of the user that will be doing the ingestion. See [Get Your Domain and Username](#get-your-domain-and-username).
5. The rest of the settings can be left as default.

## Get Connected App Client ID and Client Secret

1. From **Setup**, in **Quick Find**, search for **Apps > App Manager** .
2. Click on the dropdown next the app and choose **View**.
3. Go to **API (Enable OAuth Settings)** and click **Manage Consumer Details**.
4. Copy the **Consumer Key and Secret.**

## Get Your Domain and Username

:::warning

The username is not the same as your email. The username will look like something like this: `coalesce.support.a4bb07276aff@agentforce.com`.

:::

1. From **Setup**, in **Quick Find**, search for **Users > Users**.
2. Find the user and click on them. Copy the information in **Username.**

   

3. From **Setup**, in **Quick Find**, search for **Company Settings > My Domain**.
4. Copy **Current My Domain URL.**

## Connect Catalog and Salesforce

1. Go to [Settings > Integrations](https://app.castordoc.com/settings/integrations).
2. Find and click the Salesforce tile.
3. Select **Managed By Catalog**.
4. Give the source a name.
5. Enter the credentials and click **Save**.

```json
{
    "baseUrl": "<Your Salesforce host>",
    "clientId": "<The connected app consumer key>",
    "clientSecret": "<The connected app consumer secret>"
}

// example

{
    "baseUrl": "some-url-628db8712c-dev-ed.develop.my.salesforce.com",
    "clientId": "IitzUN13UdTsbHAQVr02NAVpTEMhYyQwJ",
    "clientSecret": "ENSGbqHF6QV7zTEfC3Z5VUW08"
}
```

:::info

For your first sync, it will take up to 48 hours and we will let you know when it is complete.

:::

### Troubleshooting

OAuth errors that match a wider pattern across Catalog integrations (redirect URIs, scopes, or credential updates in Catalog after a vendor change) are covered in [Integration OAuth troubleshooting][]. The sections below focus on Salesforce-specific checks.

#### Check IP Restrictions

You might need to adjust your IP settings.

1. From **Setup**, in **Quick Find**, search for **Connected Apps > Manage Connected Apps** .
2. Click Edit next to the app.
3. Review:
    1. IP Relaxation: Enforce IP restrictions
    2. Refresh Token Policy: Refresh token is valid until revoked

#### Need the Client ID and Client Secret

1. From **Setup**, in **Quick Find**, search for **Apps > App Manager** .
2. Click on the dropdown next the app and choose **View**.
3. Go to **API (Enable OAuth Settings)** and click **Manage Consumer Details**.
4. Copy the **Consumer Key and Secret.**

#### How To Test Your Connection Settings

You can make an API request to check that your credentials are correct.

The request can be done using cURL and should be `x-www-form-urlencoded`. Using a tool such as Postman or Insomnia makes it easy.

```bash

POST

curl --location 'your-domain-url.my.salesforce.com/services/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_id='consumer key' \
--data-urlencode 'client_secret=consumer secret' \
--data-urlencode 'grant_type=client_credentials'

```

#### View Login Information

You can also check the login history to see if the connection was successful.

1. From **Setup**, in **Quick Find**, search for **Users > Users**.
2. Click on the Catalog user.
3. Scroll to **Login History**. You’re looking for the application along with the status message.

#### Reset Keys

- The Consumer Key and Consumer Secret can't be reset. Create a new app.

## Client Managed

Follow these instructions if you want to upload your own data.

### Running a One-Time Extract

For your trial, you can give Coalesce a one-time view of your BI tools and upload them directly in the [Coalesce App](https://app.castordoc.com/settings/integrations).

### Running the Extraction Package

#### Install the PyPI Package

```bash
pip install castor-extractor[salesforce-viz]
```

:::warning
Make sure to use a version newer than 0.26.18 from which OAuth2.0 Client Credentials flow is supported.
:::
:::tip
Run `pip install --upgrade castor-extractor` to upgrade to the latest version.
:::
For more details, see the [castor-extractor PyPI page](https://pypi.org/project/castor-extractor/#pip-install) or our documentation on the [extractor](/docs/catalog/developer/castor-extractor).

#### Run the PyPI Package

After the package is installed, run the following command in your terminal:

```bash
castor-extract-salesforce-viz [arguments]
```

#### Credentials

Use these arguments to provide your Salesforce credentials:

- `-c`, `--client-id`: Salesforce client ID.
- `-s`, `--client-secret`: Salesforce client secret.
- `-b`, `--base-url`: Salesforce instance URL.

:::tip
Basic Auth is deprecated, but you can still use `username, password, security-token` if you need to.
Make sure NOT to use any of `username, password, security-token` for OAuth2.0 Client Credentials flow.
:::

#### Other Arguments

- `-o`, `--output`: Target folder to store the extracted files.

:::tip Getting Help
You can also run `--help` for a full list of arguments.
:::

### Scheduling and Pushing to Catalog

When you move out of your trial, you'll want to refresh your Salesforce Reporting content in Catalog. The Catalog team will provide you with the following:

1. **Catalog Identifier:** An ID to match your Salesforce files with your Catalog instance.
2. **Catalog Token:** An API token.

Use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

- `-k`, `--token`: Token provided by Catalog.
- `-s`, `--source_id`: Account ID provided by Catalog.
- `-t`, `--file_type`: Source type to upload. Currently supported types are `DBT`, `VIZ`, and `WAREHOUSE`.

#### Target Files

To specify target files, provide one of the following:

- `-f`, `--file_path`: Push a single file.
- `-d`, `--directory_path`: Push several files at once.

:::caution Verify Directory Contents

The tool uploads all files in the given directory. Make sure the directory contains only the extracted files before pushing.

:::

After configuring your target files, schedule the script run and the push to Catalog using your preferred scheduler to create this job.

---

## What’s Next

- [Error: grant type not supported](https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_troubleshoot_auth_errors.htm)

---

[Integration OAuth troubleshooting]: /docs/catalog/integrations/oauth-troubleshooting
[New Connected Apps Can No Longer Be Created in Spring ‘26 for Salesforce]: https://help.salesforce.com/s/articleView?id=005228017&type=1

---

## Snowflake(Data-warehouses)

Connect Catalog to your Snowflake instance to extract metadata, lineage, and tags. This guide covers what Catalog extracts, how to set up the integration, and how to sync documentation back to Snowflake.

## What Catalog Extracts from Snowflake

Catalog extracts and displays in the app:

- All your data assets: Database, Schemas, Tables and their Columns
  - Tables and Columns with their type and description
- Lineage between Tables and Dashboards
- Lineage between Columns and Dashboards
- Tables tags
- Columns tags

## Before You Start

Complete the technical onboarding with Catalog by granting access to your Snowflake metadata and running a couple of tests.

You'll need one of the following top-level roles on Snowflake to create the Catalog user:

- SECURITYADMIN
- ACCOUNTADMIN
- SYSADMIN

If your configuration allows, you can use different roles.

## What Catalog Does with Its Access

Your Snowflake instance contains a `SNOWFLAKE` database. This database is system defined, read-only, and contains only metadata. It's similar to PG-catalog in classic PostgreSQL databases.

Catalog only has access to the `SNOWFLAKE` database. It runs queries on the tables of the `ACCOUNT_USAGE` schema. Catalog only accesses the metadata, not the data itself.

For more information on the database `SNOWFLAKE` and privileges, review the Snowflake documentation:

- [Database SNOWFLAKE documentation][]
- [Privileges and access documentation][]

### Queries Ran by the Catalog User

Catalog runs the following queries to retrieve your metadata.

<details>
<summary> Catalog queries</summary>

**Get Databases:**

```sql
SELECT
    database_id,
    database_name,
    database_owner,
    is_transient,
    comment,
    last_altered,
    created,
    deleted
FROM SNOWFLAKE.ACCOUNT_USAGE.DATABASES
```

**Get Schemas:**

```sql
SELECT
    schema_id,
    schema_name,
    catalog_id as database_id,
    catalog_name as database_name,
    schema_owner,
    is_transient,
    comment,
    last_altered,
    created,
    deleted
FROM SNOWFLAKE.ACCOUNT_USAGE.SCHEMATA
```

**Get Tables:**

```sql
SELECT
    table_id,
    table_name,
    table_schema_id AS schema_id,
    table_schema AS schema_name,
    table_catalog_id AS database_id,
    table_catalog AS database_name,
    table_owner,
    table_type,
    is_transient,
    row_count,
    bytes,
    comment,
    created,
    last_altered,
    deleted
FROM SNOWFLAKE.ACCOUNT_USAGE.TABLES
WHERE deleted is null
```

**Get Columns:**

```sql
SELECT
    column_id,
    column_name,
    table_id,
    table_name,
    table_schema_id AS schema_id,
    table_schema AS schema_name,
    table_catalog_id AS database_id,
    table_catalog AS database_name,
    ordinal_position,
    column_default,
    is_nullable,
    data_type,
    maximum_cardinality,
    character_maximum_length,
    character_octet_length,
    numeric_precision,
    numeric_precision_radix,
    numeric_scale,
    datetime_precision,
    interval_type,
    interval_precision,
    comment,
    deleted
FROM SNOWFLAKE.ACCOUNT_USAGE.COLUMNS
```

**Get Users:**

```sql
SELECT
    name,
    created_on,
    deleted_on,
    login_name,
    display_name,
    first_name,
    last_name,
    email,
    has_password,
    disabled,
    snowflake_lock,
    default_warehouse,
    default_namespace,
    default_role,
    bypass_mfa_until,
    last_success_login,
    expires_at,
    locked_until_time,
    password_last_set_time,
    comment
FROM SNOWFLAKE.ACCOUNT_USAGE.USERS
```

**Get Queries:**

```sql
SELECT
    query_id,
    -- dropping INSERT values
    IFF(
        query_type = 'INSERT',
        REGEXP_REPLACE(query_text, 'VALUES (.*)', 'DEFAULT VALUES'),
        query_text
    ) AS query_text,
    database_id,
    database_name,
    schema_id,
    schema_name,
    query_type,
    session_id,
    user_name,
    role_name,
    warehouse_id,
    warehouse_name,
    execution_status,
    error_code,
    error_message,
    CONVERT_TIMEZONE('UTC', start_time) as start_time,
    CONVERT_TIMEZONE('UTC', end_time) as end_time,
    total_elapsed_time,
    bytes_scanned,
    percentage_scanned_from_cache,
    bytes_written,
    bytes_written_to_result,
    bytes_read_from_result,
    rows_produced,
    rows_inserted,
    rows_updated,
    rows_deleted,
    rows_unloaded,
    bytes_deleted,
    partitions_scanned,
    partitions_total,
    compilation_time,
    execution_time,
    queued_provisioning_time,
    queued_repair_time,
    queued_overload_time,
    transaction_blocked_time,
    release_version,
    is_client_generated_statement
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE TRUE
    AND DATE(CONVERT_TIMEZONE('UTC', start_time)) = DATEADD(DAY,-1,CURRENT_DATE)
    AND execution_status = 'SUCCESS'
    AND query_type NOT IN ('SHOW', 'USE', 'ROLLBACK', 'DESCRIBE', 'ALTER_SESSION')
```

</details>

## Step 1: Whitelist Catalog IP on Snowflake

This step is only needed if you've set up network policies, meaning your Snowflake instance only accepts connections from specific IPs. Review the [Snowflake network policies documentation][] for more information.

Here are the Catalog fixed IPs for the whitelist:

- For instances on [app.us.castordoc.com][]: `34.42.92.72`
- For instances on [app.castordoc.com][]: `35.246.176.138`

Review [Snowflake's instructions on how to whitelist an IP address][] for details.

## Step 2: Create and Test the Catalog User on Snowflake

Enter credentials in the Catalog App interface. This section walks you through creating the user, role, and warehouse, then testing the setup.

:::tip[Run the Complete Script]

If you prefer to run all the SQL at once instead of step-by-step, use the following script. It includes all commands from Steps 2.1 through 2.7 and the test queries. Skip the `GRANT APPLY TAG` command if you're using Snowflake Standard Edition.

<details>
<summary>Complete script Steps 2.1 through 2.7</summary>

```sql
-- Create a dedicated role
USE ROLE SECURITYADMIN;
CREATE ROLE METADATA_VIEWER_ROLE;
GRANT ROLE METADATA_VIEWER_ROLE TO ROLE SYSADMIN;

-- Skip the following line if using Snowflake Standard Edition
GRANT APPLY TAG ON ACCOUNT TO ROLE METADATA_VIEWER_ROLE;

-- Create dedicated warehouse
USE ROLE SYSADMIN;
CREATE WAREHOUSE METADATA_WH
WITH WAREHOUSE_SIZE = SMALL
  AUTO_SUSPEND = 59
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE
  COMMENT = 'metadata reader warehouse';

-- Grant usage on the created warehouse to the created role
GRANT USAGE ON WAREHOUSE METADATA_WH TO ROLE METADATA_VIEWER_ROLE;

-- Grant usage on Snowflake shared database to the created role
USE ROLE ACCOUNTADMIN;
-- It's highly likely that you need to use this role
-- to grant access to this specific database
GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE METADATA_VIEWER_ROLE;

-- Create a Catalog user
USE ROLE SECURITYADMIN;
CREATE OR REPLACE USER CATALOG
    LOGIN_NAME = CATALOG
    DEFAULT_ROLE = METADATA_VIEWER_ROLE
    DEFAULT_WAREHOUSE = METADATA_WH;

-- Grant the Catalog user the dedicated role
GRANT ROLE METADATA_VIEWER_ROLE TO USER CATALOG;

-- Test the Catalog user
USE ROLE METADATA_VIEWER_ROLE;

SHOW WAREHOUSES;
-- You must see the warehouse METADATA_WH as the default one

SHOW ROLES;
-- You must see the role METADATA_VIEWER_ROLE as the current and default one

SELECT
    database_id,
    database_name
FROM SNOWFLAKE.ACCOUNT_USAGE.DATABASES
WHERE NOT is_transient
    AND deleted IS NULL;
-- This query must return all databases that exist on your Snowflake instance
```

</details>
:::

### 2.1 Create a Dedicated Role

Create a role called `METADATA_VIEWER_ROLE`. This role inherits the public role from your instance and won't have any rights initially. This role will be assigned to the Catalog user.

```sql
USE ROLE SECURITYADMIN;

CREATE ROLE METADATA_VIEWER_ROLE;
GRANT ROLE METADATA_VIEWER_ROLE TO ROLE SYSADMIN;
```

### 2.2 Grant Access to Snowflake Object Tagging

:::info[Snowflake Standard Edition]

Skip this step if you're using Snowflake Standard Edition.

:::

The `APPLY TAG` permission is required to retrieve your Snowflake Tags. Review the Snowflake [documentation on Object Tagging][] for more information.

```sql
USE ROLE SECURITYADMIN;

GRANT APPLY TAG ON ACCOUNT TO ROLE METADATA_VIEWER_ROLE;
```

### 2.3 Create a Dedicated Warehouse

The warehouse size must be small at least. Catalog only retrieves your data model and queries a few times a day and the auto-suspend setup is on. This size is needed because the tables within the `ACCOUNT_USAGE` schema aren't always optimized by Snowflake.

```sql
USE ROLE SYSADMIN;

CREATE WAREHOUSE METADATA_WH
WITH WAREHOUSE_SIZE = SMALL
  AUTO_SUSPEND = 59
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE
  COMMENT = 'metadata reader warehouse';
```

### 2.4 Grant Usage on the New Warehouse to New Role

Add rights to the role you created in Step 2.1. This role is granted usage rights to the new warehouse you created in Step 2.3.

```sql
GRANT USAGE ON WAREHOUSE METADATA_WH TO ROLE METADATA_VIEWER_ROLE;
```

### 2.5 Grant Usage on Snowflake Shared Database to the Created Role

Add further rights to the role you created in Step 2.1. The following grants usage of the database `SNOWFLAKE`.

You must use a role that can grant usage to this database. The role `ACCOUNTADMIN` always can. Feel free to pick another one based on your role configuration.

```sql
USE ROLE ACCOUNTADMIN;
-- It's highly likely that you need to use the ACCOUNTADMIN role
-- to grant access to this specific database

GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE METADATA_VIEWER_ROLE;
```

### 2.6 Grant Monitor on Dynamic Tables

In order to have lineage on Snowflake Dynamic Tables, the METADATA_VIEWER_ROLE needs further rights.

The following SQL statements need to be run for all Snowflake Databases with Dynamic Tables relevant to Catalog:

```sql
GRANT MONITOR ON ALL DYNAMIC TABLES IN DATABASE <DATABASE_NAME> TO ROLE METADATA_VIEWER_ROLE;
GRANT MONITOR ON FUTURE DYNAMIC TABLES IN DATABASE <DATABASE_NAME> TO ROLE METADATA_VIEWER_ROLE;
```

### 2.7 Create the Catalog User on Snowflake

Create a user called CATALOG.

```sql
USE ROLE SECURITYADMIN;
-- You may need to use ACCOUNTADMIN
CREATE OR REPLACE USER CATALOG
    LOGIN_NAME = CATALOG
    DEFAULT_ROLE = METADATA_VIEWER_ROLE
    DEFAULT_WAREHOUSE = METADATA_WH;
```

### 2.8 Grant the Catalog User the Dedicated Role

Grant the Catalog user the role you created in Step 2.1.

```sql
GRANT ROLE METADATA_VIEWER_ROLE TO USER CATALOG;
```

### 2.9 Test the Catalog User

The Catalog user should now be correctly set up. Before you give Catalog the credentials, run a test to make sure everything works.

Use the created role and run the following statements.

**Check Warehouse:**

You must see the warehouse `METADATA_WH` as the default one.

```sql
USE ROLE METADATA_VIEWER_ROLE;

SHOW WAREHOUSES;
```

**Check Role:**

You must see the role `METADATA_VIEWER_ROLE` as the current and default one.

```sql
SHOW ROLES;
```

**Check Databases:**

This query must return all databases that exist on your Snowflake instance.

```sql
SELECT
    database_id,
    database_name
FROM SNOWFLAKE.ACCOUNT_USAGE.DATABASES
WHERE NOT is_transient
    AND deleted IS NULL;
```

## Step 3: Create a Key Pair

Review the Snowflake [documentation about Key Pair Authentication][] for more details.

### 3.1 Create Private Key

Open the terminal on your computer and generate a private key without a passphrase using the following command:

```bash
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
```

### 3.2 Create Public Key

Generate the public key using the private key you created above with the following command:

```bash
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
```

The key generated will be in the following format:

```text
-----BEGIN PUBLIC KEY-----
MIIBIj...
-----END PUBLIC KEY-----
```

### 3.3 Assign Public Key to the Catalog User

Execute an `ALTER USER` command to add the public key to the Catalog user. Insert the public key without its delimiters in the command. Here is an example:

```sql
ALTER USER CATALOG SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...';
```

## Step 4: Add New Credentials in Catalog App

:::info[Admin Access Required]

You must be a Catalog admin to complete this step.

:::

You can now enter the newly created credentials in the Catalog App [integrations][].

1. Go to **Settings** > **Integrations**.
2. Click **Snowflake Add**.
3. Add the following credentials.
   1. **Account** field, enter the Account Locator of your Snowflake. This value can be found in the hostname, AWS Private Link, or Azure Private Link endpoint for your instance. Here is an example: `example.us-west-1`
   2. **Username** field, put the `LOGIN_NAME` of the user created.
   3. **Private Key**, paste it in the Private Key field using the following format:

    ```text
    -----BEGIN PRIVATE KEY-----
    <PRIVATE_KEY>
    -----END PRIVATE KEY-----
    ```

:::caution[Account Format]

Don't add `.snowflakecomputing.com` in the Account form. Only include the account and region.

:::

## Sync Back to Snowflake

Make Catalog your source of documentation and truth, then spread knowledge back to your instance:

<ButtonLink to="/docs/catalog/integrations/sync-back/sync-back-to-snowflake">
  Sync back to Snowflake
</ButtonLink>

## Soon to Come

### Sensitive Data Detection (Beta)

Coming soon.

## What's Next?

- [Sync back to Snowflake][] to push documentation from Catalog to your Snowflake instance

---

[Database SNOWFLAKE documentation]: https://docs.snowflake.com/en/sql-reference/account-usage.html#what-is-the-snowflake-database
[Privileges and access documentation]: https://trevorscode.com/comprehensive-tutorial-of-snowflake-privileges-and-access-control/
[Snowflake network policies documentation]: https://docs.snowflake.com/en/user-guide/network-policies.html
[app.us.castordoc.com]: https://app.us.castordoc.com/
[app.castordoc.com]: https://app.castordoc.com/
[Snowflake's instructions on how to whitelist an IP address]: https://docs.snowflake.com/en/user-guide/network-policies.html#network-policy-properties
[documentation on Object Tagging]: https://docs.snowflake.com/en/user-guide/object-tagging#summary-of-ddl-commands-operations-and-privileges
[documentation about Key Pair Authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[integrations]: https://app.castordoc.com/settings/integrations
[Sync back to Snowflake]: https://docs.coalesce.io/docs/catalog/integrations/sync-back/sync-back-to-snowflake

---

## SQL Server(Data-warehouses)

This page explains how to configure a SQL Server integration in Catalog. It focuses on network access, permissions, and validation steps required to allow Catalog to extract metadata in a read-only and secure way.

## Prerequisites

## How Catalog Discovers Databases

By default, Catalog runs in multi-database discovery. It connects to SQL Server with the provided credentials, lists the databases visible to the user, optionally filters them using `db_allowed` or `db_blocked` settings, and extracts metadata from each selected database.

```sql title="Database Discovery Query"
SELECT
    db.database_id,
    database_name = db.name
FROM 
    sys.databases AS db
WHERE 
    db.name NOT IN ('master', 'model', 'msdb', 'tempdb', 'DBAdmin')
```

## 1. Whitelist Catalog's IP on SQL Server

To allow Catalog to connect to SQL Server, you need to whitelist Catalog's IP address.

Here are the fixed IPs:

- For instances on [app.us.castordoc.com](https://app.us.castordoc.com/): `34.42.92.72`
- For instances on [app.castordoc.com](https://app.castordoc.com/): `35.246.176.138`

:::info SQL Server Connector

Catalog's SQL Server connector executes SQL queries directly against your SQL Server databases without an HTTP proxy. Clients frequently provide Catalog access to a replica or read-only instance.

:::

## 2. Create Catalog User on SQL Server

First, create the server-level login:

```sql title="Server-Level Login"
CREATE LOGIN <login_name> WITH PASSWORD = '*****';
GO
```

### Repeat for Each Database

Then repeat this sequence for each database you want Catalog to extract:

```sql title="Database-Level User"
USE <database>;
GO

CREATE USER <username> FOR LOGIN <login_name>;
GO

GRANT CONNECT TO <username>;
GRANT VIEW DEFINITION TO <username>;
GO
```

## 3. Enable Query Store Extraction - Optional

This step is optional. If you want Catalog to extract SQL query history, you must enable Query Store on all databases that Catalog has access to. Query history is necessary to compute lineage and show table source queries.

Follow the [Microsoft documentation on Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store?view=sql-server-ver17&redirectedfrom=MSDN#enable-the-query-store).

Once Query Store is enabled, you must also give the `VIEW DATABASE STATE` permission to the user on each database:

```sql title="Database-Level Permissions"
USE <database>;
GO

GRANT VIEW DATABASE STATE TO <username>;
```

Catalog reads from Query Store system views (`sys.query_store_*`) to extract SQL queries. In most environments, `VIEW DATABASE STATE` is sufficient to allow this access.

## 4. Check Credentials

Before providing credentials to Catalog, validate the setup manually. Connect using the Catalog credentials and run the following queries.

```sql title="List Databases Visible to the Catalog User"
SELECT
    db.database_id,
    database_name = db.name
FROM 
    sys.databases AS db
WHERE 
    db.name NOT IN ('master', 'model', 'msdb', 'tempdb', 'DBAdmin')
```

### Repeat Credentials Check for Each Database

For each database you want Catalog to extract, connect to that database and run the checks below:

```sql
USE <database>;
GO
```

### List Schemas Visible to the Catalog User

```sql
WITH ids AS (
    SELECT DISTINCT
        table_catalog,
        table_schema
    FROM information_schema.tables
    WHERE table_catalog = DB_NAME()
)
SELECT
    d.database_id,
    database_name = i.table_catalog,
    schema_name = s.name,
    s.schema_id,
    schema_owner = u.name,
    schema_owner_id = u.uid
FROM sys.schemas AS s
INNER JOIN ids AS i
    ON s.name = i.table_schema
LEFT JOIN sys.sysusers AS u
    ON s.principal_id = u.uid
LEFT JOIN sys.databases AS d
    ON i.table_catalog = d.name;
```

### Check Query Store Access

This step is optional.

```sql
SELECT
    q.query_id,
    qt.query_sql_text AS query_text,
    rs.count_executions,
    rs.last_duration AS duration,
    rs.last_execution_time AS start_time,
    DATEADD(
        SECOND,
        last_duration / 1000000,
        DATEADD(MICROSECOND, last_duration % 1000000, rs.last_execution_time)
    ) AS end_time
FROM
    sys.query_store_runtime_stats AS rs
INNER JOIN
    sys.query_store_plan p
        ON rs.plan_id = p.plan_id
INNER JOIN
    sys.query_store_query q
        ON p.query_id = q.query_id
INNER JOIN
    sys.query_store_query_text qt
        ON q.query_text_id = qt.query_text_id;
```

## 5. Add New Credentials in Catalog App

Enter the credentials in the [Catalog App](https://app.castordoc.com/settings/integrations). You must be a Catalog admin.

Go to **Settings** > **Integrations** > **SQL Server** > **Add**.

Credentials format:

```json
{
    "port": 1234,
    "host": "host",
    "user": "user",
    "password": "*****",
    "default_db": "database"
}
```

The `default_db` field is optional and only needed in specific cases. See the section below.

### Default Database

In some environments, notably Azure SQL, the login may not have access to master. If SQL Server attempts to connect to master by default, you may see:

```text
The server principal "<user>" is not able to access the database "master"
under the current security context
```

In that case, set `default_db` to a database that the user can access. This is used to bootstrap the connection and doesn't restrict extraction by itself.

## Appendix

This section provides reference information for the SQL Server integration.

### Queries Run by Catalog

This section covers the queries Catalog runs on SQL Server. Catalog only extracts metadata such as columns, schemas, tables, users, and view definitions. Optionally, it extracts Query Store query history.

#### Databases

```sql title="Databases"
SELECT
    db.database_id,
    database_name = db.name
FROM sys.databases AS db
WHERE db.name NOT IN ('master', 'model', 'msdb', 'tempdb', 'DBAdmin')
```

#### Schemas

```sql title="Schemas"
-- Fetch database information
WITH ids AS (
    SELECT DISTINCT
        table_catalog,
        table_schema
    FROM {database}.information_schema.tables
)

SELECT
    d.database_id,
    database_name = i.table_catalog,
    schema_name = s.name,
    schema_id = CAST(d.database_id AS VARCHAR(10)) + '_' + CAST(s.schema_id AS VARCHAR(10)),
    schema_owner = u.name,
    schema_owner_id = u.uid
FROM {database}.sys.schemas AS s
INNER JOIN ids AS i
    ON s.name = i.table_schema
LEFT JOIN {database}.sys.sysusers AS u
    ON s.principal_id = u.uid
LEFT JOIN {database}.sys.databases AS d
    ON i.table_catalog = d.name
```

#### Tables

```sql title="Tables"
/*
Select all types of tables:
- Views
- Base Tables
- External Tables
*/
WITH extended_tables AS (
    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.tables

    UNION

    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.views

    UNION

    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.external_tables
),
-- Get the row count per table
partitions AS (
    SELECT
        object_id,
        row_count = SUM(rows)
    FROM sys.partitions
    GROUP BY object_id
),
-- Append row count to table properties
extended_tables_with_row_count AS (
    SELECT
        et.*,
        row_count
    FROM
        extended_tables AS et
    LEFT JOIN partitions AS p
        ON et.table_id = p.object_id

),
-- Generate table identifiers and fetch table description
table_ids AS (
    SELECT
        table_id,
        table_name,
        schema_name = ss.name,
        schema_id = ss.schema_id,
        table_owner_id,
        table_owner = u.name,
        row_count,
        comment = CONVERT(varchar(1024), ep.value)
    FROM
        extended_tables_with_row_count AS et
    LEFT JOIN sys.schemas AS ss
        ON et.schema_id = ss.schema_id
    LEFT JOIN sys.sysusers AS u
        ON et.table_owner_id = u.uid
    LEFT JOIN sys.extended_properties AS ep
        ON (
            et.table_id = ep.major_id
            AND ep.minor_id = 0
            AND ep.name = 'MS_Description'
        )
),

meta AS (
    SELECT
        database_name = t.table_catalog,
        database_id = db.database_id,
        schema_name = t.table_schema,
        t.table_name,
        t.table_type
    FROM
        information_schema.tables AS t
    LEFT JOIN sys.databases AS db
        ON t.table_catalog = db.name
    WHERE t.table_catalog = db_name()
)

SELECT
    m.database_name,
    m.database_id,
    m.schema_name,
    i.schema_id,
    m.table_name,
    i.table_id,
    m.table_type,
    i.table_owner,
    i.table_owner_id,
    i.comment,
    tuples = i.row_count
FROM
    meta AS m
LEFT JOIN table_ids AS i
    ON (m.table_name = i.table_name AND m.schema_name = i.schema_name)
```

#### Columns

```sql title="Columns"
/*
Select all types of tables:
- Views
- Base Tables
- External Tables
*/
WITH extended_tables AS (
    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.tables

    UNION

    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.views

    UNION

    SELECT
        table_id = object_id,
        table_name = name,
        table_owner_id = principal_id,
        schema_id
    FROM
        sys.external_tables
),

-- Create the column identifiers
column_ids AS (
    SELECT
        sd.database_id,
        database_name = sd.name,
        column_id = sc.column_id,
        column_name = sc.name,
        table_id,
        table_name,
        schema_name = ss.name,
        schema_id = ss.schema_id,
        comment = CONVERT(varchar(1024), ep.value)
    FROM sys.columns AS sc
    LEFT JOIN extended_tables AS et ON sc.object_id = et.table_id
    LEFT JOIN sys.schemas AS ss ON et.schema_id = ss.schema_id
    LEFT JOIN sys.databases AS sd ON sd.name = DB_NAME()
    LEFT JOIN sys.extended_properties AS ep
        ON
            sc.object_id = ep.major_id
            AND sc.column_id = ep.minor_id
            AND ep.name = 'MS_Description'
),

columns AS (
    SELECT
        i.database_name,
        i.database_id,
        schema_name = c.table_schema,
        i.schema_id,
        table_name = c.table_name,
        i.table_id,
        c.column_name,
        c.data_type,
        c.ordinal_position,
        c.column_default,
        c.is_nullable,
        c.character_maximum_length,
        c.character_octet_length,
        c.numeric_precision,
        c.numeric_precision_radix,
        c.numeric_scale,
        c.datetime_precision,
        i.comment,
        column_id = CONCAT(i.table_id, '.', c.column_name)
    FROM
        information_schema.columns AS c
    LEFT JOIN column_ids AS i
        ON
            (
                c.table_name = i.table_name
                AND c.table_schema = i.schema_name
                AND c.column_name = i.column_name
            )
)

SELECT * FROM columns
```

#### Users

```sql title="Users"
SELECT
    user_name = u.name,
    user_id = u.principal_id
FROM sys.database_principals AS u
```

#### View DDLs

```sql title="View DDLs"
SELECT
    v.name AS view_name,
    m.definition AS view_definition,
    s.name AS schema_name,
    DB_NAME() AS database_name
FROM
    sys.views v
INNER JOIN
    sys.schemas s
    ON v.schema_id = s.schema_id
INNER JOIN
    sys.sql_modules m
    ON v.object_id = m.object_id
```

#### Queries

```sql title="Queries"
SELECT
    q.query_id,
    qt.query_sql_text as query_text,
    rs.count_executions,
    rs.last_duration as duration,
    rs.last_execution_time as start_time,
    'unknown-user' as user_name,
    'unknown-user' as user_id,
    DATEADD(SECOND, last_duration / 1000000,
        DATEADD(MICROSECOND, last_duration % 1000000, rs.last_execution_time)
    ) AS end_time
FROM
    sys.query_store_runtime_stats AS rs
INNER JOIN
    sys.query_store_plan p
    ON rs.plan_id = p.plan_id
INNER JOIN
    sys.query_store_query q
    ON p.query_id = q.query_id
INNER JOIN
    sys.query_store_query_text qt
    ON q.query_text_id = qt.query_text_id
WHERE
    CAST(rs.last_execution_time AS DATE) = :day
    AND DATEPART(HOUR, rs.last_execution_time) BETWEEN :hour_min AND :hour_max
```

---

## Integrations

Connect Catalog to your data warehouses, BI tools, transformation platforms, and knowledge bases.

## Troubleshooting

When authentication errors span more than one vendor pattern, start with the shared checklist before you open a single integration guide.

- [Integration OAuth troubleshooting](/docs/catalog/integrations/oauth-troubleshooting)

## Warehouses (Data Warehouses)

- [Amazon Athena and Glue](/docs/catalog/integrations/data-warehouses/amazon-athena-and-glue)
- [BigQuery](/docs/catalog/integrations/data-warehouses/bigquery)
- [Databricks](/docs/catalog/integrations/data-warehouses/databricks)
- [Postgres](/docs/catalog/integrations/data-warehouses/postgres)
- [Redshift](/docs/catalog/integrations/data-warehouses/redshift)
- [S3](/docs/catalog/integrations/data-warehouses/s3)
- [Salesforce](/docs/catalog/integrations/data-warehouses/salesforce)
- [Snowflake](/docs/catalog/integrations/data-warehouses/snowflake)
- [SQL Server](/docs/catalog/integrations/data-warehouses/sqlserver)

## BI Tools (Data Viz)

- [Domo](/docs/catalog/integrations/data-viz/domo)
- [Looker](/docs/catalog/integrations/data-viz/looker)
- [Looker Studio](/docs/catalog/integrations/data-viz/looker-studio)
- [Metabase](/docs/catalog/integrations/data-viz/metabase)
- [Omni](/docs/catalog/integrations/data-viz/omni)
- [Power BI](/docs/catalog/integrations/data-viz/power-bi/powerbi)
- [Qlik Sense](/docs/catalog/integrations/data-viz/qlik-sense)
- [Sigma](/docs/catalog/integrations/data-viz/sigma)
- [Tableau](/docs/catalog/integrations/data-viz/tableau)
- [ThoughtSpot](/docs/catalog/integrations/data-viz/thoughtspot)

## Transformation

- [Coalesce](/docs/catalog/integrations/transformation/coalesce)
- [dbt](/docs/catalog/integrations/transformation/dbt)

## Quality

- [Coalesce Quality](/docs/catalog/integrations/transformation/coalesce/set-up)
- [dbt Tests](/docs/catalog/integrations/quality/dbt-tests)
- [Monte Carlo](/docs/catalog/integrations/quality/monte-carlo)
- [Sifflet](/docs/catalog/integrations/quality/sifflet)
- [Soda](/docs/catalog/integrations/quality/soda)

## Knowledge Bases

- [Confluence](/docs/catalog/integrations/knowledge-bases/confluence)
- [Notion](/docs/catalog/integrations/knowledge-bases/notion)

## Communication

- [Microsoft Teams](/docs/catalog/integrations/communication/microsoft-teams)
- [Slack](/docs/catalog/integrations/communication/slack)

## Sync Back

- [Sync Back to BigQuery (beta)](/docs/catalog/integrations/sync-back/sync-back-to-bigquery-beta)
- [Sync Back to Looker (beta)](/docs/catalog/integrations/sync-back/sync-back-to-looker-beta)
- [Sync Back to Snowflake](/docs/catalog/integrations/sync-back/sync-back-to-snowflake)
- [Sync Back to Tableau (beta)](/docs/catalog/integrations/sync-back/sync-back-to-tableau-beta)

## Other (Orchestrators & Version Control)

- [Airflow](/docs/catalog/integrations/other/airflow)
- [GitHub and GitLab](/docs/catalog/integrations/other/github-and-gitlab)

---

## Confluence

:::warning[Security Notice]

**Permissions from Confluence are NOT synced to Catalog.**

All synced pages become visible to all Catalog users, regardless of source permissions.
Do not sync confidential, private, or regulated content unless all Catalog users should have access.

:::

Integrate Confluence with Catalog to synchronize data documentation content. This integration ensures a constantly updated knowledge base and powers the AI Assistant with your content.

## Why Synchronize Your Confluence Pages in Catalog?

### Advantage 1: Enhanced AI Assistant Intelligence

All imported content feeds the Assistant, significantly boosting its intelligence. This empowers your teams to gain independence and efficiency in their work.

### Advantage 2: Always Up-to-Date Content

Currently, users manually copy and paste content from Confluence into Catalog, leading to redundant maintenance efforts and potential inconsistencies. With this feature, these issues are eliminated. The synchronized content becomes the single source of truth, always up-to-date and read-only in Catalog.

## How To Perform the Synchronization

Follow these steps to set up the integration:

1. **Create a Confluence account for Catalog**
   This account, with the permissions you assign to it in the Confluence space (and therefore the visibility of pages), will be used to synchronize content with Catalog.
   Location: Your Internal Confluence Space

2. **Navigate to the pages or spaces you want to share**
   Location: Your Internal Confluence Space

3. **Grant permissions**
   Location: Your Internal Confluence Space

   <figure><figcaption></figcaption></figure>

4. **Complete the integration process**
   Location: In Catalog

   <figure><figcaption></figcaption></figure>

   <figure><figcaption></figcaption></figure>

### Account ID

To find your Account ID, see [Where can you find your Account ID?](https://community.atlassian.com/t5/Jira-questions/where-can-i-find-my-Account-ID/qaq-p/976527)

<figure><figcaption></figcaption></figure>
<figure><figcaption></figcaption></figure>

### Base URL

The Base URL can be found in the same place. Use a `base_url` in this format: `https://xxxxx.atlassian.net/`

<figure><figcaption></figcaption></figure>

### API Token

To find your API Token, see [API tokens](https://id.atlassian.com/manage-profile/security/api-tokens)

:::danger[Scoped Tokens Not Supported]

Scoped tokens are not supported by Catalog. Create an unscoped token to complete this integration.

:::

## FAQ

### Questions About Page Synchronization

1. **Q: If you select a parent page, are all its children automatically ingested?** A: Yes, all child pages are automatically ingested.
2. **Q: If you want to select only one child page out of 10 ingested, do you need to unsync the other 9?** A: Yes, you need to manually unsync the unwanted child pages. Or, directly synchronize the desired one in the first place.
3. **Q: How can you ingest only a single page without its children?** A: You need to manually unsync its child pages.
4. **Q: What are the ingestion rules for private pages?** A: Private pages within a public parent page are ingested. To prevent this, manually unsync the private pages.

### Questions About Rights and Permissions

1. **Q: Are the permissions on Confluence pages reflected in Catalog?** A: No, not currently. If User A does not have rights to view a page in Confluence, but that page is synchronized to Catalog and User A has access to Catalog, they will be able to see the page content. This content will also be used by the AI Assistant.

---

## Knowledge Bases

Sync your Confluence or Notion knowledge bases with Catalog to keep documentation up to date.

Use this table to pick the connector that matches your source system:

| Method | Best for |
| :----- | :------- |
| [Confluence][] | Teams that document in Atlassian Confluence and want pages synced into Catalog, including content that powers the AI Assistant. |
| [Notion][] | Teams that document in Notion and want workspace pages synced into Catalog with the same AI Assistant benefits. |

[Confluence]: /docs/catalog/integrations/knowledge-bases/confluence
[Notion]: /docs/catalog/integrations/knowledge-bases/notion

---

## Notion

:::warning[Security Notice]

**Permissions from Notion are NOT synced to Catalog.**

All synced pages become visible to all Catalog users, regardless of source permissions.
Do not sync confidential, private, or regulated content unless all Catalog users should have access.

:::

Integrate Notion with the Catalog to enable seamless synchronization of data documentation content.

## Introduction To the Feature

This integration ensures:

* A constantly updated and comprehensive knowledge base for the Data Governance Team
* A powerful AI Assistant that utilizes the content to provide the best responses to your questions

## Why Synchronize Your Notion Pages in Catalog?

### Advantage 1: Enhanced AI Assistant Intelligence

All imported content feeds the Assistant, significantly boosting its intelligence. This empowers your teams to gain independence and efficiency in their work.

### Advantage 2: Always Up-to-Date Content

Currently, users manually copy and paste content from Notion into the Catalog, leading to redundant maintenance efforts and potential inconsistencies. With this feature, these issues are eliminated. The synchronized content becomes the single source of truth, always up-to-date and read-only in the Catalog.

## How To Perform the Synchronization

Follow these steps to set up the integration:

1. **Navigate to the integration page (as an admin)**

    Location: Your Internal Notion Space

    <figure><figcaption></figcaption></figure>

1. **Name the integration and sync it with the workspace**

    Location: Your Internal Notion Space

    <figure><figcaption></figcaption></figure>

1. **Grant permissions and copy the Internal Integration Secret token**

    Location: Your Internal Notion Space

    <figure><figcaption></figcaption></figure>

1. **Connect to a page**

    Location: Your Internal Notion Space

    <figure><figcaption></figcaption></figure>

1. **Verify the page's appearance when the integration is properly connected (and how to disconnect if needed)**

    Location: Your Internal Notion Space

    <figure><figcaption></figcaption></figure>

1. **Complete the integration process**

    Location: In Catalog

    <figure><figcaption>Start the synchronization process by clicking on the Notion icon. You will need to enter your credentials.</figcaption></figure>

## FAQ

### Questions About Page Synchronization

1. **Q: If a parent page is selected, are all its children automatically ingested?** A: Yes, all child pages are automatically ingested.
2. **Q: If you want to select only one child page out of 10 ingested, do you need to disconnect the other 9?** A: Yes, you'll need to manually disconnect the unwanted child pages. Or, you can directly synchronize the desired one in the first place.
3. **Q: How can you ingest only a single page without its children?** A: You need to manually disconnect its child pages.
4. **Q: What are the ingestion rules for private pages?** A: Private pages within a public parent page are ingested. To prevent this, manually disconnect the private pages.

### Questions About Rights and Permissions

1. **Q: Are the permissions on Notion pages reflected in Catalog?** A: No, not currently. If User A doesn't have rights to view a page in Notion, but that page is synchronized to Catalog and User A has access to Catalog, they will be able to see the page content. This content will also be used by the AI Assistant.

---

## Integration OAuth Troubleshooting

Use this guide when an integration that relies on vendor OAuth stops authenticating, returns authorization errors, or fails after you change something in your identity or security admin console. It summarizes how OAuth fits Catalog integrations, groups problems by symptom, and points you to integration-specific setup where behavior differs by product.

If you use Transform in the Coalesce App, Snowflake OAuth for Workspace connections is documented in [Snowflake OAuth][]. Catalog Public API authentication uses API tokens as described in [Catalog Public API][].

## How Catalog Uses OAuth for Integrations

Catalog reaches external systems through integrations you configure under **Settings > Integrations** in Coalesce Catalog. Some integrations authenticate with vendor OAuth, including consent screens, client id and secret, authorization codes, or refresh tokens. Others use API keys, personal access tokens, or username and password. The integration’s own documentation is the source of truth for which model applies.

When OAuth is in play, Catalog needs credentials that match the OAuth app or equivalent registration in your vendor admin console. Typical failure causes are expired or revoked tokens, a redirect URI or callback URL that no longer matches what the vendor expects, missing or insufficient API scopes, or a client id or secret that changed and was not updated in Catalog.

## Decide Whether OAuth Is the Right Layer To Fix

Use this quick check before you change network rules or re-run a full extraction:

- **The integration’s guide names OAuth, Google sign-in, or client id and secret** - Treat the problem as OAuth-related and follow the symptom sections below, then the vendor section in that guide.
- **The guide centers on an API key or personal access token** - Confirm the key is current, assigned to the right service identity, and pasted correctly in **Settings > Integrations**. OAuth flows in this hub usually do not apply.
- **Looker compared with Looker Studio** - Classic Looker uses API keys in Catalog. Looker Studio uses Google OAuth. Do not mix steps between those guides or you can see authentication errors. See [Looker][] and [Looker Studio][].

## Sync Fails, Stalls, or Looks Stale After a Vendor Change

This pattern often follows a password rotation, client secret rotation, OAuth policy change, or account lockout in the vendor system.

1. In the vendor admin console, confirm the OAuth app or integration user is still active, approved, and allowed to use the flows Catalog needs, for example, client credentials or refresh tokens, depending on the product.
2. If you rotated secrets or regenerated keys, copy the new values from the vendor console.
3. In Coalesce Catalog, open the [Integration settings page in the app][], select the source, and use **Edit credentials**, or the equivalent control your integration shows, to save updated JSON or fields your onboarding documented.
4. Trigger or wait for the next sync using your normal Catalog-managed or client-managed process. If failures continue, capture the error text or time stamp your vendor or Catalog contact can use.

## Redirect URI, Callback URL, or Environment Mismatch

Vendors often validate the exact callback or redirect URI registered for the OAuth client. A mismatch produces errors that mention redirect URIs, callbacks, or invalid redirect.

1. Open the OAuth app settings for that integration in the vendor console and read the allowed redirect or callback URLs.
2. Compare them to the values required for Catalog or listed in the integration guide. Some products use a fixed Catalog callback; others accept a placeholder when the flow does not use a browser redirect.
3. Update the vendor registration to match the documented pattern, or create a new OAuth client if your vendor does not allow edits.
4. Update saved credentials in Catalog if the client id, secret, or related fields changed when you fixed the registration.

## Insufficient Scope or Access Denied Immediately After Setup

If authorization completes in the vendor UI but API calls fail right away, scopes or API access for the OAuth client are a common cause.

1. Reopen the integration guide for your product and confirm every required OAuth scope or permission is enabled on the vendor side.
2. For products that require admin consent or security review, complete that step in the vendor tenant.
3. Save credentials again in Catalog after scope changes so the next run uses the updated consent.

## Authorization Codes Expire Before You Paste Them

Some flows issue short-lived authorization codes that often expire within minutes. If the code expires before Catalog receives the full credential bundle, authentication fails even when the client id and secret are correct.

1. Generate a new authorization code in the vendor console immediately before you send or paste it.
2. For Zoho Self Client and similar flows, work through the handoff in the same session when possible. See [Zoho][] for regional hosts, required scope, and timing.

## Update Credentials in Catalog After Fixing the Vendor Side

After you correct the OAuth app or tokens in the vendor system, Catalog still needs the latest values.

1. Sign in to Coalesce Catalog and open **Settings > Integrations**. You can also bookmark the [Integration settings page in the app][].
2. Open the integration that is failing.
3. Choose **Edit credentials**, or your source’s credential editor, and enter the updated client id, client secret, tokens, or JSON bundle exactly as your integration guide specifies.
4. Save and monitor the next extraction or sync outcome.

Exact field names and JSON shapes differ by integration. Always align with the guide for that integration.

## Integration-Specific Guides

These pages carry the detailed OAuth setup and product-specific edge cases:

- **[Salesforce][]** - External Client App, OAuth scopes, client credentials flow, IP relaxation, refresh policy, and connection tests.
- **[Zoho][]** - Self Client codes, regional `server_uri`, required scope, and common OAuth handoff failures.
- **[Looker Studio][]** - Google OAuth for Catalog-managed setup, which differs from classic Looker API keys.

Add or follow links from other integration pages as your stack grows; this hub stays focused on shared OAuth patterns.

## When To Contact Your Catalog Contact

Reach out when you have completed the checks for your symptom, updated credentials in Catalog, and failures persist, or when your organization needs help interpreting vendor error messages against Catalog’s ingestion model. Include the integration name, approximate time of failure, and any vendor error text you can share. Redact secrets before you send them.

## What's Next?

- Review administration options in [Integration settings admin documentation][].
- Return to the [Integrations][] index to open the guide for your warehouse or BI tool.
- For lineage issues after connectivity is healthy, see [Lineage troubleshooting][].

---

[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[Integration settings admin documentation]: /docs/catalog/administration/integration-settings
[Integration settings page in the app]: https://app.castordoc.com/settings/integrations
[Integrations]: /docs/catalog/integrations
[Lineage troubleshooting]: /docs/catalog/navigate/lineage/lineage-troubleshooting
[Looker]: /docs/catalog/integrations/data-viz/looker
[Looker Studio]: /docs/catalog/integrations/data-viz/looker-studio
[Salesforce]: /docs/catalog/integrations/data-warehouses/salesforce
[Snowflake OAuth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-oauth
[Zoho]: /docs/catalog/integrations/data-viz/zoho

---

## Airflow setup

This guide walks through the Beta pathway where Apache Airflow sends [OpenLineage][] events to [Marquez][] so Catalog consumes DAG lineage for warehouse tables. Marquez acts as the HTTP backend for those events. If you also push curated DAG URLs with the Catalog Public API, read the [Catalog Airflow hub][] first so automation and lineage stay aligned on naming and Workspace boundaries.

:::info[Beta Feature]

DAG lineage ingestion through Catalog is in Beta. Contact [Coalesce Support][] before you activate the feature or expand environments and table scope.

:::

## Configure Airflow

Connect Airflow to Marquez using OpenLineage, then attach the credentials and metadata Catalog expects.

### Install or Enable OpenLineage in Airflow

The OpenLineage project documents version-specific configuration. Align your Airflow version with their recommendations:

- Airflow 2.3 through 2.6: install `openlineage-airflow` with your dependency management for Airflow workers and schedulers.
- Airflow 2.7 and later: follow current OpenLineage guidance for bundled or package-based setup.

Consult [OpenLineage][] when you upgrade Airflow versions or rebuild your container images.

### Set Environment Variables Airflow Sends to Marquez

Add the variables Marquez exposes for ingestion so each DAG run can post lineage reliably:

- **`OPENLINEAGE_URL`** - Base URL for the lineage HTTP backend, your Marquez instance.
- **`OPENLINEAGE_API_KEY`** - API key Airflow presents so Marquez accepts lineage payloads.
- **`OPENLINEAGE_NAMESPACE`** - Namespace that identifies your Airflow instance in lineage graphs.

Tune values with guidance from OpenLineage, Marquez, and your infrastructure team until test DAG runs send events without errors.

## Share Credentials Catalog Needs From Marquez

Prepare the following details for onboarding with Catalog and Support:

- **Marquez API URL** - Mirrors the effective `OPENLINEAGE_URL` or a comparable API entry point Catalog should call.
- **Marquez API key** - Mirrors `OPENLINEAGE_API_KEY` or a credential Catalog accepts for read access.
- **`OPENLINEAGE_NAMESPACE`** - The namespace configured on Airflow so Catalog correlates lineage with the correct cluster of DAG identifiers.
- **Warehouse namespace Catalog expects** - The namespace OpenLineage uses for your warehouse source so lineage maps onto the right warehouse **Source > Database > Schema** assets.

Keep secrets in secured channels when you transmit them during Beta onboarding.

:::info[Need a Marquez Server]

If Marquez is not deployed in your environment yet, contact [Coalesce Support][] for options before configuring Airflow emission.

:::

Lineage ingestion does not replace automation that pushes **External Links** through the Catalog Public API unless you consolidate responsibilities. Decide whether curated URLs or lineage-derived shortcuts are authoritative for collaborators, especially when names or namespace conventions might diverge across systems.

## Read Marquez and OpenLineage Guidance When Debugging

Upstream projects publish the canonical steps for backends, extractor configuration, and release-specific behavior. Supplement this Catalog page when you troubleshoot connectivity or namespace setup in your environment:

- Use [Marquez][] for foundational terminology and server URL configuration that Airflow targets.
- Use [OpenLineage][] for Airflow-focused steps that validate lineage emission before Catalog consumes events.

## What's Next?

- Return to [Catalog Airflow hub][] when you weigh API-managed links against lineage ingestion or plan both together.
- Read [Links on warehouse tables][] for collaborator-facing External Link behavior Catalog shows after API updates.
- Use [Catalog Public API][] whenever automation drives table link metadata programmatically alongside lineage.

---

[Catalog Airflow hub]: /docs/catalog/integrations/other/airflow
[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[Coalesce Support]: mailto:support@coalesce.io
[Links on warehouse tables]: /docs/catalog/assets/tables#links
[Marquez]: https://marquezproject.ai/docs/quickstart/
[OpenLineage]: https://openlineage.io/docs/integrations/airflow/

---

## Airflow

Catalog supports two integrations with Apache Airflow that enrich warehouse table assets. You can attach curated DAG or web URLs through the Catalog Public API, or you can connect OpenLineage with Marquez so Catalog reads lineage from executed DAGs. The OpenLineage path is in Beta. This page explains how the two approaches differ so you can choose the right one. Step-by-step configuration for lineage is on [Airflow setup][].

:::tip[Different goal]

If you need procedures for running Coalesce pipeline Jobs from Apache Airflow, that workflow is documented in the broader Coalesce platform documentation for deployment and third-party orchestration, not under Catalog integrations. This Catalog section covers metadata and lineage for table assets only.

:::

## Choose How Catalog Connects Apache Airflow to Your Tables

Use these scenarios to choose your starting approach.

### You Want Curated DAG Links Without Lineage

Your automation pushes link metadata onto warehouse tables using the Catalog Public API. Collaborators then open Airflow from **External Links** on each table.

- **Maintenance** - Link rows stay trustworthy when your automation keeps API payloads aligned with DAG names, table identifiers, and URLs.
- **Scope** - This path adds curated navigation. It does not, by itself, ingest lineage from DAG runs.

For authentication, regional base URLs, and the in-app playground, start with [Catalog Public API][]. To learn how External Links behave in the Catalog UI and how Source Links relate for warehouses such as Snowflake or BigQuery, see [Links on warehouse tables][].

### You Want Lineage-Based DAG Associations and Signals in Catalog

You configure Airflow to emit OpenLineage events to Marquez so Catalog reads DAG-to-table lineage and shows pipeline-oriented signals on **Data Quality** for those tables.

- **Operational** - Marquez, OpenLineage, and the Airflow environment variables you set must stay healthy for signals to refresh as DAGs execute.
- **Rollout** - This path is Beta. Coordinate activation and Workspace scope with [Coalesce Support][] before expanding.

Configuration steps are on [Airflow setup][]. For deeper Airflow tuning, see the [OpenLineage Airflow integration][] and [Marquez quickstart][] documentation.

### You Want Curated Links and Lineage Signals Together

Both approaches can run together. Many teams automate explicit External Links for stable navigation and rely on OpenLineage and Marquez for run-derived lineage and Data Quality indicators. Decide who owns naming and namespace conventions on each side so API-driven links stay consistent with lineage identities.

Reach out to [Coalesce Support][] when you split responsibilities between automation and connector teams across Workspaces.

## Push DAG Links and Other URLs Through the Catalog Public API

Authorized automation can attach Airflow DAG URLs or other outbound URLs to a table so they appear among **External Links** for that warehouse table.

Follow [Catalog Public API][] for tokens, regional base URLs, and the API playground. Use the playground and linked API specification to build the authenticated requests your pipeline uses.

<figure><figcaption></figcaption></figure>

## Connect OpenLineage and Marquez for DAG Lineage in Catalog

With OpenLineage and Marquez configured as described on [Airflow setup][], Catalog ties DAG URLs to the warehouse tables your DAG runs touch and exposes lineage-backed status alongside other signals on **Data Quality** for the table.

Collaborators viewing a lineage-connected warehouse table can:

- Open context for the DAG that refreshes the table from lineage metadata.
- See recency indicators when lineage reflects successful DAG runs covered by your setup.

:::info[Beta Feature]

Coverage for OpenLineage through Catalog stays in Beta. Contact [Coalesce Support][] before you extend to new environments or widen production scope beyond what you agreed with Support.

:::

<figure><figcaption></figcaption></figure>

## What's Next?

- Complete environment variables and credentials for lineage on [Airflow setup][].
- Build or extend automation for table URLs using [Catalog Public API][] and confirm link behavior using [Links on warehouse tables][].
- Understand how collaborators read aggregate quality and test content on warehouse tables using [Data Quality on warehouse tables][].

---

[Airflow setup]: /docs/catalog/integrations/other/airflow/airflow-setup
[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[Coalesce Support]: mailto:support@coalesce.io
[Data Quality on warehouse tables]: /docs/catalog/assets/tables#data-quality
[Links on warehouse tables]: /docs/catalog/assets/tables#links
[Marquez quickstart]: https://marquezproject.ai/docs/quickstart/
[OpenLineage Airflow integration]: https://openlineage.io/docs/integrations/airflow/

---

## GitHub and GitLab

Push GitHub and GitLab links to your tables in the Catalog using our dedicated API endpoint.

[API endpoint][]

<figure><figcaption></figcaption></figure>

[API endpoint]: https://apidocs.castordoc.com/#76e4cb49-28d9-4697-b16f-331d1cfb2823

---

## Orchestrators

Connect Catalog to orchestration tools and version control platforms.

Use this table to pick which integration guide matches your goal:

| Method | Best for |
| :----- | :------- |
| [Apache Airflow][] | Linking DAGs or warehouse tables to Airflow using the Catalog API, or wiring OpenLineage with Marquez for run-based lineage on tables. |
| [GitHub and GitLab][] | Pushing repository links onto Catalog tables with the dedicated API so collaborators open the right repo from each asset. |

[Apache Airflow]: /docs/catalog/integrations/other/airflow
[GitHub and GitLab]: /docs/catalog/integrations/other/github-and-gitlab

---

## dbt Tests

:::info

dbt tests integration must come on top of an already existing **dbt** integration in Catalog.

:::

## dbt Tests Integration Requires Two Types of Files

- a manifest
- run results

## Fetching a Manifest

dbt tests integration is client-managed, so you should already [fetch the manifest in your dbt Core integration][]. Only one manifest is needed.

## Fetching Run Results

`run_results.json` files are generated at each run command and contain the results of your tests.

They are located in the same `/target` directory as your manifest [by default][]. You can fetch several run results files and upload them (if your tests are run on different schedules, in different DAGs).

## Catalog Managed

Input your credentials directly in the [Catalog App integration settings][]. We need:

- a **`service token`**

This should be the same as the one provided for your [dbt (Catalog managed) integration][]. We will store your credentials separately for security.

**`The dbt Base URL of your account`**

By default, `https://cloud.getdbt.com` will be used but depending on your region, the URL might change. See more in the [dbt Cloud API documentation][].

For your first sync, it will take up to 48 h and we will let you know when it is complete.

If you are not comfortable giving us access to your credentials, please continue to [Client Managed](#client-managed).

## Client Managed

### 1. One Shot Load

During your Catalog trial we offer a one-shot sync of your tests. Simply send both files to your Catalog sales representative via email or Slack.

### 2. Scheduled Sync

The Catalog team will provide you with:

- `Catalog Identifier` (an id for us to match your dbt files with your Catalog instance)
- `Catalog Token` An API Token

You can then use the `castor-upload` command:

```bash
castor-upload [arguments]
```

#### Arguments

- `-k`, `--token`: Token provided by Catalog

For security purposes, Catalog only provides **write access**. We are careful with permissions because by providing read access, a mistake could create cross-client read access and put data safety at risk. You will only be able to send files, but you will not see which files you have already sent.

See the [Uploader][] documentation.

:::info

When pushing your dbt `run_results` with the Uploader, please use `file_type`: `QUALITY`

:::

### Scheduling the Sync

- You can schedule it using your classic Airflow workflow.
- Or you can do it any way you prefer.

[fetch the manifest in your dbt Core integration]: https://docs.coalesce.io/docs/catalog/integrations/transformation/dbt/dbt-core
[by default]: https://docs.getdbt.com/reference/project-configs/target-path
[Catalog App integration settings]: https://app.castordoc.com/settings/integrations
[dbt (Catalog managed) integration]: https://docs.coalesce.io/docs/catalog/integrations/transformation/dbt/dbt-cloud#catalog-managed
[dbt Cloud API documentation]: https://docs.getdbt.com/dbt-cloud/api-v2#/
[Uploader]: https://docs.coalesce.io/docs/catalog/developer/castor-extractor

---

## Great Expectations

Catalog integrates with **Great Expectations (GX) OSS** to display validation outcomes as **Data Quality** on warehouse tables and columns already in Catalog. Catalog reads **validation result JSON** that GX writes to **Amazon S3** after checkpoint runs. This integration is separate from the Coalesce **Test Utility Package**, which runs GE-style tests inside Transform Nodes.

## What Catalog Ingests From Great Expectations

Catalog pulls **Validation Result** files from S3. Each file represents a checkpoint run for an **Expectation Suite**. Catalog maps results to tables and columns in your linked warehouse when names and schemas match assets already in Catalog.

You see quality signals at **Expectation Suite** level in the **Data Quality** tab on a table or column, not as one row per individual expectation. When a suite fails, the alert description can summarize which expectation types failed.

GX typically writes files under a path such as `validations/<suite-name>/<checkpoint-name>/<timestamp>/<file>.json`. Your validations store prefix must match what Catalog scans for your environment.

## Before You Begin

:::warning[Warehouse integration required]

A **warehouse** integration must already exist in Catalog before the first ingestion of Great Expectations results can complete.

:::

Confirm the following before you add the integration:

- **GX OSS with S3:** Checkpoints persist **validation results** to an S3-backed **validations store**. Storing expectation suites on S3 alone is not enough if checkpoint runs do not write validation JSON there.
- **AWS S3:** Catalog reads validation artifacts from **Amazon S3** for this integration pattern. Coordinate with Coalesce Support if you need a different object store.
- **Great Expectations Cloud:** Catalog does not pull quality results from the GX Cloud API. Use OSS with S3-stored validation results.
- **One warehouse link:** Associate the Great Expectations quality source with **one** warehouse integration so results attach to the correct tables and columns.

## Configure Great Expectations to Write Validation Results

Point GX at an S3 **validations store** so each checkpoint run calls `store_validation_result` and writes JSON Catalog can read.

1. In your GX project, configure a **validations store** with an S3 backend in `great_expectations.yml` (or your equivalent config). Follow the [GX guide for configuring metadata stores on Amazon S3][].
2. Ensure checkpoints use that store when they run, so new files land under your bucket prefix (commonly under `validations/`).
3. Run a checkpoint against tables you expect to see in Catalog, then confirm new `.json` files appear in the bucket.

Catalog matches results to assets using table and schema names from the validation payload. Expectations tied to **custom SQL** assets depend on query parsing; some columns or tables might not map if Catalog cannot resolve them.

## Grant Catalog Read Access to the Bucket

The Catalog integration needs permission to **list** the bucket and **read** validation result objects.

Grant the principal you use for the Catalog integration (for example, the IAM user behind your access key):

- **s3:ListBucket** on the bucket that holds validation JSON
- **s3:GetObject** on the object prefix where GX writes results

If you use more than one bucket, contact [Coalesce Support][] so bucket names are registered for your workspace. The standard integration JSON accepts **access key** and **secret** credentials; bucket scope is configured with Support when multiple buckets apply.

## Add the Integration

1. Open your [integration page][].
2. In the **Data Quality** section, add **Great Expectations**. This integration is Catalog-managed only.
3. Name the integration and enter credentials as JSON in the following format, then save:

```json
{
    "key": "*****",
    "secret": "*****"
}
```

Catalog uses these credentials to read validation result JSON from the S3 buckets configured for your account. Results appear on tables and columns **only when those assets are already integrated** in Catalog from the linked warehouse.

For your first sync, allow up to **48 hours**. Catalog notifies you when ingestion is complete.

## Link the Quality Source to Your Warehouse

After the integration is saved, link the Great Expectations source to the **one** warehouse where you run the validated tables. Without that link, suite-level results will not attach to the right **Data Quality** tab entries.

If you are unsure which warehouse to select, use the same warehouse integration that already catalogs the tables your GX checkpoints target.

## Common Issues

Use this checklist when validation results are missing or only partially appear in Catalog.

- **No Data Quality tab or empty tab:** Confirm the warehouse integration finished its first sync, the GX source is linked to that warehouse, and the table exists in Catalog with matching database/schema/table names.
- **No new results after a checkpoint:** Verify fresh JSON under your validations prefix in S3, IAM allows list and get for the integration principal, and the bucket is registered for your workspace (contact Support if you use multiple buckets).
- **First sync still pending:** Initial ingestion can take up to 48 hours after you save credentials and link the warehouse.
- **Some suites or columns missing:** Query-based expectations might not resolve to a table or column Catalog knows. Check that the asset is integrated and that table names in the validation file align with Catalog.
- **Unexpected JSON or version differences:** GX 0.13.x and 0.18.x use different field shapes. Ingestion is tuned to your validation file format; if upgrades change output, contact Support with a sample file.
- **Looking for Node-level GE tests in Coalesce:** That workflow uses the [Test Utility Package][] in Transform, not this Catalog S3 integration.

## What's Next?

- Review estate-wide coverage in the [Data Quality Dashboard][]
- Compare other [Quality integrations][] such as [dbt Tests][] and [Soda][]
- Send custom payloads with the [Quality API][] when you own the pipeline

---

[Test Utility Package]: /docs/marketplace/package/coalesce_snowflake_test-utility
[GX guide for configuring metadata stores on Amazon S3]: https://docs.greatexpectations.io/docs/oss/guides/setup/configuring_metadata_stores/configure_metadata_stores/
[Coalesce Support]: https://www.coalesce.io/contact
[integration page]: https://app.castordoc.com/settings/integrations
[Data Quality Dashboard]: /docs/catalog/administration/data-quality-dashboard
[Quality integrations]: /docs/catalog/integrations/quality
[dbt Tests]: /docs/catalog/integrations/quality/dbt-tests
[Soda]: /docs/catalog/integrations/quality/soda
[Quality API]: https://apidocs.castordoc.com/#e4604160-95c5-4f6d-88ce-7908f5e13686

---

## Quality

Integrating data quality into Catalog allows you to display test results on the assets covered by those tests.

Use this table to pick the integration that matches how you produce quality signals today:

| Method | Best for |
| :----- | :------- |
| [Coalesce Quality][] | Showing test results from Coalesce Transform on assets you already sync through the Coalesce integration. |
| [dbt][] | Showing dbt test results on dbt-backed assets when you already ship manifests and run results to Catalog. |
| [Monte Carlo][] | Surfacing Monte Carlo monitors on warehouse tables and columns already integrated in Catalog. |
| [Sifflet][] | Displaying Sifflet checks and latest results on the data assets those checks evaluate. |
| [Soda][] | Surfacing Soda Cloud checks and latest results on the warehouse assets those checks target. |
| [API Quality][] | Sending custom scores or third-party quality payloads with the Catalog data quality API when you manage the pipeline yourself. |

<figure><figcaption></figcaption></figure>

---

[Coalesce Quality]: /docs/catalog/integrations/transformation/coalesce/set-up
[dbt]: /docs/catalog/integrations/quality/dbt-tests
[Monte Carlo]: /docs/catalog/integrations/quality/monte-carlo
[Sifflet]: /docs/catalog/integrations/quality/sifflet
[Soda]: /docs/catalog/integrations/quality/soda
[API Quality]: https://apidocs.castordoc.com/#e4604160-95c5-4f6d-88ce-7908f5e13686

---

## Monte Carlo

Catalog integrates with Monte Carlo to display monitors in the Catalog app.

## Requirement

:::warning

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

Catalog needs API access through two elements:

* A key ID
* A secret

Read more about generating API keys in the [Monte Carlo documentation][].

The key you create should:

* be an [Account Service][] one

<figure><figcaption>From Monte Carlo: Only users with the Account Owner role can create account-level keys</figcaption></figure>

* have [Viewers][] Authorizations (or equivalent)

## Add the Integration

On your [integration page][], in the Data Quality section, add Monte Carlo. This integration is only Catalog-managed. Name it and provide the credentials for API access as JSON under the following format and Save.

```json
{
    "keyId": "*****",
    "secret": "*****"
}
```

From there, Catalog will get monitor results and push them to the tables and columns they evaluate, **under the condition that those tables are also integrated in Catalog**.

For your first sync, it will take up to 48 h. Catalog will let you know when it is complete.

[Monte Carlo documentation]: https://docs.getmontecarlo.com/docs/creating-an-api-token
[Account Service]: https://getmontecarlo.com/settings/api/account-service-keys
[Viewers]: https://docs.getmontecarlo.com/docs/authorization
[integration page]: https://app.castordoc.com/settings/integrations

---

## Sifflet

Connect Catalog to Sifflet to display data quality checks and their latest results on your data assets.

## What Does Catalog Get From Sifflet?

Catalog extracts every check and its latest result. These appear on the data assets they are related to.

<figure><figcaption></figcaption></figure>

## Requirements

:::warning

A warehouse integration must already be configured to complete the first ingestion of this integration.

:::

An API token with read access is required. To generate one:

1. Go to Settings, then Access Tokens.
2. Click "New Access Token".
3. Enter a token name and choose Role = Viewer.
4. Click Generate, then copy and store the resulting token.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

:::info[Tenant Name]

Also take note of your tenant name, which is the following part in the URL:

```text
https://<tenant>.siffletdata.com/
```

:::

## Catalog Managed

Input your credentials directly into your Catalog account in the following format:

```json
{
    "tenant": "your-tenant",
    "token": "your API token"
}
```

For your first sync, it will take up to 48 h and we will let you know when it is complete.

---

## Soda

Catalog integrates directly with **Soda Cloud** to display Soda checks.

## What Does Catalog Get From Soda?

Catalog extracts every check and its latest result. These appear on the data assets they are related to.

<figure><figcaption></figcaption></figure>

## Requirements

:::warning

A warehouse integration must already be configured to complete the first ingestion of this integration.

:::

You need to provide API access through two elements:

* API key ID
* API key secret

Read more about generating API keys for use with the Reporting API in [Soda documentation](https://docs.soda.io/soda-cloud/api-keys.html#generate-api-keys-for-use-with-soda-core-or-the-reporting-api). This is read-only access.

## Add the Integration

On your [integration page](https://app.castordoc.com/settings/integrations), in the Data Quality section, add Soda. This integration is only Catalog-managed. Name it and provide the credentials for API access as JSON in the following format:

```json
{
    "api_key": "*****",
    "secret": "*****"
}
```

<figure><figcaption></figcaption></figure>

From there, Catalog will get check results and push them to the datasets they evaluate, **under the condition that those datasets are also integrated in Catalog**.

For your first sync, it will take up to 48 h and we will let you know when it is complete.

---

## Sync Back

Since you can read about all your data in Catalog, you can write about it too. Sync back lets you write documentation in Catalog and have it appear natively on your tables or dashboards.

Use this table to choose a sync back destination and see what Catalog pushes and prerequisites at a glance.

| Target | What syncs | Prerequisites |
| :----- | :--------- | :------------ |
| [Coalesce][] | Table and column descriptions to your Transform Git repository | Catalog admin; Git access so Catalog can push a branch; [Coalesce Transform integration][]; request from **Settings > Integrations** |
| [dbt][] | Table and column descriptions to dbt YAML in your Git repository | Git access to your dbt repo; [dbt integration][]; request from **Settings > Integrations** |
| [Snowflake][] | Descriptions for tables, views, dynamic tables, external tables, and columns as comments in Snowflake | [Catalog-managed Snowflake integration][]; request from **Settings > Integrations**; Snowflake admin permissions to create the dedicated user and grants; key pair authentication; if you use network policies, whitelist Catalog IPs. |
| [Looker (beta)][] | Dashboard descriptions in Looker. Descriptions are limited to 255 characters. | [Catalog-managed Looker integration][], [warehouse integration][], LookML Git read access, completed first ingestion; contact [Coalesce Support][] to request |
| [Tableau (beta)][] | Certified and Deprecated statuses as Tableau tags | [Catalog-managed Tableau integration][]; contact [Coalesce Support][] to request |
| [BigQuery (beta)][] | Table and column descriptions as comments in BigQuery | [Catalog-managed BigQuery integration][]; contact [Coalesce Support][] to request |

:::info[Self-service sync back]

Coalesce, dbt, and Snowflake show a sync back action in **Settings > Integrations** when the integration is configured. For [Looker (beta)][], [Tableau (beta)][], and [BigQuery (beta)][], contact [Coalesce Support][] to coordinate sync back.

:::

Catalog does not overwrite existing descriptions or tags on Snowflake, Looker, BigQuery, or Tableau where those guides document that behavior. Coalesce and dbt push descriptions that are not already on the target branch.

## Troubleshooting

For delayed jobs, stale Git branches, idle Coalesce Transform requests, stale BI metadata after a successful run, and content freeze pauses, use [Sync Back troubleshooting][].

---

[Coalesce]: /docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce
[dbt]: /docs/catalog/integrations/transformation/dbt/sync-back-to-dbt
[Snowflake]: /docs/catalog/integrations/sync-back/sync-back-to-snowflake
[Looker (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-looker-beta
[Tableau (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-tableau-beta
[BigQuery (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-bigquery-beta
[Coalesce Transform integration]: /docs/catalog/integrations/transformation/coalesce/set-up
[dbt integration]: /docs/catalog/integrations/transformation/dbt
[Catalog-managed Snowflake integration]: /docs/catalog/integrations/data-warehouses/snowflake
[Catalog-managed Looker integration]: /docs/catalog/integrations/data-viz/looker#catalog-managed
[warehouse integration]: /docs/catalog/integrations/data-warehouses
[Catalog-managed Tableau integration]: /docs/catalog/integrations/data-viz/tableau
[Catalog-managed BigQuery integration]: /docs/catalog/integrations/data-warehouses/bigquery
[Coalesce Support]: mailto:support@coalesce.io
[Sync Back troubleshooting]: /docs/catalog/integrations/sync-back/sync-back-troubleshooting

---

## Sync Back to BigQuery (Beta)

Push documentation from Catalog to your BigQuery instance. Sync table and column descriptions so they appear as comments in BigQuery.

## Requirements

:::info

You need a [Catalog-managed BigQuery integration](https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/bigquery) to push back information to it.

:::

## How to Sync Back

On your Integration tab (in Settings), click **Sync back to BigQuery**.

<figure><figcaption></figcaption></figure>

## What Can You Sync Back to BigQuery?

Within the BigQuery Sync back feature, you can push a **description** to your tables and columns in BigQuery.

1. Write as many descriptions as you want in Catalog.
2. Request a Sync back to BigQuery from Settings > Integration tab.
3. Once notified that the Sync back succeeded, your comments are updated in BigQuery.

## What Happens If a Description Already Exists in BigQuery?

- **Catalog will not overwrite** any existing description.
- You can write and store another description in Catalog.
- Once you delete the existing description in BigQuery, a Sync back will push the description you wrote in Catalog to BigQuery.

## Do You Need To Provide Additional Access?

No. The permission level of the existing integration remains unchanged.

---

## Sync Back to Looker (Beta)

Push descriptions from the Catalog back to your Looker dashboards.

If your Catalog workspace includes a combined Coalesce, Looker, and Transform setup, this page covers Sync back to Looker only (dashboard descriptions in Looker). To push table and column descriptions to Coalesce Transform, see [Sync Back to Coalesce][].

## Before You Start

Complete these prerequisites before you click **Sync back to Looker**. That way your integration, ingestion, and Catalog descriptions are ready when you request a sync.

- A [warehouse integration][] is configured in Catalog (required before the first Looker ingestion).
- A [Catalog-managed Looker integration][] is configured with Looker admin credentials and an admin API key.
- Catalog has [read-only access to your LookML Git repository][] (needed for Looker ingestion and lineage).
- The first Looker ingestion has finished (this can take up to 48 hours; Catalog notifies you when it completes).
- You have written dashboard descriptions in Catalog for the dashboards you want in Looker.
- You understand that sync back pushes dashboard descriptions only and that Catalog does not overwrite existing Looker descriptions (see [What Happens If a Description Already Exists in Looker?][] below).

You do not need to grant additional Looker or Git access beyond what you already configured for the Looker integration. See [Do You Need To Provide Additional Access?][] below.

## What Can You Sync Back to Looker?

The Looker sync back feature pushes a description to your dashboards in Looker. Catalog does not sync other dashboard fields through this flow.

## How to Sync Back to Looker

To request a sync:

1. Write dashboard descriptions in Catalog for the dashboards you want in Looker.
2. Go to **Settings** and open the **Integration** tab.
3. Click **Sync back to Looker**.

    <figure><figcaption></figcaption></figure>

4. Once Catalog notifies you that the sync back succeeded, your descriptions appear in Looker.

:::warning[255-Character Limit]

Looker descriptions are limited to 255 characters.

:::

## What Happens If a Description Already Exists in Looker?

Catalog does not overwrite existing descriptions on Looker dashboards. If a dashboard already has a description, Catalog leaves it unchanged.

You can still write and store a different description in Catalog. Once you delete the existing description in Looker, sync back can push the description you wrote in Catalog to Looker.

## Do You Need To Provide Additional Access?

No. The permission level of your existing Looker integration stays the same. No extra access is required for sync back beyond what you configured for the Looker integration.

## What's Next?

- [Sync Back hub][]
- [Looker integration][]
- [Sync Back to Coalesce][]

---

[Sync Back to Coalesce]: /docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce
[warehouse integration]: /docs/catalog/integrations/data-warehouses
[Catalog-managed Looker integration]: /docs/catalog/integrations/data-viz/looker#catalog-managed
[read-only access to your LookML Git repository]: /docs/catalog/integrations/data-viz/looker/provide-access-to-your-lookml
[Do You Need To Provide Additional Access?]: #do-you-need-to-provide-additional-access
[What Happens If a Description Already Exists in Looker?]: #what-happens-if-a-description-already-exists-in-looker
[Sync Back hub]: /docs/catalog/integrations/sync-back
[Looker integration]: /docs/catalog/integrations/data-viz/looker

---

## Sync Back to Snowflake

Automatically push documentation from Catalog to your Snowflake instance. You can sync descriptions, also called comments, for tables, views, dynamic tables, external tables, and columns so they appear natively in Snowflake.

## Prerequisites

You need a [Catalog managed Snowflake integration][] to push back information to Snowflake. Configure it first if you haven't already.

## How It Works

### What Is Synchronized?

Catalog synchronizes descriptions for:

- Tables
- Views
- Dynamic tables
- External tables
- Columns

### When Does It Run?

Once Sync Back is enabled, Catalog automatically checks for new or updated descriptions every 24 hours.

### How Are Descriptions Updated?

Catalog only writes descriptions when the corresponding object has no description in Snowflake.
It never overwrites an existing Snowflake comment.

## Setup

Complete the Snowflake procedure, role, and credential configuration below to enable Sync Back.

:::info[Two Separate Setups]

Activating the Snowflake integration data source and setting up Sync Back are two distinct configurations. Completing the integration setup does not enable Sync Back. Follow the steps below to configure it.

:::

### Technical Details

Catalog leverages Snowflake Procedures in order to safely update the descriptions in your Snowflake instance. As Snowflake states, procedures can run user-defined code with privileges of the role that owns the procedure, rather than with the privileges of the role that runs the procedure.

One of the top-level roles on Snowflake will create and own a secure procedure specifically made for updating comments in Snowflake and will delegate to a specific user and role only the ability to call said procedure.

Finally, enter the credentials for the user in Catalog. Use this user for Sync Back only.

For more information on how Stored Procedures work, review the Snowflake documentation:

- [Snowflake Procedures][]

## Before You Start

Complete the steps in the **Technical Details** and procedure sections below.

You'll need one of the following top-level roles on Snowflake to create the Catalog user:

- SECURITYADMIN
- ACCOUNTADMIN
- SYSADMIN

If your configuration allows, you can use different roles.

The role used for this setup needs rights to update comments on all objects in the Snowflake instance. If the top-level role cannot update a specific object, Catalog Sync Back cannot update it either.

## Step 1: Create the Sync Back Catalog User

### 1.1 Create a Dedicated Role

Create a role called `CATALOG_SYNC_BACK_ROLE`. This role inherits the public role from your instance and won't have any rights initially. Assign this role to the Catalog Sync Back user.

```sql
USE ROLE SECURITYADMIN;

CREATE ROLE CATALOG_SYNC_BACK_ROLE;
GRANT ROLE CATALOG_SYNC_BACK_ROLE TO ROLE SYSADMIN;
```

### 1.2 Create a Dedicated Warehouse

Catalog only updates your table and column descriptions once per day and the auto-suspend setup is on.

```sql
USE ROLE SYSADMIN;

CREATE WAREHOUSE CATALOG_SYNC_BACK_WH
WITH WAREHOUSE_SIZE = XSMALL
  AUTO_SUSPEND = 59
  AUTO_RESUME = TRUE
  INITIALLY_SUSPENDED = TRUE
  COMMENT = 'warehouse used for catalog sync-back';
```

### 1.3 Grant Usage on the New Warehouse to New Role

Add rights to the role you created in Step 1.1. This role is granted usage rights to the new warehouse you created in Step 1.2.

```sql
GRANT USAGE ON WAREHOUSE CATALOG_SYNC_BACK_WH TO ROLE CATALOG_SYNC_BACK_ROLE;
```

### 1.4 Create the Catalog User on Snowflake

Create a user called `CATALOG_SYNC_BACK`.

```sql
USE ROLE SECURITYADMIN;
-- You may need to use ACCOUNTADMIN
CREATE OR REPLACE USER CATALOG_SYNC_BACK
    LOGIN_NAME = CATALOG_SYNC_BACK
    DEFAULT_ROLE = CATALOG_SYNC_BACK_ROLE
    DEFAULT_WAREHOUSE = CATALOG_SYNC_BACK_WH;

-- Setting DEFAULT_ROLE does not grant the role. Grant it explicitly, otherwise
-- the user falls back to PUBLIC and cannot use the warehouse or the procedure.
GRANT ROLE CATALOG_SYNC_BACK_ROLE TO USER CATALOG_SYNC_BACK;
```

## Step 2: Create the Snowflake Procedure

Create a procedure called `CATALOG_SYNC_BACK_DESCRIPTION`.

### Step 2.1: Create the Catalog Utility Database

Before creating the procedure, you need a database and schema for it. You can use any database and schema you like. Note its full path, `DATABASE.SCHEMA.PROCEDURE`, because you'll enter it in the Catalog settings later. We recommend a dedicated one:

```sql
USE ROLE SECURITYADMIN;
-- You may need to use ACCOUNTADMIN

CREATE DATABASE IF NOT EXISTS CATALOG_UTILS;
CREATE SCHEMA IF NOT EXISTS CATALOG_UTILS.PUBLIC;
```

### Step 2.2: Create the Sync Back Procedure

Create the Snowflake procedure that will be used by the `CATALOG_SYNC_BACK` user.
This procedure properly treats the inputs in order to prevent the possibility of SQL injections.

```sql
USE ROLE SECURITYADMIN;
-- You may need to use ACCOUNTADMIN

CREATE OR REPLACE PROCEDURE CATALOG_UTILS.PUBLIC.CATALOG_SYNC_BACK_DESCRIPTION(
    OBJECT_TYPE  STRING,   -- 'TABLE' | 'VIEW' | 'DYNAMIC TABLE'
    OBJECT_NAME  STRING,   -- fully qualified: DB.SCHEMA.OBJECT
    COLUMN_NAME  STRING,   -- NULL/'' for object-level comment
    NEW_COMMENT  STRING    -- the description ('' clears it)
)
COPY GRANTS   -- keep existing grants when re-running this CREATE OR REPLACE
RETURNS STRING
LANGUAGE PYTHON
RUNTIME_VERSION = '3.11'
PACKAGES = ('snowflake-snowpark-python')
HANDLER = 'run'
EXECUTE AS OWNER
AS
$$
def run(session, object_type, object_name, column_name, new_comment):
    # 1) OBJECT_TYPE is a keyword, not an identifier -> whitelist it.
    otype = (object_type or '').strip().upper()
    if otype not in ('TABLE', 'VIEW', 'DYNAMIC TABLE'):
        raise ValueError('object_type must be TABLE, VIEW, or DYNAMIC TABLE.')

    if not (object_name and object_name.strip()):
        raise ValueError('object_name is required.')
    obj = object_name.strip()

    # 2) Comment literal: cannot be bound in DDL -> escape single quotes.
    safe_cmt = (new_comment or '').replace("'", "''")

    # 3) Asset name via IDENTIFIER(?) with a BIND -> the value is passed as data,
    #    never concatenated, and IDENTIFIER interprets it as a name only.
    #    Column name: IDENTIFIER() is not accepted in ALTER COLUMN, so we
    #    double-quote it (escaping any embedded ") to contain it safely.
    col = (column_name or '').strip()
    if col:
        col_q = '"' + col.replace('"', '""') + '"'
        stmt = f"ALTER {otype} IDENTIFIER(?) ALTER COLUMN {col_q} COMMENT '{safe_cmt}'"
    else:
        stmt = f"ALTER {otype} IDENTIFIER(?) SET COMMENT = '{safe_cmt}'"

    session.sql(stmt, params=[obj]).collect()
    return 'OK: ' + stmt.replace('?', obj)
$$;
```

:::caution[Signal Failures by Raising]

If you customize this procedure, make sure it **raises an exception** on any failure. Never return an error string instead: Catalog treats a call that returns without raising as a success, and would mark the description as synced even though Snowflake was not updated.

:::

### Step 2.3: Grant Usage on the Database, Schema, and Procedure to the Sync Back Role

```sql
USE ROLE SECURITYADMIN;
-- You may need to use ACCOUNTADMIN

-- The role must be able to traverse the database and schema to reach the procedure.
-- Without these, the procedure call fails with a misleading "does not exist" error.
GRANT USAGE ON DATABASE CATALOG_UTILS TO ROLE CATALOG_SYNC_BACK_ROLE;
GRANT USAGE ON SCHEMA CATALOG_UTILS.PUBLIC TO ROLE CATALOG_SYNC_BACK_ROLE;

GRANT USAGE ON PROCEDURE CATALOG_UTILS.PUBLIC.CATALOG_SYNC_BACK_DESCRIPTION(STRING, STRING, STRING, STRING)
TO ROLE CATALOG_SYNC_BACK_ROLE;
```

### Step 2.4: Test the Sync Back User

The Catalog Sync Back user should now be correctly set up. Before you give Catalog the credentials, run a test to make sure everything works.
Use the created role and run the following statements.

#### Check Warehouse

You must see the warehouse `CATALOG_SYNC_BACK_WH` as the default one.

```sql
USE ROLE CATALOG_SYNC_BACK_ROLE;

SHOW WAREHOUSES;
```

#### Check Role

You must see the role `CATALOG_SYNC_BACK_ROLE` as the current and default one.

```sql
SHOW ROLES;
```

#### Check Procedure

You must see the procedure `CATALOG_UTILS.PUBLIC.CATALOG_SYNC_BACK_DESCRIPTION` in the list of returned values.

```sql
USE ROLE CATALOG_SYNC_BACK_ROLE;

SHOW PROCEDURES IN ACCOUNT;
```

## Step 3: Create a Key Pair

Review the Snowflake [documentation about Key Pair Authentication][] for more details.

### 3.1 Create Private Key

Open the terminal on your computer and generate a private key without a passphrase using the following command:

```bash
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
```

### 3.2 Create Public Key

Generate the public key using the private key you created above with the following command:

```bash
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
```

The key generated will be in the following format:

```text
-----BEGIN PUBLIC KEY-----
MIIBIj...
-----END PUBLIC KEY-----
```

### 3.3 Assign Public Key to the Catalog User

Execute an `ALTER USER` command to add the public key to the Catalog user. Insert the public key without its delimiters in the command. Here is an example:

```sql
ALTER USER CATALOG_SYNC_BACK SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...';
```

## Step 4: Add New Credentials in Catalog App

:::info[Admin Access Required]

You must be a Catalog admin to complete this step.

:::

You can now enter the newly created credentials in the Catalog App.

1. Go to **Settings** > **Integrations**.
2. Click the **Sync Back** button for your Snowflake integration.

   

3. Click the **Edit Sync Back Credentials** button in the Snowflake Sync Back section.

   

4. Add the following credentials.
   1. **Account** field: enter the **Account Locator** of your Snowflake instance. You can find it in either of these ways:
      - From the **hostname**, AWS Private Link, or Azure Private Link endpoint of your instance (for example `example.us-west-1`).
      - By running the following query in a Snowflake worksheet:

        ```sql
        SELECT t.value:host::string AS host
        FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) t
        WHERE t.value:type = 'SNOWFLAKE_DEPLOYMENT';
        ```

        This returns your account hostname, for example `example.eu-west-2.aws.snowflakecomputing.com`. Enter only the part before `.snowflakecomputing.com`, for example `example.eu-west-2.aws`.
   2. **Username** field: enter the `LOGIN_NAME` of the user you created.
   3. **Procedure Path** field: paste the path of the procedure you created, for example `CATALOG_UTILS.PUBLIC.CATALOG_SYNC_BACK_DESCRIPTION` if you did not rename it.
   4. **Private Key** field: paste the private key using the following format:

      ```text
      -----BEGIN PRIVATE KEY-----
      <PRIVATE_KEY>
      -----END PRIVATE KEY-----
      ```

:::caution[Account Format]

Don't add `.snowflakecomputing.com` in the Account form. Only include the account and region.

:::

## What's Next?

- [Sync Back to Coalesce][]
- [Sync Back to dbt][]
- [Snowflake integration][]

[Catalog managed Snowflake integration]: /docs/catalog/integrations/data-warehouses/snowflake
[Sync Back to Coalesce]: /docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce
[Sync Back to dbt]: /docs/catalog/integrations/transformation/dbt/sync-back-to-dbt
[Snowflake integration]: /docs/catalog/integrations/data-warehouses/snowflake
[Snowflake Procedures]: https://docs.snowflake.com/en/developer-guide/stored-procedure/stored-procedures-overview
[documentation about Key Pair Authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth

---

## Sync Back to Tableau (Beta)

## Requirement

:::info

You need a [Catalog-managed Tableau integration][] to push information back to it.

:::

## How To

On your Integration tab (in Settings), click the **Sync back to Tableau** button.

<figure><figcaption></figcaption></figure>

## What Can You Sync Back to Tableau

You can sync back to Tableau statuses input in Catalog ([Certified][], [Deprecated][]).

1. Mark Workbooks in Catalog
2. Request a Sync back to Tableau in your Settings > Integration tab
3. Once notified that the Sync back succeeded, statuses will appear in Tableau

:::info

The tags will appear as Tags in Tableau. You can then filter on tags, for example to see Workbooks that are Certified.

:::

## What Happens If Tags Already Exist in Tableau

- **Catalog will not overwrite** any existing tag
- Tags are added independently

[Catalog-managed Tableau integration]: https://docs.coalesce.io/docs/catalog/integrations/data-viz/tableau#castor-managed
[Certified]: https://docs.coalesce.io/docs/catalog/assets/dashboards#details-panel
[Deprecated]: https://docs.coalesce.io/docs/catalog/assets/dashboards#details-panel

---

## Sync Back Troubleshooting

Use this guide when sync back does not move documentation to your target, when Git branches look stale, when a Coalesce Transform request seems idle, when warehouse or BI metadata still looks old after a successful run, or when you need to pause sync back during a content freeze.

For destination setup and prerequisites, start on [Sync Back][] and open the guide for your target from the comparison table.

## You Requested Sync Back but Documentation Has Not Moved Yet

Jobs queue behind other automation and may wait for validation steps inside Catalog or Git hosts.

**Resolution:**

1. Open the destination-specific guide linked from [Sync Back][] and confirm prerequisites such as tokens, branches, and roles.
2. Allow enough time for scheduled automation before assuming failure. Many teams batch pushes nightly.

## Git-Hosted Destinations Show an Older Branch Than Your Latest Catalog Edits

Branches sometimes refresh only after upstream merges complete or when scheduled extractors rerun.

**Resolution:**

1. Confirm no pending merge conflicts exist on the repository Catalog pushes to.
2. Trigger another sync after large documentation edits so the pipeline captures the newest revision.

## First Sync Back From Catalog to Coalesce Transform Seems Idle

Professional services or operations teams may monitor these requests manually during onboarding.

**Resolution:**

1. Reply to any confirmation thread from your rollout team with timestamps if nothing moves within the agreed SLA.
2. Escalate through [Coalesce Support][] when deadlines slip without status updates.

## Warehouse-Native Sync Writes Succeed in Logs but BI Tools Still Show Old Descriptions

Some targets cache metadata locally until their native refresh cycle runs.

**Resolution:**

1. Force a refresh or republish inside the BI tool after Catalog reports success.
2. Re-read the guide for your beta destination, such as [Looker (beta)][], [Tableau (beta)][], or [BigQuery (beta)][], when APIs remain in staged rollout.

## You Need to Pause Sync Back During Content Freezes

Automation hooks continue until disabled at the source.

**Resolution:**

1. Work with your Catalog administrator to disable schedules or tokens during freeze windows.
2. Document restart steps before re-enabling pushes.

---

[Sync Back]: /docs/catalog/integrations/sync-back
[Coalesce Support]: mailto:support@coalesce.io
[Looker (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-looker-beta
[Tableau (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-tableau-beta
[BigQuery (beta)]: /docs/catalog/integrations/sync-back/sync-back-to-bigquery-beta

---

## Coalesce Owners

Catalog can show **Technical Owners** on tables built with **Coalesce Transform**. These owners reflect who has contributed to the underlying node definitions in your linked Git repository. They are synced into Catalog and are not the same as **Owners** you assign manually in the asset page.

<figure><figcaption>Technical owners on a Coalesce-built table</figcaption></figure>

## Requirements

Owner Auto Detection needs both a **Coalesce Transform API** connection and read access to the Git repositories that store your Coalesce project. See the feature matrix on the [Coalesce integration overview][].

You need Catalog admin access to request GitHub analysis.

Enable the integration:

<ButtonLink to="/docs/catalog/integrations/transformation/coalesce/set-up">
  Set Up
</ButtonLink>

Grant the Catalog technical GitHub user `techcastor` contributor access to each relevant Coalesce repository, as described in [Grant Access to Your Git Repositories][] on the overview page.

## How Top Editors Are Determined

Catalog identifies top editors by analyzing Git history for files that correspond to your Coalesce nodes.

1. Catalog reads contribution data from the current state of your repository (`HEAD`) using Git blame on the relevant node files.
2. For each file, Catalog counts how many lines each author last touched.
3. Authors are ranked by total line count for that node.
4. Catalog keeps up to three contributors per table who meet these rules:
   - They have at least five lines attributed to them on that node.
   - Their commit email maps to your organization’s domain (internal contributors). Personal or unrecognized emails are excluded.

The people who qualify are shown as **Technical Owners** on the matching warehouse table in Catalog. This reflects recent edit activity in Git, not job titles or ownership fields inside Coalesce Transform.

:::info[Technical Owners vs Owners]

**Technical Owners** are imported from the integration and cannot be edited on the asset page. **Owners** are people or teams you assign in Catalog for governance. See [Ownership][] for how both roles work together.

:::

## When Owner Lists Update

Technical Owners for Coalesce aren't refreshed on every Coalesce metadata sync.

| What runs | How often | What it updates |
| --------- | --------- | ---------------- |
| Coalesce Transform ingest | Daily | Node mapping, descriptions, lineage markers, and related metadata |
| Git contribution analysis for owners | On request | Technical Owners from repository contributions |

The daily Coalesce ingest job syncs nodes and descriptions from the Coalesce API. It doesn't recompute Technical Owners from Git on that schedule.

To refresh Technical Owners from Git, follow [Request GitHub Analysis][]. Updated owners appear in Catalog after that run completes. There is no real-time update when someone pushes to Git.

If you need owners to change after large repository changes, request another analysis once those changes are merged.

## Request GitHub Analysis

Use these steps when you need to refresh Technical Owners from Git after merges or ownership changes.

1. Sign in to Catalog.
2. Go to **Settings > Integrations** and open your **Coalesce Transform** source.
3. In **Display Coalesce Owners**, confirm the feature shows as activated and that `techcastor` has access to the repository.
4. Select **Request GitHub Analysis**.

Catalog notifies the operations team to run the analysis. Allow time for processing before you check tables in Catalog.

## Troubleshooting

If Technical Owners are missing, outdated, or not who you expect, work through these checks.

### Technical Owners Look Wrong or Outdated

- Confirm the table is tied to a Coalesce node and that the Git repository Catalog analyzes matches the project you expect.
- Check that recent commits use email addresses on your company domain. Contributors with fewer than five attributed lines or external emails are filtered out.
- Request a new GitHub analysis after merges or ownership changes in Git.

### No Technical Owners on a Coalesce-Built Table

- Verify API credentials and Git access on the [Coalesce integration overview][].
- Confirm **Owner Auto Detection** prerequisites are met.
- Run **Request GitHub Analysis** if owners have never been computed for this source.

### You Expected a Different Person

Technical Owners favor the top three Git contributors by line count on the node file. Someone who owns the pipeline in Coalesce but rarely edits the repository may not appear. Assign **Owners** in Catalog when you need an explicit governance contact.

[Request GitHub Analysis]: #request-github-analysis
[Coalesce integration overview]: /docs/catalog/integrations/transformation/coalesce
[Grant Access to Your Git Repositories]: /docs/catalog/integrations/transformation/coalesce#grant-access-to-your-git-repositories
[Ownership]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/ownership

---

## Coalesce

Our integration with **Coalesce Transform** brings deeper visibility and context to your data lineage, documentation, and quality monitoring. By connecting your Coalesce environment to the Catalog, you can trace, understand, and trust your data pipelines.

## Use Cases

Connect your Coalesce environment to Catalog to unlock the following capabilities.

### 1. **Understand Data Origins at a Glance**

Quickly identify which tables in your warehouse were built with **Coalesce**, directly from the Catalog interface. This gives data teams immediate visibility into how and where transformation logic is applied.

### 2. **Explore Lineage with Visual Cues**

See **Coalesce-built tables** marked with the Coalesce logo directly in the **lineage graph**. This allows for fast troubleshooting and intuitive exploration of upstream dependencies and downstream consumers.

<figure><figcaption></figcaption></figure>

### 3. **Enrich Documentation Automatically**

Pull in **external table and column descriptions** directly from Coalesce, ensuring your documentation stays consistent and up to date across platforms. This feature currently supports:

* **Full-stack integration ✅**: Direct sync of descriptions
* **Extraction-based sync ⚙️**: Pull-only (read-only) metadata

<figure><figcaption></figcaption></figure>

### 4. **Monitor Data Quality**

Activate **Coalesce Quality** as a **Data Quality Source** to bring test results into the Catalog. This helps teams proactively identify issues and surface data quality insights alongside technical metadata.

<figure><figcaption></figcaption></figure>

:::info[Test Sync Timing]

When a new test is added to a transform in Coalesce, it will appear in the Catalog after the next hourly data quality sync following the test's first run. For example, if you create and deploy a test at 9:15 AM, and it runs at 9:45 AM, the test will be visible in the Catalog after the 10:00 AM sync

:::

### **5. Sync Back Descriptions: From Catalog to Coalesce Transform**

<ButtonLink to="/docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce">
  Sync Back Descriptions to Coalesce Transform
</ButtonLink>

## Required Setup

:::warning[Warehouse Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

Two configurations are required to unlock all features:

* Setting up a Coalesce Transform API access
* Granting read access to your Git repositories

See below which features require which setup.

| Feature                          | API Connection | Git Repository Access  |
| -------------------------------- | -------------- | ---------------------- |
| Table or Node: Mapping           | Required       | Not Required           |
| Transform Descriptions Retrieval | Required       | Not Required           |
| Data Quality Test in Catalog     | Required       | Not Required           |
| Description Sync Back            | Required       | Required               |
| Owner Auto Detection             | Required       | Required               |

## Set up the API Connection

In the [Catalog integration page][], scroll to the Transformation or Quality section, click on the Coalesce integration.

For credentials, you only have to paste your Coalesce Transform token, which you can find on your [deploy homepage][].

Replace the placeholders with your host and token, then paste this code:

```json
{
  "host": "https://<your_host>.coalescesoftware.io",
  "token": "<****>"
}
```

## Grant Access to Your Git Repositories

:::info[All Repositories]

Make sure to take this action for all your relevant repositories

:::

### For GitHub

* We only need you to grant contributor access to our technical GitHub user `techcastor` to your Coalesce GitHub repositories so we can push a new branch with the updates. For more details, see [managing teams and people with access to your repository][]
* The GitHub user is available on the [GitHub user page][]

### For GitLab

* We need a Project Access Token (PAT) to your dbt GitLab repository. For more information, see [create a project access token][]
* In token configuration the only scope needed is _write\_repository_ and _Contributor_ role must be selected so we can push a new branch with the updates
* The token should then be sent to our support either through the [support team][] or Slack alongside the URL needed to clone the repository

---

[Catalog integration page]: https://app.castordoc.com/settings/integrations
[deploy homepage]: https://app.coalescesoftware.io/deploy
[managing teams and people with access to your repository]: https://docs.github.com/en/enterprise-cloud@latest/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-teams-and-people-with-access-to-your-repository#inviting-a-team-or-person
[GitHub user page]: https://github.com/techcastor
[create a project access token]: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#create-a-project-access-token
[support team]: mailto:support@coalesce.io

---

## Set Up

## Create the Coalesce Transform Source in the Catalog

:::warning

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

In the [Catalog integration page][], scroll to Transformation, and click the Coalesce integration.

For credentials, paste your Coalesce Transform token. You can find it on your [deploy homepage][].

Replace with your token and paste this code. Update the host if you have a custom one:

```json
{
    "host": "https://<your_host>.coalescesoftware.io",
    "token": "****"
}
```

## Create the Coalesce Quality Source in the Catalog

In the [Catalog integration page][], scroll to the Quality section, and click the Coalesce integration.

For credentials, paste your Coalesce Transform token. You can find it on your [deploy homepage][].

Replace with your token and paste this code. Update the host if you have a custom one:

```json
{
    "host": "https://app.coalescesoftware.io",
    "token": "****"
}
```

[Catalog integration page]: https://app.castordoc.com/settings/integrations
[deploy homepage]: https://app.coalescesoftware.io/deploy

---

## Sync Back to Coalesce

Learn how to sync table and column descriptions from Catalog to Coalesce Transform and Snowflake.

## Why Sync Back?

Sync back lets you document in Catalog and propagate those descriptions to your transformation layer. Here is what you get:

1. Use Catalog to efficiently write table and column descriptions.
2. Automatically push these to Coalesce Transform and Snowflake.
3. Keep all descriptions in sync.

## Requirements for Opening a Merge Request to Your Git Repo

Before requesting a sync back, you must grant Catalog access to your Git repositories. Choose your provider below.

### For GitHub

* We only need you to grant contributor access to our technical GitHub user `techcastor` to your main Coalesce GitHub repository so we can push a new branch with the updates. For more details, see [managing teams and people with access to your repository][]
* The GitHub user is available on the [GitHub user page][]

### For GitLab

* We need a Project Access Token (PAT) to your GitLab repository. For more information, see [create a project access token][]
* In token configuration the only scope needed is `write_repository` and `Contributor` role must be selected so we can push a new branch with the updates
* The token should then be sent to our support either through the [support team][] or Slack alongside the URL needed to clone the repository

## How to Sync Back

:::info[Admin Only]

Only Catalog admins can request a sync back.

:::

You need to have set up your Git repository access. See the [setup guide][] for instructions.

<ButtonLink to="/docs/catalog/integrations/transformation/coalesce/set-up">
  Set Up Git Repository Access
</ButtonLink>

### Request the Sync

Request the sync from the [integrations page][].

<figure><figcaption></figcaption></figure>

In the Catalog app, on the integration list page, click the **Request sync back** button. The Coalesce support team will run the sync back process in the next 24 hours and reach out to you.

### What Happens Under the Hood?

When you request a sync back from the Catalog, it will:

* Create a new branch in your Coalesce Transform Git repositories
* Push to this branch any table or column descriptions in the Catalog but not yet on the Coalesce Transform Nodes

### Review the Changes

When the branch is ready to review, the support team will share the branch name and optional GitHub branch URL. Now you only have to:

* Open your favorite Coalesce Transform Workspace
* Checkout the branch
* Review the last commit

### Merge Them

This is the classic merge and deploy step you would take to deploy, except this time you will add descriptions to Snowflake assets.

* Merge the branch into your production branch
* Deploy again

---

[managing teams and people with access to your repository]: https://docs.github.com/en/enterprise-cloud@latest/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-teams-and-people-with-access-to-your-repository#inviting-a-team-or-person
[GitHub user page]: https://github.com/techcastor
[create a project access token]: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#create-a-project-access-token
[support team]: mailto:support@coalesce.io
[setup guide]: https://docs.coalesce.io/docs/catalog/integrations/transformation/coalesce/set-up
[integrations page]: https://app.castordoc.com/settings/integrations

---

## dbt Cloud

## Requirements

:::warning

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

:::info

You need to be a dbt Cloud administrator to provide the following.

Please reach out to your Catalog point of contact to set up **multiple dbt projects** as a unique integration.
:::

## Catalog Managed

To get started with dbt Cloud in Catalog, you need to provide:

**`A service token`**

Follow the [dbt guide for service tokens][] to generate it.

Then set up those permissions:

- If you are on a _Team plan_: add **Read-Only** to all the relevant projects
- If you are on an _Enterprise plan_: add **Stakeholder** to all the relevant projects

When creating a token, it can only be accessed at creation. Afterwards, it will no longer be visible. Make sure you copy it somewhere safe.

**`The dbt Base URL of your account`**

By default, `https://cloud.getdbt.com` will be used but depending on your region, the URL might change. See more in the [dbt Cloud API documentation][].

**`The ID of your main 'run' job in production`**

Among several jobs you run in production, one often actualizes all tables of your project.

On [dbt Cloud][], go to Deploy > Jobs and select the job that matches the description above. The ID is in the URL, right after `jobs/`.

You can input your API Token and job ID directly in the [Catalog App integration settings][].

<figure><figcaption></figcaption></figure>

For your first sync, it will take up to 48 h and we will let you know when it is complete.

## Going Further

Once your dbt Cloud integration is set up, explore more:

- [dbt Sync back][]
- [dbt Owners][]

[dbt guide for service tokens]: https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens
[dbt Cloud API documentation]: https://docs.getdbt.com/dbt-cloud/api-v2#/
[dbt Cloud]: https://cloud.getdbt.com/
[Catalog App integration settings]: https://app.castordoc.com/settings/integrations
[dbt Sync back]: https://docs.coalesce.io/docs/catalog/integrations/transformation/dbt/sync-back-to-dbt
[dbt Owners]: https://docs.coalesce.io/docs/catalog/integrations/transformation/dbt

---

## dbt Core

Catalog integrates well with [dbt](https://www.getdbt.com/). It only requires the `manifest.json` generated by dbt to be sent to the Catalog whenever it's updated or on a daily basis.

For sync back to dbt, please refer to [Sync Back to dbt](/docs/catalog/integrations/transformation/dbt/sync-back-to-dbt).

:::warning[Warehouse Required]

A Warehouse type integration must already be configured to complete the first ingestion of this integration.

:::

## Where Is the Manifest?

It's within the `target` directory. If you have many manifests files see [Multiple Manifests](#multiple-manifests) below.

:::info[Production Manifest]

Make sure to retrieve the manifest from your production environment. The one that actually points to your production tables.

Otherwise the matching done by `database.schema.table.column` won't work.

:::

## Multiple Manifests
If you have multiple manifests, you should push a singular file called ```manifest.json``` which contains an array of all the manifests.
Something like :
```json
[
  manifest_1,
  manifest_2,
  manifest_3,
  ...
]
```

## How and How Often To Send the Manifest

1. During your trial, you can simply upload it via the Integration page.
2. Afterwards, we'll provide you with a `Source ID` and a `Catalog Token` to use our [Catalog Uploader](/docs/catalog/developer/castor-extractor).

## 1. One Shot Load

During your Catalog trial we offer a one-shot sync of your descriptions. Simply upload it in the [App](https://app.castordoc.com/settings/integrations) when creating your dbt Managed by Yourself integration.

## 2. Scheduled Sync

The Catalog team will provide you with:

1. A `Catalog Source id` (to send your manifest to)
2. A `Catalog Token`

To be used with our uploader:

<ButtonLink to="/docs/catalog/developer/castor-extractor">
  Uploader
</ButtonLink>

We can also share a standalone Python script to upload your manifest without the need of our PyPI package. Please reach out to your Catalog point of contact for further information.

## Scheduling the Sync

* You can schedule it using your classic Airflow workflow.
* If your dbt is on GitHub see the page link below.
* Or you can do it any way you deem better.

<ButtonLink to="/docs/catalog/developer/github-actions-dbt">
 GitHub Actions dbt
</ButtonLink>

## Going Further

Once your dbt integration is set up, seize the full potential of dbt and GitLab:

* [dbt Sync back](/docs/catalog/integrations/transformation/dbt/sync-back-to-dbt)

---

## dbt

<ButtonLink to="/docs/catalog/integrations/transformation/dbt/dbt-cloud">
  dbt Cloud
</ButtonLink>

<ButtonLink to="/docs/catalog/integrations/transformation/dbt/dbt-core">
  dbt Core
</ButtonLink>

<ButtonLink to="/docs/catalog/integrations/transformation/dbt/sync-back-to-dbt">
  Sync Back to dbt
</ButtonLink>

<ButtonLink to="/docs/catalog/integrations/quality/dbt-tests">
  dbt Tests
</ButtonLink>

---

## Sync back to dbt

Now that you've documented your tables and columns in the Catalog, you can sync those descriptions back to your dbt YAML files.

## Requirements

Before requesting a sync back, you must grant Catalog access to your Git repositories. Choose your provider below.

### For GitHub

* We only need you to grant contributor access to our technical GitHub user `techcastor` to your dbt GitHub repository so we can push a new branch with the updates. For more details, see [managing teams and people with access to your repository][]
* The GitHub user is available on the [GitHub user page][]
* Works with both dbt Core and dbt Cloud, as long as the underlying Git repository is on GitHub

### For GitLab

* We need a Project Access Token (PAT) to your dbt GitLab repository. For more information, see [create a project access token][]
* In token configuration the only scope needed is _write\_repository_ and _Contributor_ role must be selected so we can push a new branch with the updates
* The token should then be sent to our support either through the [support team][] or Slack alongside the URL needed to clone the repository

## How to Sync Back

Request the sync from the integrations page, then review the new branch when it appears.

### Step 1: Request the Sync

On the integration page, click the **Request sync back** button.

<figure><figcaption></figcaption></figure>

:::info[GitHub Access Required]

If you haven't granted GitHub access prior to clicking the button, the Catalog team will reach out.

:::

### Step 2: Review the New Branch

A new branch called _castor-sync-back_ will appear on your repository within 24 hours.

<figure><figcaption></figcaption></figure>

---

[managing teams and people with access to your repository]: https://docs.github.com/en/enterprise-cloud@latest/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-teams-and-people-with-access-to-your-repository#inviting-a-team-or-person
[GitHub user page]: https://github.com/techcastor
[create a project access token]: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#create-a-project-access-token
[support team]: mailto:support@coalesce.io

---

## Transformation

Connect the Catalog to your transformation tools. Integrate with dbt and Coalesce to sync metadata, lineage, and documentation.

<DocCardList />

---

## User Guide

Coalesce Catalog helps companies find, understand, and use their data. It is a data catalog that bridges the worlds of business and engineering teams. It is designed for everyone to use without training and to establish consensus.

:::tip[How Does It Work?]

Sounds simple enough, but how does it do that?

:::

<YouTube id="gj2SmryOnIE" />

## Find Your Data

Coalesce Catalog retrieves, processes, and maps all your metadata from your warehouses, BI tools, and transformation tools. It provides a bird's-eye view of every asset: queries, descriptions, frequent users, lineage, and more.

With all this information centralized, you can search through it to quickly find the data you need.

If you are overwhelmed by the volume of data and do not know where to start, Coalesce Catalog computes popularity scores for each table so the most useful tables stand out.

## Understand Your Data

Coalesce Catalog centralizes the list of your tables and dashboards and all the documentation around them.

It scans through tools such as dbt where documentation efforts have already been deployed, and documents tables from popular APIs (for example, Salesforce). All the knowledge surrounding your data can be placed in the Catalog App.

Part of the metadata we extract includes the queries run on the data. This allows us to establish links between different elements and enhances your understanding of where your data comes from and how it was created.

## Use Your Data

After helping you find and understand your data, Coalesce Catalog helps you use it.

With the understanding you have gained, you will trust the analyses performed by others and will not need to duplicate the effort.

To perform new analyses, Coalesce Catalog helps you write your queries. For each table, it lists the related queries so you can use them as a starting point.

---

## Company-Wide Install

Make the Catalog extension available for a group of users or your whole company. You must be a Google Workspace admin to perform the actions below.

Actions are done from [Google Admin](https://admin.google.com/).

## Company-Wide Extension Install

The Catalog Extension ID is: `ahpbhnhlmahkodkgcacbmjaboiaimcjo`

<YouTube id="Av94Rdcfx3U" />

---

## Browser Extensions

Get Catalog metadata directly from your BI tools and SQL editors with the Catalog browser extension.

:::info[Quick Install]

Install from the [Chrome Web Store][].

:::

![Catalog browser extension showing metadata alongside a dashboard](/img/catalog/assets/browser_extension_example.png)

## Supported Browsers

* Chrome
* Edge

## What You Can Do With It

From your BI tools and SQL editors, get access to relevant metadata with the Catalog Chrome extension.

You will see descriptions, owners, and popularity. You can also interact with owners and ask questions about a given dashboard or table.

## How to Install the Extension

Install from [Catalog on Chrome Web Store][] (works for Edge as well).

:::info[Edge Users]

For users on Edge, allow extensions from other stores. See [Add, turn off, or remove extensions in Microsoft Edge][].

:::

## Technologies Covered

We add new technologies every month. The extension currently covers:

* Domo
* [Looker][Looker browser extension]
* Looker Studio
* Metabase
* Power BI
* Sigma
* Strategy
* Tableau
* Qlik

### Coming Soon

* Snowflake
* BigQuery

## Troubleshooting

These fixes cover installation problems and missing Catalog panels inside supported BI tools.

### Extension options show the wrong Catalog region or zone

The widget reads region preferences from extension settings. When the settings screen omits your preferred zone, sign-in attempts may hit the wrong Catalog hostname and return authentication errors.

**Resolution:**

1. Remove the extension, restart the browser, and reinstall from [Catalog on Chrome Web Store][] so Chrome reloads default permissions.
2. Sign in through the Catalog web app first with the correct workspace URL, then reopen the embedded widget inside your BI tool.
3. If region toggles still never appear, capture a screenshot of the extension options screen and contact [Coalesce Support][].

### Sign-in fails after selecting the widget gear icon

Cached credentials or conflicting Chrome profiles block OAuth flows between Domo, Tableau, or other embedded hosts.

**Resolution:**

1. Clear site data for both Catalog and your BI vendor domain, then retry once.
2. Disable conflicting extensions temporarily to rule out script blockers.

### Edge cannot install the package

Edge requires permission to install Chrome Web Store extensions.

**Resolution:**

1. Follow Microsoft guidance linked in **Edge Users** above to allow extensions from other stores.
2. Install using the same Chrome Web Store listing as Chrome users.

### Toolbar panels stay empty even though Catalog has metadata

Catalog must finish indexing assets before the extension can pull descriptions.

**Resolution:**

1. Confirm the underlying integration (warehouse or BI source) completed ingestion in **Settings > Integrations**.
2. Validate that you browse the same Catalog hostname your administrator configured for SSO.

### Technology listed under **Technologies Covered** still shows no overlay

Some hosts load dashboards inside iframes where third-party extensions cannot inject scripts.

**Resolution:**

1. Open the asset in a top-level browser tab when supported by your BI vendor.
2. Ask [Coalesce Support][] whether your exact embedding pattern is on the roadmap.

[Chrome Web Store]: https://chrome.google.com/webstore/detail/castor-chrome-extension/ahpbhnhlmahkodkgcacbmjaboiaimcjo
[Catalog on Chrome Web Store]: https://chrome.google.com/webstore/detail/castor-chrome-extension/ahpbhnhlmahkodkgcacbmjaboiaimcjo
[Looker browser extension]: /docs/catalog/integrations/data-viz/looker/looker-chrome-extension
[Add, turn off, or remove extensions in Microsoft Edge]: https://support.microsoft.com/en-us/microsoft-edge/add-turn-off-or-remove-extensions-in-microsoft-edge-9c0ec68c-2fbc-2f2c-9ff0-bdc76f46b026
[Coalesce Support]: mailto:support@coalesce.io

---

## Catalog and Marketplace Interfaces

Learn how the Catalog and Marketplace interfaces differ, when to use each one, and how to publish consumption-ready data products to the Marketplace.

Coalesce Catalog includes two interfaces: the Catalog and the Marketplace.

## The Catalog: Your Main Interface

The Catalog is the primary interface where you find all assets from your data stack and documentation.

All use cases described in the rest of the documentation refer to the Catalog interface unless explicitly stated otherwise.

## The Data Marketplace: For Data Consumers

The Marketplace is a curated, consumer-friendly interface that showcases only production-ready data products in a simplified, business-oriented view.

### Key Benefits of the Marketplace

The Marketplace helps business stakeholders find trusted data without navigating the full Catalog.

- Assets are organized by Domain, using familiar business language
- Only consumption-ready assets are shown
- It acts as a central entry point for business stakeholders to explore and use data easily

## How to Set Up Your Data Marketplace

### Step 1: Define Guidelines

Before publishing, align your organization on what qualifies an asset for the Marketplace:

- What assets should be surfaced?
- What are the minimum requirements, for example definition, owner, and domain?

Contributors cannot publish assets that do not meet your organization's criteria.

### Step 2: Publish Assets

When an asset meets your guidelines, publish it from the Catalog:

- Open the asset in the Catalog interface
- Enable the **Data product** toggle once all requirements are completed
- The asset is now automatically visible in the Marketplace

### Step 3: Share With Consumers

Share the Marketplace with the audiences who will consume the data:

- Point stakeholders to the Marketplace view
- Filter by domain and share targeted URLs for specific audiences

Marketplace is available on select accounts. Contact [Coalesce Support][] if you want Marketplace enabled for your organization.

## What's Next?

- Review who can publish to the Marketplace in [User Roles in Catalog][]
- See the **Data product** toggle and other asset metadata on [Dashboards][]
- Continue onboarding with the [Catalog Onboarding Guide][]

[Catalog Onboarding Guide]: /docs/catalog/catalog-onboarding-guide
[Coalesce Support]: mailto:support@coalesce.io
[Dashboards]: /docs/catalog/assets/dashboards
[User Roles in Catalog]: /docs/catalog/administration/user-roles-in-catalog

---

## Lineage(Lineage)

Navigate the lineage graph to understand how your data flows between tables, dashboards, and BI tool models.

## What Is It?

### Where Does Data Come From? Does a Dashboard Use This Data?

Lineage answers many questions. Among them:

* You have found an interesting dashboard, but you want to make sure it uses the right source of data
* You know a table has the information you need. Does a dashboard already provide analysis for it?

### Asset Level vs Field Level

When we talk about asset-level lineage, we want to understand how tables, dashboards, and BI tool models are linked to one another.

For more specific questions, field-level lineage helps. You can understand data flows at the column level.

### Sources

Ever wondered which tables are the ancestors of a table or dashboard?

_Sources_ travels up the lineage to the first ancestral elements.

### Parents and Children

Find out which tables and dashboards are directly related to your tables upstream and downstream.

By clicking the "Export Children" button, you can also export and download all lineage children in a CSV file.

<figure><figcaption></figcaption></figure>

## Lineage Graph

Lineage graphs are not only nice to look at, they are also useful.

In Catalog, we want you to focus on the most useful use cases and keep graphs as clear as possible.

**You start small, then expand to get the information you need.**

### Going Downstream

You need to make a modification to a table and check the impact? Or check where a table is used?

By clicking "+" on the right of any table in the graph lineage, you will discover assets created from the selected one.

### Going Upstream

You need to troubleshoot a problem in a table? Checking where the data comes from is always a good idea.

Deep dive into the origins of the data by clicking the "+" on the left of a table.

### Dashboards and Tables

Dashboards and tables come from different technologies. Understanding the link between them is not straightforward.

Catalog can help. Full links between tables and dashboards can be established with Graph lineage.

## Lineage Graph Refresh

Every time your data is synchronized into Catalog, the lineage is refreshed. The lineage graph displayed in Catalog uses data from the last 30 days.

**Still seeing some old links in your graph?** Try updating the time range below so you can see lineage computed on less than 30 days and see the impact of your recent SQL changes.

---

## Lineage Troubleshooting

Use this guide to resolve common lineage issues in Catalog, including missing upstream lineage, truncated queries, temporary table conflicts, and timeout errors.

## Source Query and Upstream Lineage Not Visible

Lineage is computed based on the last 30 days. Confirm that the asset has been refreshed during this period.

## Source Query Visible but No Upstream Lineage

When you see a source query but no upstream lineage, work through the checks below.

### Verify That Parent and Child Assets Are in Catalog

Catalog computes lineage between assets that are available in Catalog. Verify that Catalog has sufficient rights to extract missing metadata.

### Multiple Layers of Temporary Tables

Catalog supports 2 levels of temporary tables by default. If you use more, reach out to your Catalog point of contact to request a higher limit.

## Source Query Visible but Truncated

Catalog supports full visualization of queries that have up to 100,000 characters. When a query passes this limit, it is displayed with `<< TRUNCATED >>` and only table-level upstream lineage is calculated.

## Source Query of Temporary Table Shown for Materialized Table

To match a source query with its associated table, Catalog uses the path of the asset. If you have a temporary table with the exact same name and path as a materialized one, there will be a conflict.

Rename the temporary table so it has a unique name.

## Looker-Specific Lineage Issues

These issues apply to Looker integrations specifically.

### Cannot See Lineage for Looker Assets

Catalog computes lineage by analyzing your LookML. Confirm that Catalog has access to your LookML repo. See the [Looker LookML access documentation][] for more information.

### Lineage Disappears After a Database or Connection Change in LookML

You might see column to field lineage and table to dashboard lineage drop in bulk after you change LookML, a Looker connection, or base view targets so Explores resolve to a different warehouse database than before. The change usually shows up after the next successful Looker sync and lineage processing.

Catalog builds Looker lineage by combining LookML with warehouse metadata that is already in Catalog. Each Explore resolves to concrete warehouse coordinates: database, schema, and tables through the model and connection. Catalog matches lineage only where those names correspond to databases and tables your Warehouse integration has ingested. If Explores point at a database Catalog does not ingest, links that depended on the earlier match can disappear after extraction.

Use these practices to avoid the mismatch:

- **Coordinate ingestion with LookML changes** - Before Explores reference a new database in production, include that database in your Warehouse integration scope, including credentials, allow lists, and any filters your team uses.
- **Align Git branches with what Catalog reads** - If Catalog reads a development branch while production LookML points Explores at production only in another branch, ingestion scope and the branch Catalog indexes must still line up with the coordinates Explores expose.
- **Treat development and production as separate checks** - If only a development database is ingested but LookML switches Explores to production, or the inverse, lineage can fail until both the LookML target and ingestion scope match.

To recover when lineage has already dropped:

1. Identify which database and schema Looker resolves for the affected Explores. Use Looker model documentation or admin tools in your organization to confirm the mapping.
2. Confirm your Warehouse integration ingests that database and that **Settings > Integrations** shows a healthy run for the Warehouse source.
3. Update LookML or the Looker connection so Explores reference ingested databases, then merge or publish the branch Catalog reads for LookML if you use version control.
4. Run a full Looker extraction and upload to Catalog using your normal Catalog-managed or client-managed workflow. See the [Looker integration documentation][] for `castor-upload` and scheduling guidance.

To verify the fix:

1. Wait for the Looker ingestion you triggered to finish.
2. Open a sample Explore in Catalog and confirm column lineage and upstream dashboards look complete again.

If access and warehouse alignment look correct but lineage is still missing, review [Cannot See Lineage for Looker Assets][] and [Some Looker Assets Do Not Have Lineage][] in this guide.

### Some Looker Assets Do Not Have Lineage

Catalog consistently improves the lineage engine with new scenarios. Here is a non-exhaustive list of scenarios not yet covered, with support coming soon:

- `{view_name.SQL_TABLE_NAME}` pattern
- Hidden fields used in calculated fields
- Conditional lineage: Liquid
- Native derived table
- Refined explores

## Power BI Field Lineage After Column Renames

When field lineage in Catalog looks wrong after you rename columns in Power BI Power Query, the limitation is often how mashup text expresses renames, for example with `Table.RenameColumns`, rather than a generic upstream lineage gap. See [Troubleshoot Power BI Lineage When Columns Are Renamed][] for supported patterns, how to validate admin settings, refresh, warehouse sync, and extraction, and when to contact Support.

## Lineage View Is Slow or Returns Timeout Errors

Large environments with many databases, schemas, or tables can produce lineage graphs that are slow to load or return timeout errors. Reducing the amount of data that Catalog ingests is the most effective way to improve performance.

### Reduce Ingestion Scope with Integration Filters

Several integrations support database-level or project-level allow and block lists that control which assets Catalog extracts. Configuring these filters reduces the size of the lineage graph and improves load times.

Use this table to see which integration surfaces filter-style controls today and what they scope.

| Integration | Filter Parameters | What It Scopes |
| --- | --- | --- |
| Snowflake | `--db-allowed`, `--db-blocked`, `--query-blocked` | Databases and query patterns |
| BigQuery | `--db-allowed`, `--db-blocked` | GCP projects |
| SQL Server | `--db-allowed`, `--db-blocked` | Databases |
| Databricks | `--catalog-allowed`, `--catalog-blocked` | Unity Catalog catalogs |
| Looker Studio | `--db-allowed`, `--db-blocked`, `--source-queries-only` | GCP projects and source queries |
| MicroStrategy | `--project-ids` | MicroStrategy projects |
| Tableau | `--skip-columns`, `--skip-fields` | Column and field extraction |
| Power BI | Folder path allow or block patterns configured with your Catalog team; optional Microsoft Fabric capacity boundaries when agreed with your Catalog team | Power BI folders as Catalog stores them in asset paths, often including workspace segments, and capacities that host those workspaces |
| Confluence | `--space-ids-allowed`, `--space-ids-blocked` | Confluence spaces |

Power BI scope is managed on the Catalog integration using folder path patterns rather than the extractor CLI flags listed for some other tools in this table. For design guidance, CI/CD considerations, and how to request changes, read [Power BI Setup][].

For the full list of parameters, see the [castor-extractor documentation][] for your integration.

### Database-Level Filtering in the Lineage View

The lineage view does not support filtering or hiding specific databases from the graph. You can show or hide dashboards and limit lineage to queries from a recent time window, but you can't exclude individual databases after ingestion.

To control which databases appear in lineage, configure allow or block lists at the integration level before the next ingestion run.

### Resolve Timeout Errors

Timeout errors mean the request exceeded the allowed server response window. If reducing your ingestion scope doesn't resolve the issue:

1. Stagger extraction schedules so large integrations don't run at the same time.
2. Retry the request after a few minutes.
3. If the error persists, reach out to your Catalog point of contact with the time stamp of the error and a description of what you were doing when it occurred.

[Looker LookML access documentation]: /docs/catalog/integrations/data-viz/looker/provide-access-to-your-lookml
[Looker integration documentation]: /docs/catalog/integrations/data-viz/looker
[Troubleshoot Power BI Lineage When Columns Are Renamed]: /docs/catalog/integrations/data-viz/power-bi/power-bi-lineage-column-rename
[castor-extractor documentation]: /docs/catalog/developer/castor-extractor
[Cannot See Lineage for Looker Assets]: /docs/catalog/navigate/lineage/lineage-troubleshooting#cannot-see-lineage-for-looker-assets
[Some Looker Assets Do Not Have Lineage]: /docs/catalog/navigate/lineage/lineage-troubleshooting#some-looker-assets-do-not-have-lineage
[Power BI Setup]: /docs/catalog/integrations/data-viz/power-bi/powerbi#scope-power-bi-ingestion

---

## People

This page provides a list of users and their information relevant to your organization. Here you can find email addresses, roles (admin, contributor, viewer), sources where they are connected, creation dates for the accounts, and links to their profiles. This information can be useful for discovering your data.

## People Page

<figure><figcaption></figcaption></figure>

## Profile

This page provides access to features specific to your account. Whether you are looking to manage your dashboards, tables, or knowledge articles, or keep track of your contributions to the platform, this page is the place to be.

### Profile Page: Dashboards

You can view and manage all the dashboards that you own, have favorited, or have edited. Whether you are creating new dashboards, updating existing ones, or exploring the options available, this is the place to do so.

<figure><figcaption></figcaption></figure>

### Profile Page: Knowledge

You can view and manage all the knowledge articles that you own, have favorited, or have edited.

<figure><figcaption></figcaption></figure>

### Profile Page: Tables

You can view and manage all the tables that you own, have favorited, or have edited. From here you can make changes to your tables, such as describing columns.

<figure><figcaption></figcaption></figure>

### Profile Page: Queries

You can access a list of all the queries you have performed. This is a great way to keep track of your own work and that of others.

<figure><figcaption></figcaption></figure>

### Profile Page: Contributions

You can view all your contributions to the platform, including content you have created, edits you have made, and comments you have left. This is a great way to keep track of your progress and see how you and your coworkers are contributing to Catalog.

<figure><figcaption></figcaption></figure>

---

## Search Your Asset

Use Catalog search to move from a name or tag to the right table, dashboard, or knowledge page. Quick search surfaces top matches, while advanced search adds filters for owners, certification, and more.

## Search Bar

Search is a powerful feature in Catalog.

:::info[Keyboard Shortcuts]

At any moment you can press the following shortcuts to run a search:

- From a Mac: ⌘ + K
- From a PC: Ctrl + K

:::

### Quick Search Results

The search is performed through all elements in the Catalog App.

Quick search shows the top results within [Tables][], [Dashboards][], visualization models, and [Knowledge][].

### Advanced Search Results

You can access advanced search results by clicking "Show more."

Advanced search results allow you to filter by result type:

- [Tables][]
- Columns
- [Dashboards][]
- Visualization models: Looker Explores, Tableau data sources, Power BI data sets, Sigma data sets, and Qlik apps
- Fields
- [Knowledge][]

Additionally, in the advanced search you can filter results by:

- is Certified
- has PII
- Tags on columns and assets; you can choose AND, OR, or EXCLUSION behaviors
- Owner: individual contributors or teams; you can choose AND, OR, or EXCLUSION behaviors

<figure><figcaption>Advanced search results</figcaption></figure>

### How Is Search Performed

<BetterMDTables
  headers={['Asset', 'Quick search', 'Advanced search']}
  rows={[
    ['Dashboards', 'Dashboard name', 'Dashboard name, Dashboard description, Dashboard external description, Applied tags'],
    ['Visualization models', 'Visualization model name', 'Visualization model name, Visualization model description, Visualization model external description, Applied tags. If dashboard field check box is checked: Field name and label, Field label'],
    ['Dashboard fields', 'N/A', 'Visualization model name, Field name, Field description, Field external description'],
    ['Tables', 'Database name, Schema name, Table name', 'Database name, Schema name, Table name, Table description, Table external description, Applied tags. If column check box is checked: Column name, Column description, Column external description'],
    ['Columns', 'N/A', 'Column name, Column description, Column external description'],
    ['Knowledge pages', 'Knowledge page name', 'Knowledge page name, Knowledge description, Knowledge applied tags']
  ]}
/>

## Left Hand Menu

One of Catalog's key features is to allow you to search for your data. You can search through the left-hand menu or through the search bar, accessible from everywhere in the Coalesce App.

Catalog organizes your assets in three categories:

- [Data][]: the elements in your warehouses. The hierarchy is defined by:
  - Warehouse technology
  - Database
  - Schema
  - Table
- [Dashboards][]: the elements in your data visualization tools
  - Visualization technology
  - Inherited hierarchy from the technology
- [Knowledge][]: widely used KPIs definition and documentation

## Troubleshooting

These scenarios explain common misunderstandings about Catalog search behavior.

### You Cannot Find Raw SQL Fragments Across Every Dashboard at Once

Catalog indexes curated metadata and fields documented through integrations. It does not provide a global plain-text search across every underlying SQL statement inside third-party BI servers.

**Resolution:**

1. Use advanced search filters for visualization models, fields, and knowledge pages that relate to the metric or column names you remember.
2. When you must trace literal SQL, inspect connector extracts or your BI tool's native query logs for that asset.

### Advanced Search Returns Fewer Assets Than Quick Search

Advanced search applies explicit filters such as certification, PII tags, or owners. Quick search ranks broader matches.

**Resolution:**

1. Clear optional filters you no longer need.
2. Expand asset types (for example include visualization models and fields) when looking for lineage terminology carried on fields rather than tables.

### Keyboard Shortcut Does Nothing

Browser extensions, embedded BI frames, or OS-level shortcuts can intercept ⌘ K / Ctrl + K before Catalog receives it.

**Resolution:**

1. Click inside the Catalog chrome, outside embedded iframes, before invoking the shortcut.
2. Open search through the magnifying glass icon if shortcuts conflict with another application.

### Results Look Stale After Large Documentation Imports

Search indexes refresh after ingestion and indexing jobs complete.

**Resolution:**

1. Wait several minutes after bulk imports or API pushes, then retry.
2. If assets appear in browse trees but not search, contact [Coalesce Support][].

### Left-Hand Explorer Expands but Search Cannot Jump to Deep Tables

Hierarchy navigation and search index slightly different timing depending on ingestion order.

**Resolution:**

1. Navigate through **Data** until the database finishes syncing, then retry quick search on the exact table name.
2. Use advanced search with schema filters when multiple warehouses reuse identical table names.

[Tables]: /docs/catalog/assets/tables
[Dashboards]: /docs/catalog/assets/dashboards
[Knowledge]: /docs/catalog/assets/knowledge
[Data]: /docs/catalog/assets/tables
[Coalesce Support]: mailto:support@coalesce.io

---

## Popularity

<figure><figcaption></figcaption></figure>

## What Is Popularity

> A table or Dashboard popularity tells you how frequently it is used by human users.

The popularity is a score given to data assets from 1 to 1,000,000 and then scaled down to a 5-star system.

The popularity computation is a quantile computation based on the number of queries (for tables) or number of views (for dashboards) amongst all tables and dashboards of the same source.

:::info

We may notably exclude table queries from specific users (settings on the extraction), usually coming from bots or services.

:::

## How Is It Computed

- **Ranking**: We sort the assets with respect to their score _(number of queries or views)_
- **Bucketing**: We put the asset into different buckets according to their rank, also known as "global scoring"
  - There are **8 buckets** of **varying size** with the following thresholds:

    ```tsx
    Bucket 0: assets ranked between 0% and 33% -- Bottom 33%
    Bucket 1: assets ranked between 33% and 48%
    Bucket 2: assets ranked between 48% and 63%
    Bucket 3: assets ranked between 63% and 73%
    Bucket 4: assets ranked between 73% and 83%
    Bucket 5: assets ranked between 83% and 93%
    Bucket 6: assets ranked between 93% and 98%
    Bucket 7: assets ranked between 98% and 100% -- Top 2%
    ```

- **In-bucket ranking**: Within a single bucket, we sort again the assets and share them equally.
- **Final Score**: Finally we compute a score out of the max popularity for all assets of a given source
  - With _**MAX**_ being `1 000 000`
  - With the number of buckets being `8`
- **From the number to the stars**
  - Everything before this step gives us a score that we store in our database. However, another process happens in the frontend to show you the number of stars according to the score.
  - As mentioned, the popularity can range from **0** to **1 000 000** (or be undefined). Then we bucket it down to 11 states, corresponding to stars and half stars: 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5.

## Which Queries Are Used for Computation

First of all, popularity is calculated on 30 days of activities following the last refresh of your source.

Then there are a few exclusions that allow us to determine a more accurate popularity:

- We only use read queries; we want to measure usage, not updates
- We exclude queries that are immense or too small
- We exclude service accounts from the calculation as we want to determine human usage and behavior. Queries by service accounts are reflected in the lineage as you will find parent and child assets there.

## Some Facts

- There is **exactly 1 asset** per source with a perfect score of 1M
- Due to the way the bucketing is done, two assets with the same number of queries might end up in 2 different buckets
- The top 2% of assets are all in the 8th bucket and hence have a score over 875,000 which means a number of stars between 4.5 and 5
- The bottom 33% of assets are all in the first bucket and as such have a score lower than 125,000 which means a number of stars between 0 and 0.5

---

## Quick Set Up

Get Catalog up and running in three steps: sign up, connect your warehouse, and connect your BI tool.

## Sign Up

After your Catalog space is set up, sign up using the link below. Your Catalog admin or Coalesce contact will confirm when your space is ready.

<ButtonLink to="https://app.castordoc.com/auth/sign-up">
  Catalog
</ButtonLink>

## Connecting Your Warehouse

Catalog connects to your warehouse to extract metadata (table and column names, types, lineage, and so on). For full details and supported technologies, see [Warehouses][].

Warehouse onboarding requires a dedicated Catalog user in your warehouse and shared credentials. The dedicated account has very limited access: it can read metadata only, not your data.

:::info[Metadata Only]

The dedicated Catalog user can access your metadata (schema, column names, query logs) but cannot read or modify your data.

:::

### Step 1: Find the Right Person

Identify who can create a user in your warehouse. If you're not sure, ask: "Who is responsible for onboarding new analysts?" That person typically has the right permissions.

### Step 2: Create a User With the Necessary Rights

Have them create a warehouse user with the rights Catalog needs. Technology-specific instructions are in [Warehouses][].

### Step 3: Share Credentials With Catalog

Share the credentials through the Catalog app. Go to **Settings** > **Integrations** and add your warehouse using the credentials you created.

### What Happens Next

Catalog tests the connection, applies [settings](#settings), and runs the first sync (usually a couple of hours). You'll be notified when the first sync finishes. After that, Catalog syncs once per day with your warehouse, so table changes appear in Catalog the next day.

## Connecting Your BI Tool

Catalog ingests metadata from your BI tools (dashboards, reports, data sources) to power discovery and lineage. For full details and supported technologies, see [BI Tools][].

BI tool onboarding follows one of two paths:

* **Catalog managed**—You share admin credentials; Catalog handles extraction and daily syncs. No further action on your part.
* **Client managed**—You run the `castor-extractor` package locally to extract metadata and send the output files to Catalog. Catalog never accesses your BI tool directly.

:::info[Client Managed: Your Data Stays Local]

Client-managed integration requires some setup on your side, but Catalog never gets access to your BI tool. Catalog only receives the metadata files you send.

:::

### Catalog Managed

Securely share credentials with access to your tool's API. Catalog uses those credentials to extract metadata once per day. During onboarding, you'll be notified when the first sync completes.

### Client Managed: During Your Trial

Catalog performs a one-time load of your BI tool metadata. You run the extractor and send the output.

1. Find someone in your organization with access to your tool's API.
2. Have them run the package locally or duplicate the [Colab Notebook][] to perform extraction.
3. Share the output files through Slack or email.

Catalog uploads the files and notifies you when it's done.

### Client Managed: After Your Trial

To keep your BI tool metadata in sync, you'll schedule extraction and push the output files to a GCP bucket that Catalog provides. Sync frequency is up to you, up to once per day. Catalog provides credentials, Catalog IDs, and Python scripts to push to GCP (in the `castor-extractor` package).

## Other Integrations

:::tip[Have Existing Descriptions?]

See [Upload existing descriptions][] to load table and column descriptions from a CSV.

:::

<details>

<summary>dbt</summary>

Catalog extracts existing documentation and tags from dbt. If you use dbt but have no documentation yet, you can skip this. Otherwise:

* **During trial**—Send your dbt manifest through Slack or email.
* **After trial**—Schedule a push of your manifest to a GCP bucket that Catalog provides.

</details>

<details>

<summary>Slack App</summary>

Follow the steps in the Coalesce Catalog app and have your Slack admin approve the integration.

</details>

<details>

<summary>Microsoft Teams App</summary>

Activate the connector from the Coalesce Catalog Integrations page and follow the steps. A Microsoft 365 admin is required to grant the necessary permissions.

</details>

## What If a Technology Isn't Covered?

If your warehouse or BI tool isn't listed as a supported integration, you can use the [Warehouse API][] or [Dashboard API][]. Catalog provides templates for the metadata format it needs; you explore what metadata you can access and format it to match.

If the scope feels large, start by filling in a few examples by hand. That lets you test Catalog's main features before scaling up. Contact the sales team for more details.

## Settings

Catalog manages these settings. You can view or request changes at any time.

### Account Settings

<details>

<summary>Account Domains</summary>

This setting controls who can create an account linked to your Catalog space. It's configured based on the email domains of people from your organization who have been in contact with Coalesce. Multiple domains are supported. Contact Coalesce to modify it.

</details>

<details>

<summary>Sign-In Strategies</summary>

By default, everyone with the right domain can use Google SSO or sign up or sign in directly in the app.

**Other strategies**—Okta SSO can be set up (outside of trial). See [Okta SSO setup instructions][].

**Limiting strategies**—Sign-in can be restricted to a single strategy. Contact Coalesce to modify.

</details>

<details>

<summary>Default User Role</summary>

By default, everyone who signs up to Catalog gets the ADMINS role. Other roles are CONTRIBUTOR and VIEWER. Contact Coalesce to change this.

</details>

### Warehouse Settings

<details>

<summary>Table Allow or Block</summary>

By default, Catalog includes all databases and schemas in your warehouse. The only constraint is that Catalog can handle approximately 300,000 columns. Allowlisting or blocklisting at the database or schema level is possible. Contact Coalesce to configure it.

</details>

<details>

<summary>User Classifications</summary>

Catalog classifies accounts into three types: human, BI tool, and other non-human. This affects two features:

* **Popularity**—Computed from the number of READ queries performed by human and BI tool accounts.
* **Read Queries**—Only human read queries appear in each table's "queries" tab.

</details>

## What's Next?

* [Upload existing descriptions][] if you have table or column docs elsewhere
* [Explore Catalog][] to search, discover lineage, and document your data

---

[Warehouses]: https://docs.coalesce.io/docs/catalog/integrations/data-warehouses/index
[BI Tools]: https://docs.coalesce.io/docs/catalog/integrations/data-viz/index
[Colab Notebook]: https://colab.research.google.com/drive/1e6fDexKyLWRykocmQsw6nZ1EpEuGuzis
[Upload existing descriptions]: https://docs.coalesce.io/docs/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions
[Warehouse API]: https://docs.coalesce.io/docs/catalog/developer/catalog-apis/warehouse-importer
[Dashboard API]: https://docs.coalesce.io/docs/catalog/developer/catalog-apis/bi-importer
[Okta SSO setup instructions]: https://docs.coalesce.io/docs/catalog/administration/authentication/okta
[Explore Catalog]: https://docs.coalesce.io/docs/catalog/use-cases/five-minute-quick-start-guide

---

## Catalog AI Safety

Catalog AI is powered by [OpenAI][]. Rest assured, your metadata is shared only if you choose to activate it on your account.

Catalog AI does not use ChatGPT but uses the same underlying AI models. OpenAI will not use your metadata to retrain their models (see [OpenAI API data usage policies][]).

## What Is Shared With OpenAI?

Catalog only has access to your metadata. No actual data can be shared with OpenAI. Nevertheless, we are extra careful with what we share with OpenAI, and rest assured, your metadata is shared only if you choose to activate it on your account.

### Query Explainer and Scribe

For [Query Explainer][] and [Scribe][], the metadata shared with OpenAI includes:

- SQL queries
- Table names
- Column names

For a query to be sent to OpenAI, the following conditions must be met:

- An admin of your account requested Catalog to activate the AI features
- A query is displayed in Catalog: our PII removal engine cleans all queries displayed from potential PII
- A user requests an explanation for a given query or an associated table or dashboard

For table names and column names to be sent to OpenAI, the following conditions must be met:

- An admin of your account requested Catalog to activate the AI features
- A table is present in Catalog
- A user requests documentation by AI

Then, and only then, is your metadata sent to OpenAI and your user will see the output.

No information about your users is shared with OpenAI.

## U.S. Privacy Laws and GDPR

Catalog and OpenAI share a Data Processing Addendum in addition to OpenAI's standard terms, where OpenAI agrees to comply with data privacy and data protection laws, including U.S. Privacy Laws and GDPR.

[OpenAI]: https://openai.com/
[OpenAI API data usage policies]: https://openai.com/policies/api-data-usage-policies
[Query Explainer]: https://docs.coalesce.io/docs/catalog/document-your-data/query-explainer
[Scribe]: https://docs.coalesce.io/docs/catalog/document-your-data/catalog-scribe/index

---

## How to reach us

Reach the Catalog team through a shared channel, in-app support, or email.

## Shared Slack or Teams channel

We systematically offer to create a Slack Connect channel with your team and our Customer Success team. We frequently have Product Managers and Solution Engineers joining for direct interactions.

If you do not have such a channel, you may want to reach out to your Catalog admin to be added to the existing channel.

If you're an admin, use the button below to learn how to create a Slack Connect channel.

<ButtonLink to="https://slack.com/intl/en-au/resources/using-slack/getting-started-with-slack-connect">
  Getting started with Slack Connect
</ButtonLink>

## In-App Support

We offer in-app support through live interactions with our Customer Success and Support teams.

<figure><figcaption></figcaption></figure>

## Email Support

You can always contact our [support team][].

---

[support team]: mailto:support@coalesce.io

---

## Service Level Terms

Catalog will provide the Service to Customer with at least 98% availability measured on a monthly basis (downtime is a "Disruption"); provided that it will not be a Disruption if the Service is unavailable because of (i) general internet problems or outages caused by power supply carriers; (ii) malfunction of equipment, systems software, network connections or other infrastructure not owned or operated by Catalog; (iii) events beyond Catalog's reasonable control; or (iv) scheduled service or maintenance or reasonable emergency maintenance.

Catalog will credit Customer 5% of the monthly portion of Service fees for each period of 60 or more consecutive minutes of Disruption; provided that Catalog will issue a maximum of one such credit per day and a monthly maximum of five credits.

---

## Advanced Search

Advanced Search is powerful when you know some information about your asset, such as name, a term in the description, owner, tag, domain, and more.

Use the search bar to type the name or partial name of an asset, keywords that are part of a description of tables, dashboards, reports, data sets, columns, and fields, as well as knowledge pages.

Use the filters on the right to reduce the scope of your search.

Learn more about the [Advanced Search screen][].

<figure><figcaption></figcaption></figure>

[Advanced Search screen]: https://docs.coalesce.io/docs/catalog/navigate/search

---

## AI Search

Whether you are looking to define a business metric, locate a popular dashboard, find tables related to a specific topic, or understand how a field is calculated, Catalog's AI assistant can help.

Integrated directly into the Catalog, in your company conversational tool, and in your BI tool, the assistant leverages advanced AI-powered search and natural language processing to help you discover data assets faster and more intuitively. Instead of manually navigating through documentation or relying on tribal knowledge, you can simply ask questions in plain language, such as "What does the `customer_lifetime_value` metric mean?" or "Where does the 'revenue' field come from?" and get accurate, context-aware responses.

Designed to reduce friction in your data discovery process, the AI assistant accelerates your ability to find, understand, and trust your data. Learn more about how Catalog's AI assistant can transform the way you explore data in the [AI assistant introduction][].

<figure><figcaption></figcaption></figure>

[AI assistant introduction]: https://docs.coalesce.io/docs/catalog/ai-assistant/introduction

---

## Data Discovery

Catalog centralizes metadata from your warehouses, BI tools, and transformation tools into a single, unified experience, making data discovery seamless and intuitive.

With Catalog, your data is instantly accessible. Easily search or browse across domains, metrics, dashboards, tables, and columns. Find answers to your questions, understand the context behind your data, and uncover the story powering your organization's knowledge.

Catalog offers six starting points depending on how well you know the information you are searching for.

Use this table to choose where to start:

| Method | Best for |
| :----- | :------- |
| [Navigation Menu][] | You know how your environment is organized and want to browse databases, schemas, and folders in the left panel. |
| [Lineage][] | You know a dashboard or table and want to follow relationships to related upstream or downstream assets. |
| [People and Teams][] | You know who owns or uses the asset, or you want to see what matters to a team from usage and ownership. |
| [Advanced Search][] | You have a name fragment, description keyword, owner, tag, domain, or similar detail and want filters to narrow results. |
| [AI Search][] | You want natural-language questions about a metric or field and AI-assisted answers inside Catalog. |
| [Knowledge Section][] | You are looking for metrics, terms, or domain definitions with links to related tables, fields, dashboards, and reports. |

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

For an even more seamless experience, use the AI assistant to find what you are looking for. Whether you are looking for a metric definition, a table, or a reference dashboard, AI Search and the AI conversational assistant will help you get an answer. Learn more in the [AI assistant introduction][].

<figure><figcaption></figcaption></figure>

[Navigation Menu]: /docs/catalog/use-cases/discover/navigation-menu
[Lineage]: /docs/catalog/use-cases/discover/lineage
[People and Teams]: /docs/catalog/use-cases/discover/people-and-teams
[Advanced Search]: /docs/catalog/use-cases/discover/advanced-search
[AI Search]: /docs/catalog/use-cases/discover/ai-search
[Knowledge Section]: /docs/catalog/use-cases/discover/knowledge-section
[AI assistant introduction]: https://docs.coalesce.io/docs/catalog/ai-assistant/introduction

---

## Knowledge Section

Not everything is documented on a dashboard or a table. Some concepts, such as metrics, specific terms, or domain definitions, have their own pages, typically found under the Knowledge section.

These pages can include links to related data assets. For example, a metric page will have its definition but can also include links to the tables or fields used in generating that metric, or to dashboards and reports where analysis on that metric has been produced.

These links are found under the [pinned assets][] section of the page.

<figure><figcaption></figcaption></figure>

[pinned assets]: https://docs.coalesce.io/docs/catalog/document-your-data/pinned-assets

---

## Lineage(Discover)

When you are looking for an asset related to a dashboard or a table you are familiar with, you can navigate to the asset you know and use lineage to find the asset you are looking for.

<figure><figcaption></figcaption></figure>

---

## Navigation Menu

When you are familiar with the structure of your data environment, you can navigate in the left panel through databases, schemas, and folders to find the tables or dashboards you are looking for.

<figure><figcaption></figcaption></figure>

---

## People and Teams

Use the People or Teams page when you know who is using the asset you are looking for. See what they own, frequent queries, tables for which they are frequent users, and knowledge or asset contributions.

This can be useful when you are interested in knowing what is important for a given team based on their usage.

<figure><figcaption></figcaption></figure>

---

## AI-Powered Descriptions

The Catalog offers AI-powered documentation generation to help you quickly create high-quality descriptions for your tables, dashboards, and columns using available metadata and lineage context. With a few clicks, you can reduce the time and effort required to keep clear, consistent documentation across your data environment.

## AI-Powered Descriptions for Columns

You can leverage AI to document your columns as well by clicking on this button  in the Description column on your table.

It will generate the documentation for the first thirty columns at a time. For each documented column you'll see "From AI Assistant" so you can easily identify them.

## AI-Powered Descriptions for Tables and Dashboards

AI can also be leveraged to generate descriptions at the asset level.

### Key Capabilities

* **Metadata-aware generation** - The AI uses table and dashboard metadata, along with upstream lineage, to generate accurate and relevant descriptions.
* **Template-driven consistency** - You can structure documentation with your [description templates][], which keeps tone aligned with internal standards.
* **Multi-query insight** - The assistant queries upstream dependencies to add context beyond a single metadata snapshot.
* **Faster, more complete documentation** - You can produce comprehensive documentation in seconds instead of lengthy manual passes.

### How To Use It

1. **Navigate to the documentation section** of any Table or Dashboard in the Catalog.
2. Click **"Describe with AI"**.
3. **Select a template** that matches your documentation standards.
4. The AI analyzes the available metadata, applies the selected template, and auto-generates a description ready for review or immediate use.

> Tip: You can always edit or enhance the generated content to better fit your specific use cases or internal standards.

<figure><figcaption></figcaption></figure>

## Troubleshooting

These situations appear when AI workflows overlap with permissions, templates, or automation.

### Public API Calls Return Errors When Writing Descriptions Programmatically

Workspace-level gates can block GraphQL mutations such as `updateTableDescriptions` even when read queries succeed with the same token.

**Resolution:**

1. Confirm with your administrator whether programmatic documentation writes are enabled for your workspace. Review [Catalog Public API][] for supported authentication flows.
2. Request activation through [Coalesce Support][] if automation requires description-write scopes your token cannot unlock yet.
3. Retry mutations after activation propagates (typically minutes, not hours).

### Column Batches Stop After Thirty Columns

The in-product generator processes columns in chunks so browsers stay responsive.

**Resolution:**

1. Run the AI describe action again on the remaining columns.
2. Prioritize high-impact columns first when tables are very wide.

### Generated Text Does Not Match Internal Vocabulary

Templates steer tone, but AI still blends terminology from lineage and metadata snapshots.

**Resolution:**

1. Choose the template closest to your internal style guide under **Describe with AI**.
2. Edit generated paragraphs manually and save; Catalog stores curator edits as the source of truth.

### AI Cannot Infer Lineage Context for an Asset

Upstream lineage must exist in Catalog before the assistant can reference it.

**Resolution:**

1. Finish warehouse and BI ingestion jobs so lineage graphs populate.
2. Link related tables manually through knowledge pages when integrations lag behind governance deadlines.

### Duplicate Descriptions After Rerunning Automation

Repeated runs without reviewing drafts can stack similar paragraphs.

**Resolution:**

1. Delete outdated AI paragraphs before regenerating content.
2. Coordinate schedules so API jobs and UI actions do not race on the same asset during testing.

[description templates]: /docs/catalog/use-cases/document/content-and-context/descriptions/templates-examples
[Catalog Public API]: /docs/catalog/developer/catalog-apis/public-api
[Coalesce Support]: mailto:support@coalesce.io

---

## Descriptions

Choose the right approach for adding descriptions to your Catalog assets:

| Method | Best for |
| :----- | :------- |
| [Ingested from source][] | Descriptions already in your warehouse, BI tool, or dbt. Catalog pulls them automatically. |
| [Upload CSV][] | Migrating from Snowflake or BigQuery exports, spreadsheets, or tools Catalog doesn't connect to. |
| [Column propagation][] | Downstream columns that pass through unchanged from a single parent. Descriptions propagate automatically. |
| [Metadata Editor bulk edit][] | Adding or replacing descriptions across many assets at once. The Metadata Editor page covers **Select all matching assets**, filter scope, and how many assets **Apply All** can process in one run. |
| [AI-powered descriptions][] | Generating descriptions from metadata and lineage when you don't have them yet. |
| [User-generated][] | Writing descriptions manually in the Catalog UI for individual assets. |

[Ingested from source]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/descriptions/ingested-descriptions-from-source
[Upload CSV]: https://docs.coalesce.io/docs/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions
[Column propagation]: https://docs.coalesce.io/docs/catalog/document-your-data/catalog-scribe/column-description-propagation
[Metadata Editor bulk edit]: https://docs.coalesce.io/docs/catalog/document-your-data/metadata-editor
[AI-powered descriptions]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/descriptions/ai-powered-descriptions
[User-generated]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/descriptions/user-generated-descriptions

---

## Ingested Descriptions from Source

If documentation already exists in the source where assets live, Catalog can pull that information in automatically during metadata ingestion. This includes column, field, table, and dashboard descriptions.

**Data warehouses:** Snowflake, BigQuery, and Databricks (using Unity Catalog) ingest table and column descriptions from `INFORMATION_SCHEMA`, DDL comments, and metadata.

**BI tools:** Looker, Tableau, and Power BI bring in dashboard and field descriptions from semantic models and workbooks.

**Transformation tools:** dbt (from `manifest.json` and `schema.yml`) and Coalesce Transform (node and column descriptions) flow descriptions into Catalog automatically.

**SaaS sources:** Many Fivetran-connected sources (for example, Salesforce, HubSpot, Google Ads, Facebook Ads, Marketo) auto-document standard fields from API documentation.

Once ingested, you're free to leave the source descriptions as they are or enrich them by adding a Catalog-native description.

This method helps keep documentation in sync with upstream systems and reduces duplication of effort. It’s especially useful when teams already maintain good metadata practices at the source.

<figure><figcaption>In this example, a table description was imported directly from Snowflake and a Catalog description has been added on top.</figcaption></figure>

## Troubleshooting

If descriptions don't appear or behave unexpectedly, use this table to diagnose and resolve common issues.

| Symptom | Likely cause | Action |
| :------ | :----------- | :----- |
| Descriptions not appearing | Source lacks descriptions, Catalog lacks access, or ingestion not run | Confirm the source has descriptions and Catalog has access. Check that the ingestion schedule has run. |
| Ingestion configured but descriptions still missing | Configuration or sync issue | Contact [support@coalesce.io](mailto:support@coalesce.io) for assistance. |
| Existing assets vs net-new | Timing of when descriptions appear | For existing assets, descriptions are pulled on the next ingestion run. For net-new assets, descriptions come in with the first ingestion. |

If your source descriptions can't be ingested (for example, custom sources or tools Catalog doesn't connect to), use [Upload existing descriptions](/catalog/document-your-data/upload-existing-documentation/upload-existing-descriptions.md) to load them from a CSV.

---

## Template Examples

Templates for asset documentation in Catalog allow administrators to establish standardized formats for documentation, ensuring consistency and clarity across all materials. By defining templates, admins can guide contributors in structuring their documentation, thereby improving its quality and usability.

Below you will find template examples created by the Catalog team to help you document your assets. These templates can be filled out manually but, more importantly, can also be used as prompts for [AI-powered descriptions][].

We provide template examples for:

- [Tables](#table-template-example)
- [Dashboards](#dashboard-template-example)
- [Metric or Term](#metric-or-term-template-example)
- [Domains](#domain-template-example)

Copy and paste them in your [Templates section][] to help you get started with your documentation initiative.

## Table Template Example

### Table Purpose

**Replace this line with the answer (and repeat below for all the questions):** Why the table exists or main use cases?

### Table Audience

Who should use it?

### Table Content

#### Table Key Fields

Mention the fields related to key metrics and concepts (this helps in the discovery process in particular if the metric has a different name than the technical field).

### Table Good to Know

Any calculations, filters, or exceptions that are worth mentioning?

### Table Source

Where is the data sourced from?

### Table Data Update Frequency

How often should the data be updated?

---

## Dashboard Template Example

### Dashboard Purpose

**Replace this line with the answer (and repeat below for all the questions):** Describe why the dashboard exists or main use cases.

### Dashboard Audience

Who is the target audience?

### Dashboard Good to Know

Any calculations, filters, or exceptions that are worth mentioning?

### Dashboard Content

#### Dashboard Metrics

What are the key metrics included in this dashboard? Provide links to their definitions (do not forget to pin all the relevant dashboards to the metric definition).

#### Dashboard Dimensions

What are the key dimensions included in this dashboard? Provide links to their definitions (do not forget to pin all the relevant dashboards to the definition).

### Dashboard Source

Where is the data sourced from (Salesforce, Jira, etc.)?

### Dashboard Data Update

How often is the data updated?

---

## Metric or Term Template Example

### Metric Purpose

**Replace this line with the answer (and repeat below for all the questions):** Why the metric or business term (dimension) exists or main use cases?

### Metric Audience

Who should use it?

### Metric Calculation

#### Metric Key Fields

How is the metric or business term (dimension) calculated?

### Metric Good to Know

Any filters or exceptions that are worth mentioning?

### Metric Source

Where is the data sourced from?

### Metric Data Update Frequency

How often is the metric updated?

---

## Domain Template Example

### Mission

What is the objective and scope of the domain?

### Key People

Who are the key people and their roles?

### Domain Organization

How is it organized? Which departments does it serve (you can use tags)?

### Key Performance Metrics

Key KPIs.

### Key Business Concepts

Key concepts and glossary terms.

### Key Assets and Dashboards

What are the main go-to dashboards and source tables?

### Anything Else You Should Know

Useful presentations or links.

[AI-powered descriptions]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/descriptions/ai-powered-descriptions
[Templates section]: https://app.castordoc.com/governance/templates

---

## User-Generated Descriptions

Documentation makes your data easier to discover, understand, and trust. It helps reduce knowledge gaps, supports onboarding, and enables better collaboration across teams.

One way to document assets is by writing descriptions manually. You can add information to different parts of an asset:

- Use the **README section** on table and dashboard pages for rich, structured documentation at the asset level.

<figure><figcaption></figcaption></figure>

- Use the **description field** to add short descriptions to individual **columns** and **fields**.

<figure><figcaption></figcaption></figure>

When working in the README section of a table or dashboard, you can also choose to apply a **template**. Templates provide a consistent structure, for example, sections like overview, data sources, usage, and limitations. You fill out the template manually, helping keep documentation aligned across similar assets.

Manual documentation is especially helpful when you want to capture business context, describe how data is used, or share important notes that are not available in source systems.

---

## Governance Concepts

Catalog connects business knowledge to tables, dashboards, and knowledge pages through governance features that link assets, assign accountability, and record metadata changes over time.

Use this table to pick the concept guide that matches what you need to set up first:

| Method | Best for |
| :----- | :------- |
| [Tags][] | Grouping assets with domain, classification, or lifecycle labels so you can filter and browse consistent groupings. |
| [Pinned Assets][] | Adding cross-links and mentions so an asset surfaces related tables, dashboards, or knowledge next to the record you already opened. |
| [Ownership][] | Assigning accountable owners so documentation and stewardship stay clear as assets change. |
| [History][] | Reviewing an audit trail of metadata changes for governance reviews on tables, dashboards, and knowledge pages. |

[Tags]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/tags
[Pinned Assets]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions
[Ownership]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/ownership
[History]: /docs/catalog/collaborate/history

---

## Ownership

Ownership is an important data governance concept in Catalog. Each asset should have an owner responsible for its documentation and maintenance.

:::tip[Owner Candidates]

When you choose owners, keep the following in mind:

- The owner of an asset could be an individual or a team.
- Each asset can and should have one or more owners.
- Frequent users, editors, and technical owners are often good candidates to be owners of data and BI assets.

:::

<figure></figure>

## What It Means To Be Assigned Owner

As an owner, you are:

- Responsible for ensuring the documentation of the asset is up to date
- The one who can provide more information on the asset
- Notified about changes and requests on the asset
- The go-to channel for any questions regarding the asset

## Custom Ownership Labels

Some accounts use custom ownership labels so the same owner or team can appear with different roles, for example as a technical owner or a business owner. Admins define those labels under **Account Configuration > Ownership Labels**.

When labels are enabled, each new owner assignment should include the label that fits that relationship. Older assignments don't receive a label automatically. Until you update them in the [Metadata Editor][], Catalog can show **No label** for those owners. To apply labels across many assets at once while keeping the same principal, follow [Assign Custom Ownership Labels in Bulk][].

---

[Metadata Editor]: /docs/catalog/document-your-data/metadata-editor
[Assign Custom Ownership Labels in Bulk]: /docs/catalog/document-your-data/metadata-editor#assign-custom-ownership-labels-in-bulk

---

## Pinned Assets and Mentions

## Pinned Assets

In Catalog you can create references between objects and pages by using the pinned assets section at the bottom of each README section on every asset.

Pinned assets are a great way of linking two assets, for example a business definition like a KPI and data assets related to it, such as a dashboard where the KPI exists or tables from where it is calculated or sourced.

:::tip[Recommendation]

We recommend that every knowledge page should have at least one pinned data asset.

:::

Pinned asset relationships are recursive. The source asset is represented under the "mentioned in" section in the pinned assets area.

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

Good use of pinned assets can complete a discovery journey. For example, when a user looks for a KPI, they would typically also like to see the dashboards where the KPI is used, or, if you are an analyst, the tables where you can find the KPI for further analysis.

<figure><figcaption>Above is a diagram with examples on the types of relationships that pinned assets can help represent.</figcaption></figure>

## In-Page Mentions

One small tip is to use Catalog URLs when you refer to another asset in a README page. The link will be prettified automatically, as in the example below.

<figure><figcaption></figcaption></figure>

---

## Tags

Tags are a common yet powerful way to identify and group your assets together.

You can create different types of tags (by using the semicolon symbol, for example `tag_type:tag_name`):

- **Domain (and subdomain) tags** to identify the assets used by a given domain, or what domains are making use of a given asset
- **Data classification tags** to highlight where PII or confidential information is stored
- **Deployment stage tags** to highlight things that are in development, production, or UAT

<figure><figcaption></figcaption></figure>

## How To Use Tags

- To create a tag, click the ⊕ sign next to tags, type your tag, and select the create a new tag button.
- Alternatively, admins and data stewards can create and manage tags directly in the Tag Manager tab in the Governance section.

## Tips

- For the purpose of grouping and identifying knowledge or documentation, we recommend creating domain tags (see [Knowledge pages][]): a tag for the first level in your domain hierarchy.
- Use the domain tags to mark those assets that are being used by a given domain. An asset can have, and most likely will have, more than one domain tag.
- Similarly you can create team tags or data product tags. Try not to have more than 30 tags per tag type, as that will make it difficult for users to remember the values they would need to use.

For more on tagging strategy, faceted tags, and tag types, see [Tagging and Classification][].

[Knowledge pages]: https://docs.coalesce.io/docs/catalog/use-cases/document/documentable-assets/knowledge-pages
[Tagging and Classification]: https://docs.coalesce.io/docs/catalog/document-your-data/tags

---

## Content and Context

There are several ways to generate knowledge and enrich the context of an asset in Catalog.

Use this table to pick the area that matches the work you want to do next:

| Method | Best for |
| :----- | :------- |
| [Descriptions][] | Writing, syncing, or generating text that explains tables, columns, and dashboards. |
| [Tags][] | Labeling assets so domains, sensitivity, or lifecycle show up consistently in search and browse. |
| [Pinned Assets][] | Linking related assets and mentions so readers see the right context on each record. |
| [Ownership][] | Defining who keeps each asset documented and accountable over time. |

[Descriptions]: /docs/catalog/use-cases/document/content-and-context/descriptions
[Tags]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/tags
[Pinned Assets]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions
[Ownership]: /docs/catalog/use-cases/document/content-and-context/governance-concepts/ownership

---

## Dashboard Assets

## What Is a Dashboard Asset?

Under the [Dashboard section][] you will find assets synchronized from your Business Intelligence and Analytics platforms. Here is where you will be able to find semantic models or data sets, as well as dashboards and reports.

<figure><figcaption></figcaption></figure>

## Dashboard and Reports Documentation

The descriptions for dashboards and reports are automatically synchronized from the BI or Analytics source.

You can also add extra documentation in the README section for each dashboard and report. When doing that, the best practice is to think about the questions you would like your documentation to answer. Some examples of such questions can be found below:

- **Why do we have this dashboard or report?** Describe why the dashboard exists or its main use cases
- **Who should use it?** Target audience
- **What does it contain? What does it measure?** Describe what the dashboard or report contains: Key KPIs, Measures, and Attributes. Consider linking to a Knowledge page and pinning it (see [Pinned assets and mentions][]) if the measure, KPI, or attribute will be reused in other dashboards
- **How and when?** Describe the source of the data (for example, CRM, payment system, etc.) and how often it will be refreshed in the dashboard
- **Anything particular we should know?** Any particular calculations happening in the dashboard

[Dashboard section]: https://app.castordoc.com/terms
[Pinned assets and mentions]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions

---

## Data Assets

## What Is a Data Asset?

In the left panel of your Catalog instance, under the [Data section][], you will find tables and views sourced directly from data warehouses and grouped under databases and schemas.

<figure><figcaption></figcaption></figure>

## Tables and Columns Documentation

Documentation can be added at both the table level, in the README section, and at the columns level, on the right-hand side of each column. Documentation can also be imported directly from the data source if it exists.

Below are some examples of questions that your **table README** documentation could or should answer:

- **Why do we have the table? What is its purpose?** Describe why the table exists or main use cases: for example, part of the ETL process that calculates x or main transactions table used for basket revenue analysis
- **Who should use it?** Target audience
- **What does it contain?** Describe what the table contains: Key measures and attributes. History: years of data. Granularity: daily or transaction level, etc. Key sources (if not automated and present in tags)
- **How and when is it refreshed?** Describe how the data is captured and how frequently it is refreshed
- **Anything special we should know?** Mention particular calculations, joins, etc. If this is an asset used in the presentation layer for analysis, consider creating separate Knowledge entries for metrics or important attributes
- **Technical observations:** Specific SQL examples, etc.

Similarly, for **column documentation**, ensure you add a short description of the meaning of the column, enumerate values if it is an enum type, and consider creating a knowledge page and pinning it (see [Pinned assets and mentions][]) if it represents a major metric or business concept.

[Data section]: https://app.castordoc.com/data
[Pinned assets and mentions]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions

---

## Documentable Assets

Documentable assets are the Catalog records you can open, describe, and connect to governance workflows. They include synced BI content, warehouse tables and columns, and knowledge pages you create in Catalog.

Use this table to pick the guide that matches the kind of asset you need to document:

| Method | Best for |
| :----- | :------- |
| [Dashboard Assets][] | BI dashboards, reports, and charts from Tableau, Looker, Power BI, and similar tools. |
| [Data Assets][] | Tables, views, and columns from data warehouses. |
| [Knowledge Pages][] | Free-form documentation pages you create in Catalog. |

[Dashboard Assets]: /docs/catalog/use-cases/document/documentable-assets/dashboard-assets
[Data Assets]: /docs/catalog/use-cases/document/documentable-assets/data-assets
[Knowledge Pages]: /docs/catalog/use-cases/document/documentable-assets/knowledge-pages

---

## Knowledge Pages

## What Is the Knowledge Section?

You can think of the Knowledge section as the place where you link business concepts and definitions to data assets.

You can create and group your documentation in different sections with pages related to your:

- Organization domains
- Business processes
- Metrics and KPIs
- Glossary terms
- Business concepts
- Policies
- FAQs

Below are some examples of the key categories of knowledge. The list is not exhaustive and can differ from organization to organization.

<figure><figcaption></figcaption></figure>

## Domains Documentation and Structure

A data domain is a broad grouping of data assets based on its function, usage, or subject matter. Each domain is typically associated with specific governance policies, ownership, and quality standards.

Typical examples of domains:

- Customer: including data assets and metrics related to customer lifecycle, customer health, customer attributes, etc.
- Finance: that includes data assets related to financial transactions, budgets, revenue, or expenses
- Employee: including job information, talent retention, hiring, etc.

:::tip[Domain Tips]

- You can have multiple teams contributing or using information from a single domain.
- Typically, there are no more than 10 data domains as part of an organization.
- If your organization is not used to the concept of domain, you can start by simply using your department hierarchy instead.

:::

### How To Structure Your Domains

A basic structure in the domain structure can follow the domain list and the key areas for each domain. For example:

- Typically an organization will have activities split by lifecycle and you can split it based on its stage: prospect, customer, etc.
  - For prospects you will have qualification stages and sales stages, each with its own processes and KPIs
  - For existing customers you will have onboarding processes, but also data assets related to customer deals

:::tip[Start Simple]

It is a good idea to start with just one or two levels in the domain hierarchy before expanding further.

:::

<figure><figcaption></figcaption></figure>

### How To Document Your Domains

Some typical questions your domain or team pages should answer:

- **Why?** What the domain, team, or data product does or its mission
- **Who?** Key people working on it
- **How?** Key processes used to achieve the mission. These will be child pages below
- **What?** Key assets or metrics used to achieve the mission
- **Anything else you should know?** Useful presentations or links

For more complex organizations you might want to consider using domains and teams at the same time. In that case you should have separate hierarchies, using tags to intersect.

## Metrics Documentation

A _metric_ or _Key Performance Indicator (KPI)_ is a quantifiable measurement used to evaluate the performance or success of an organization, process, team, or individual.

:::tip[Metrics Organization]

Putting metrics under a domain or a team bears the risk of adding multiple definitions for the same business concept. For that reason we recommend having a separate section for metrics that is either flat or is only grouped by the type of metric (and not by domain, team, etc.).

For example you can group MAU, WAU, Number of events, etc. under Adoption. But we do not recommend grouping them under Product, for example, as other departments or teams might play a part in influencing them.

:::

When documenting a metric you should consider what questions you want the documentation to answer. Some question examples can be found below:

- **Why was the metric created?** Describe the purpose of the metric
- **Who should use it?** Target audience
- **What does it measure?** Describe where it is commonly used
- **How and when?** How is it calculated, when and how often is it refreshed
- **Anything special we should know?** For example, always uses a specific filter

Metrics and KPIs will be present in one or more dashboards, data sets or semantic models, and fields, and can be sourced from one or two tables and columns. We recommend using [Pinned Assets][] in order to highlight these relationships.

:::tip[Best Practices]

- Each metric should have at least one owner and corresponding domain or team tags.
- Each metric should have at least one pinned asset to either a dashboard or source table used in the calculation for the metric.

:::

## Glossary Documentation

Under glossary (or business concepts) you can add any of those business terms that do not fit under the metric umbrella.

Some examples could be:

- Acronyms: for example, GDPR, QBR, etc.
- Business groupings: for example, territory, EMEA, APAC, etc.
- Business terms: for example, active employee, commercial or enterprise account, customer, etc.

The documentation of these terms should include a business definition, a technical one, and possibly the source of the data.

## Other Categories of Documentation

### Data Products

- You can group tables, models, and dashboards into data products
  - Use the knowledge pages to document the data product (purpose, frequency, sources, etc.)
  - Use the asset documentation sections to document the individual components

### Policies

- Add knowledge pages for your data and governance policies

### FAQ

- A place to add your training and common questions received from your users

[Pinned Assets]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions

---

## How To Approach a Documentation Initiative

High-quality documentation is the foundation of effective data discovery. By enriching your assets with clear, consistent, and searchable metadata, teams ensure that data can be easily understood, trusted, and reused. The better documented your assets are, the more autonomous your teams become.

However, launching a documentation initiative can be daunting as the scope can be intimidating. This section is for you.

- You're part of your company governance team and have to launch a documentation initiative but don't really know how to go about it in Catalog
- Or you've been appointed owner of several assets in Catalog and don't know where to start or how to document them

We recommend following these five initial steps to start documenting in Catalog.

Use this table to open the step that matches where you are in the rollout:

| Method | Best for |
| :----- | :------- |
| [Step 1: Create Your Domains][] | Defining domains, generating domain tags, and assigning domain owners before you expand scope. |
| [Step 2: Tag Your Assets][] | Applying domain tags across tables, views, dashboards, reports, and knowledge pages so each domain has a clear footprint. |
| [Step 3: Create Key Metrics and Glossary Terms][] | Capturing KPIs and business glossary entries that anchor how each domain talks about its data. |
| [Step 4: Assign Ownership][] | Giving tables, dashboards, and knowledge pages accountable owners after the first documentation pass. |
| [Step 5: Set Targets and Execute][] | Breaking remaining documentation into sprints, using templates, and tracking progress with analytics. |

<figure><figcaption></figcaption></figure>

[Step 1: Create Your Domains]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-1-create-your-domains
[Step 2: Tag Your Assets]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-2-tag-your-assets
[Step 3: Create Key Metrics and Glossary Terms]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-3-create-key-metrics-and-glossary-terms
[Step 4: Assign Ownership]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-4-assign-ownership
[Step 5: Set Targets and Execute]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-5-set-targets-and-execute

---

## Step 1: Create Your Domains

Create your domain structure as the first step in your documentation initiative.

* [x] First things first, create your domain structure in the [Knowledge section][]
* [x] For each domain, generate a [tag][]
* [x] Don't forget to assign the domain [owner][]

:::danger[Start Small]

Don't boil the ocean. Start with one domain that can be used as an example.

:::

## What Is a Data Domain?

A data domain is a broad grouping of data assets based on its function, usage, or subject matter. Each domain is typically associated with specific governance policies, ownership, and quality standards.

Typical examples of domains:

* Customer: including data assets and metrics related to customer lifecycle, customer health, customer attributes, etc.
* Finance: that includes data assets related to financial transactions, budgets, revenue, or expenses
* Employee: including job information, talent retention, hiring, etc.
* etc.

<figure><figcaption></figcaption></figure>

:::info[Multiple Teams]

You can have multiple teams contributing or using information from a single domain.

:::

:::info[Domain Count]

Typically, there are no more than 10 data domains as part of an organization. If your organization is not used to the concept of domain, you can start by simply using your department hierarchy instead.

:::

:::info[Start With Few Levels]

It is a good idea to start just with 1 or 2 levels in the domain hierarchy before expanding further.

:::

## How To Structure Your Domains

A basic structure in the domain can follow the domain list and the key areas for each domain. For example:

* Typically an organization will have activities split by lifecycle and you can split it based on its stage: prospect, customer, etc.
  * For prospects we will have qualification stages and sales stages each with its own processes and KPIs
  * For existing customers we will have onboarding processes, but also data assets related to customer deals

<figure><figcaption></figcaption></figure>

[Knowledge section]: /catalog/use-cases/document/documentable-assets/knowledge-pages.md
[tag]: /catalog/use-cases/document/content-and-context/governance-concepts/tags.md
[owner]: /catalog/use-cases/document/content-and-context/governance-concepts/ownership.md

---

## Step 2: Tag Your Assets

Use the domain tags defined in the previous step to generate the scope of the domain by tagging all the relevant assets used in that respective domain. That includes tables and views, dashboards, and reports, as well as other related knowledge pages.

<figure><figcaption></figcaption></figure>

:::info[Scope Building]

This step will also build the scope for documentation or knowledge generation.

:::

---

## Step 3: Create Key Metrics and Glossary Terms

- [x] Create the key metrics used in each domain or team in scope
- [x] Create a glossary with the key business concepts used in each domain or team in scope

<figure><figcaption></figcaption></figure>

:::info[Scope Building]

This step will also build the scope for documentation or knowledge generation.

:::

## Metrics Documentation

A _metric_ or _Key Performance Indicator (KPI)_ is a quantifiable measurement used to evaluate the performance or success of an organization, process, team, or individual.

:::tip[Recommendation]

Putting metrics under a domain or a team bears the risk of adding multiple definitions for the same business concept. For that reason we recommend having a separate section for metrics that is either flat or is only grouped by the type of metric (and not by domain, team, etc.).

For example you can group MAU, WAU, Number of events, etc. under Adoption. But we do not recommend grouping them under Product, for example, as other departments or teams might play a part in influencing them.

:::

When documenting you should consider what questions you want the documentation to answer. Some question examples can be found below:

- **Why was the metric created?** Describe the purpose of the metric
- **Who should use it?** Target audience
- **What does it measure?** Describe where it is commonly used
- **How and when?** How is it calculated, when and how often is it refreshed
- **Anything special we should know?** For example, always uses a specific filter

Metrics and KPIs will be present in one or more dashboards, data sets or semantic models, and fields, and can be sourced from one or two tables and columns. We recommend using [Pinned Assets][] in order to highlight these relationships.

:::info[Best Practices]

- Each metric should have at least one owner and corresponding domain or team tags.
- Each metric should have at least one pinned asset to either a dashboard or source table used in the calculation for the metric.

:::

## Glossary Documentation

Under glossary (or business concepts) you can add any of those business terms that do not fit under the metric umbrella.

Some examples could be:

- Acronyms: for example, GDPR, QBR, etc.
- Business groupings: for example, territory, EMEA, APAC, etc.
- Business terms: for example, active employee, commercial or enterprise account, customer, etc.

The documentation of these terms should include a business definition, a technical one, and possibly the source of the data.

[Pinned Assets]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/governance-concepts/pinned-assets-and-mentions

---

## Step 4: Assign Ownership

You've already assigned an owner to each domain (see Step 1). Now that you've created your initial scope of documentation, it is time to assign owners to the other assets (tables, dashboards, other knowledge pages).

[Ownership][] is an important data governance concept in Catalog. Each asset should have an owner responsible for its documentation and maintenance. This could be an individual or a team.

<figure><figcaption></figcaption></figure>

## Owner Not Known?

Sometimes we don't know who should own a given asset. Catalog provides information that could help determine the owner, for example:

- Frequent Users (the users with most consumption queries on the asset)
- Technical Owners (sourced from external sources, typically editors of the asset)
- Creators (the users that created the asset)

The owners will be responsible for ensuring that the assets they own are documented to the required standard. Owners will be notified about changes and requests.

## What Does It Mean To Be Owner of an Asset?

An owner is:

- Responsible for ensuring the documentation of the asset is up to date
- The one who can provide more information on the asset
- Notified about changes and requests on the asset
- The go-to channel for any questions regarding the asset

[Ownership]: https://docs.coalesce.io/docs/catalog/use-cases/document/content-and-context/governance-concepts/ownership

---

## Step 5: Set Targets and Execute

Now that you've completed steps 1 through 4, if you look back at what has been added to the knowledge and the documentation scope, it can look overwhelming.

We recommend splitting the work into smaller chunks, prioritizing, and executing on documenting in one- to two-week sprints.

:::info[Make It Fun]

Have you thought about having documentation hours?

:::

## Tips for Better Documentation in Catalog

- Create and use [templates][] to establish standardized formats for documentation, ensuring consistency, quality, and clarity across all materials
- Keep generating new knowledge pages as you identify new metrics and business concepts
- Don't forget to pin relevant assets to illustrate the relationships between your assets
- Use the documentation coverage [analytics dashboards][] to monitor your progress, identify your data champions, and take action when things are not progressing as expected

<figure><figcaption></figcaption></figure>

<figure><figcaption></figcaption></figure>

[templates]: https://docs.coalesce.io/docs/catalog/document-your-data/templates
[analytics dashboards]: https://docs.coalesce.io/docs/catalog/administration/analytics

---

## Documenting in Catalog

High-quality documentation is key to building trust, clarity, and usability across your data ecosystem. Catalog is your central hub for capturing and sharing knowledge, making it easy for data consumers and collaborators to understand what your data represents, how it is structured, and how to use it.

Whether you are onboarding new team members, managing compliance, or supporting self-service analytics, documentation transforms your catalog into a powerful source of context.

Catalog brings together various elements from across your data and business landscape, including:

- [Data assets][] such as tables and views
- [BI and Analytics assets][] such as dashboards and reports
- [Business concepts][] including KPIs, metrics, domains, and terms

These assets can be linked and organized using governance tools like tags, ownership, and pinned assets, making it easier to maintain structure, consistency, and discoverability at scale.

<figure><figcaption></figcaption></figure>

[Data assets]: https://docs.coalesce.io/docs/catalog/use-cases/document/documentable-assets/data-assets
[BI and Analytics assets]: https://docs.coalesce.io/docs/catalog/use-cases/document/documentable-assets/dashboard-assets
[Business concepts]: https://docs.coalesce.io/docs/catalog/use-cases/document/documentable-assets/knowledge-pages

---

## 5-Minute Quick Start Guide

Welcome to **Coalesce Catalog**. We’re thrilled to have you on board. In just a few minutes, you’ll be able to explore your data landscape with clarity and confidence.

## Why Coalesce?

Coalesce Catalog is like your data’s GPS, helping you find what you need, understand where it comes from, and trust where it's going. It’s smart, searchable, and built to make your whole team feel like data pros.

## Logging In

To get started:

* Head to [https://app.castordoc.com/](https://app.castordoc.com/)
* Sign in using your SSO or directly using your app credentials.
* If you're having trouble, contact your support team.

## Explore the Interface

Here's a quick visual walkthrough of the core areas:

* **Step 1: Left Panel**: Browse your data ecosystem by Warehouse, BI tool, Business Knowledge, and People in your organization.

<figure><figcaption></figcaption></figure>

* **Step 2: Search Bar**: Use the Advanced Search and its filters to find assets fast.

  * Type something familiar, like `revenue`, `orders`, or `customer`
  * Use filters (type, tags, etc.) to narrow results if needed

  <figure><figcaption></figcaption></figure>

* **Step 3: Asset Zoom**: Click into a table or dashboard and discover its metadata such as lineage, descriptions, and popularity.

<figure><figcaption></figcaption></figure>

* **Step 4: AI Assistant**: You can also use **natural language** and ask your questions directly to the AI assistant, such as: “Do we have a dashboard about churn?”

<figure><figcaption></figcaption></figure>

## Your Toolkit

Here are resources to help you:

* 📖 [Catalog Documentation](/docs/catalog/)
* 🎓 Training Videos (coming soon)
* 🔌 [Catalog Public API](https://apidocs.castordoc.com/)

## Need Help?

If you need assistance:

* **Support Email**: [support@coalesce.io](mailto:support@coalesce.io)
* **In-app support:**  In-app chat on your Catalog instance

---

## Asset Management: Query and Cost Optimization

Catalog provides you with a number of useful tools to support better maintenance and cost optimization of your data environments.

Reports on query utilization and individual queries are starting points that help identify those tables and queries that have a high number of hits or take a long time, or return a high number of records.

## Reports

The User Activities and Table Analytics reports can be found under the Reports tab of your Governance section and offer information on queries by user and tables. This information can be used to identify improvement opportunities, for example, a high query run time against a table or tables with many hits.

The Unused Asset report is provided on demand by Catalog. This will highlight those objects that are not being used that could be removed from the data estate. These reports take into consideration child dependencies when an object is being flagged as unused.

<figure><figcaption></figcaption></figure>

## Queries

Once you have identified a potential opportunity to optimize an object, you can navigate to that object.

In the data view you can navigate to Queries to visualize all the read statements and investigate if the queries could be optimized further.

<figure><figcaption></figcaption></figure>

---

## Dependency Management: Impact Assessment and Migrations

- You want to understand the downstream impact of a change you are planning to make
- There is a data issue with the numbers you see on a dashboard and you are trying to understand the source of it
- You identified a bug in one of the tables and you would like to identify which reports and users are affected

All of these questions and more can be answered by Catalog's full data warehouse (DWH) lineage, for your dashboards, tables, and fields or columns.

## Lineage

By navigating to the Lineage tab you can quickly identify the sources of a dashboard, table, or fields or columns and its immediate parents and children.

You can also navigate further using the **lineage graph** or **export a downstream node report** to get all the given asset dependencies.

<figure><figcaption>Lineage tab view</figcaption></figure>

<figure><figcaption>Lineage graph view</figcaption></figure>

**Column Lineage** also provides end-to-end dependency information between fields and columns. It can help you answer questions like:

- What is the source of a field?
- Where is this field generated?
- What is the impact if you change a column?

<figure><figcaption>Field lineage graph view</figcaption></figure>

**Extract all downstream nodes** is an extremely helpful report that provides info about all downstream children of an asset or a column:

- Node (where to find)
- Source (database technology, BI tool)
- Depth (how far from the parent)
- Type (table, view, dashboard, etc.)
- Parent (immediate parent)
- Popularity (calculated based on consumption queries)
- Owners (useful if they have to be notified)
- Frequent Users (as above)

<figure><figcaption></figcaption></figure>

Catalog can provide **other reports** that might be useful for specific use cases (for example, migrations, environment mergers). Please contact your Catalog team for more details.

Find out more about lineage in Catalog in the [Lineage section][].

[Lineage section]: https://docs.coalesce.io/docs/catalog/navigate/lineage

---

## Using Catalog To Migrate Data Solutions

Migrating data solutions is a strategic but complex initiative that requires visibility, coordination, and clear documentation. Coalesce Catalog supports this process at every stage, from scoping and planning to execution and launch. By surfacing dependencies, identifying unused assets, and tracking ownership, the Catalog helps teams streamline migrations, reduce risk, and maintain data trust.

This section outlines how to leverage the Catalog effectively to support a smooth and informed migration process.

## 1. Scope, Ownership, and Plan

Use [lineage and downstream reports][] to understand dependencies, workloads, and plan sprints. Contact your Catalog team before launching this exercise as some of these reports are only available on demand.

- Identify Unused Assets and remove them from the scope of the project
- Prioritize Data Products (Dashboards, Tables, Schemas) based on popularity
- Assign ownership using the bulk editor, most frequent users, or creator

## 2. Develop

- Migrate while using lineage for change progress and dependency management (for example, lineage computing, testing, debugging)
- Data Quality or Testing
- Document while executing. Use Castor automated features to propagate definitions

## 3. Launch

- Use Catalog to complement the launch of a data product to provide all the necessary documentation
- Review popularity and documentation for most popular assets
- Review and assign owners
- Group assets into Data Products or Domains by using the Knowledge section
- Promote collaboration to improve data literacy

[lineage and downstream reports]: https://docs.coalesce.io/docs/catalog/use-cases/manage-dependencies-impact-assessment-and-migrations

---

## PII and Sensitive Data

The Catalog can help you identify your PII. How does it work? The Catalog can highlight columns that are likely to include PII based on the metadata received.

Start your journey by requesting a PII report from your Catalog team. This report provides the likelihood that a column includes PII. Once validated, you can tag columns as PII and report on them using advanced search or Governance reports.

## Step 1: Start With the PII Likelihood Report

Ask your Catalog team to provide the PII likelihood report. This report analyzes all your metadata and provides the likelihood that a column might include PII based on the naming convention and a few other factors.

## Step 2: Tag PII

Once confirmed, tag your PII columns by selecting the PII tag against it.

If a table includes PII, the PII tag will appear on the top right corner.

<figure><figcaption></figcaption></figure>

## Step 3: Find PII

You can find your PII assets in advanced search by selecting the has PII checkbox.

<figure><figcaption></figcaption></figure>

You can also report all PII fields by downloading the PII columns report.

<figure><figcaption></figcaption></figure>

---

## Command Line Interface

Coalesce ships a command line interface, `coa`, for automated deploys, refreshes, and CI/CD. The CLI docs are split by major version because behavior and setup differ in important ways.

## Coalesce CLI (COA 7.33 and Above)

Use this section when you run `coa` 7.33 or newer on Snowflake. These docs describe local development (work from your version control checkout, connect to the warehouse, iterate without routing SQL through Coalesce cloud) and cloud operations (plan, deploy, and refresh through Coalesce using access tokens and Environment IDs).

With the newer CLI, you can author and run SQL locally: edit Node SQL, preview generated DDL, create tables, run DML, and validate results in your own SQL client while you build. When the pipeline is ready, you promote it through the same `coa` commands for cloud-backed deploy and refresh. Start with [Command Line Interface][] for install, authentication, and configuration, then follow the task docs below.

- [Command Line Interface][] - Install, auth, `workspaces.yml`, and `~/.coa/config`
- [Build Pipelines With the Coalesce CLI][] - Local build loop and pipeline authoring
- [Deploy Pipelines to the Coalesce App Using COA][] - Cloud deploy workflow
- [CLI Commands][] - Full command reference
- [Troubleshoot the Coalesce CLI][] - Common errors and fixes
- [CLI Support Policy & Usage Recommendations][] - Support scope and usage guidance

## Coalesce CLI (COA 7.32 and Below)

Use this section when you're still on `coa` 7.32 or older. The model emphasizes connecting through the earlier configuration paths documented for that release. If you are unsure which path you use, check `coa --version` and talk to your admin about upgrade timing.

- [Legacy CLI overview][] - How the earlier CLI fits CI/CD and on-prem workflows
- [CLI setup (7.32 and below)][] - Install, profiles, tokens, and Snowflake auth
- [CLI commands (7.32 and below)][] - Command reference for older releases
- [CLI support policy (7.32 and below)][] - Support and usage notes for legacy versions

## What's Next?

- Read [Command Line Interface][] if you're on 7.33 or newer and need end-to-end setup
- Open [CLI setup (7.32 and below)][] if you must stay on 7.32 or below for now
- See [Deploy Using the CLI][] for how the CLI fits deploy and refresh in the product

---

[Command Line Interface]: /docs/coa/version-733-and-above
[Build Pipelines With the Coalesce CLI]: /docs/coa/version-733-and-above/coa-building-pipelines
[Deploy Pipelines to the Coalesce App Using COA]: /docs/coa/version-733-and-above/coa-deploying-pipelines
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Troubleshoot the Coalesce CLI]: /docs/coa/version-733-and-above/coa-troubleshooting
[CLI Support Policy & Usage Recommendations]: /docs/coa/version-733-and-above/coa-support-policy-usage-recommendations
[Legacy CLI overview]: /docs/coa/version-732-and-below
[CLI setup (7.32 and below)]: /docs/coa/version-732-and-below/coa-setup
[CLI commands (7.32 and below)]: /docs/coa/version-732-and-below/coa-commands
[CLI support policy (7.32 and below)]: /docs/coa/version-732-and-below/coa-support-policy-usage-recommendations
[Deploy Using the CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli

---

## CLI Commands

<!-- markdownlint-disable MD036 -->
<!-- markdownlint-disable MD033 -->

<!-- vale Coalesce.Headings = NO -->

## $ `coa` [options] [command]

```bash
Usage: coa [options] [command]

Options:
  -v, --version                               output the version number
  -b, --debug                                 Output extra debugging (default: false)
  --verbose                                   Log more verbosely (default: false)
  --config <coa-config-location>              coa config file location
  --changeLogLevel <loggingArea>:<logLevel>   INTERNAL:Change logging level of logging area (default:
                                              false)
  -h, --help                                  display help for command

Commands:
  deploy [options]                            run a coalesce deployment plan
  plan [options]                              plan a Coalesce deployment
  refresh [options]                           runs a Coalesce pipeline
  rerun [options] <runID>                     re-runs a failed Coalesce pipeline. Runtime parameters from
                                              the previous run will not be preserved.
  environments                                Environments
  nodes                                       Nodes
  jobs                                        Jobs
  gitAccounts                                 Git account management
  workspace-nodes                             Nodes
  projects                                    Projects
  runs                                        Runs
  cancel [options] <runID>                    Cancels a running pipeline.
  help [command]                              display help for command
```

## $ `coa` plan [options]

Generate a `coa-plan.json` file that can be deployed.

Run `coa plan` in the same directory as the locally cloned copy of your Coalesce project's Git repository.

:::info[Caching Deployment Plan]

If you have a large project, using `--enableCache` could [speed up your deployments][].

:::

**Flags**

`coa plan -h`

```bash
plan a Coalesce deployment

Options:
  --domain <value>                  Coalesce domain name to use for API requests
                                    (default: "https://app.coalescesoftware.io")
  --environmentID <value>           Environment ID
  --parameters <value>              Parameters
  --platformKind <value>            Platform Kind (Snowflake, Databricks) (default:
                                    "Snowflake")
  --profile <value>                 Profile To Use
  --token <value>                   Coalesce Refresh Token
  --databricksAuthType <value>      Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>      Databricks Client ID
  --databricksClientSecret <value>  Databricks Client Secret
  --databricksPath <value>          Databricks Path
  --databricksToken <value>         Databricks Token
  --snowflakeAuthType <value>       Snowflake Auth Type (Basic, KeyPair)
  --snowflakeKeyPairKey <value>     Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>    Snowflake Key Pair Pass
  --snowflakePassword <value>       Snowflake Password
  --snowflakeRole <value>           Snowflake Role
  --snowflakeUsername <value>       Snowflake Username
  --snowflakeWarehouse <value>      Snowflake Warehouse
  --snowflakeKeyPairPath <value>    Snowflake Key Pair Path (deprecated, please use
                                    --snowflakeKeyPairKey)
  -d, --dir <dir>                   Coalesce pipeline Yaml file path (default: ".")
  --out <plan-location>             Coalesce plan location (default: "./coa-plan.json")
  --gitsha <sha>                    Sets the git commit SHA shown in the plan manifest
                                    and UI. This does not affect which files are
                                    used—those come from the current directory or are
                                    set with the --dir parameter.
  --enableCache                     Enable caching to improve deploy plan generation
                                    times. Coalesce recommends keeping caching disabled
                                    unless you’re experiencing very long plan generation
                                    times. (default: false)
  -h, --help                        display help for command
```

## $ `coa` deploy [options]

Run an existing `coa-plan.json`. You must create a plan, before you can deploy.

Deploy is used to make structural changes to the data warehouse (typically DDL commands) and save the corresponding data warehouse configuration for the environment for subsequent refreshes. See [Deployment and Refresh][] for information on deployment.

**Deploy results**

The `coa` will return information about your run, including the following:

- `runType` - Returns the type of run. `deploy` or `refresh`.
- `runStatus` - The status of the run.
  - `canceled`
  - `completed`
  - `failed`
  - `running`
  - `waitingToRun`
- `runResults` - Includes the SQL executed, any error details, and other environment information.

**Flags**

`coa deploy -h`

```bash

run a coalesce deployment plan

Options:
  --domain <value>                  Coalesce domain name to use for API requests (default:
                                    "https://app.coalescesoftware.io")
  --environmentID <value>           Environment ID
  --platformKind <value>            Platform Kind (Snowflake, Databricks) (default: "Snowflake")
  --profile <value>                 Profile To Use
  --token <value>                   Coalesce Refresh Token
  --databricksAuthType <value>      Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>      Databricks Client ID
  --databricksClientSecret <value>  Databricks Client Secret
  --databricksPath <value>          Databricks Path
  --databricksToken <value>         Databricks Token
  --snowflakeAuthType <value>       Snowflake Auth Type (Basic, KeyPair)
  --snowflakeKeyPairKey <value>     Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>    Snowflake Key Pair Pass
  --snowflakePassword <value>       Snowflake Password
  --snowflakeRole <value>           Snowflake Role
  --snowflakeUsername <value>       Snowflake Username
  --snowflakeWarehouse <value>      Snowflake Warehouse
  --snowflakeKeyPairPath <value>    Snowflake Key Pair Path (deprecated, please use --snowflakeKeyPairKey)
  -p, --plan <plan-location>        Coalesce plan location (default: "./coa-plan.json")
  -o, --out <output json>           Run Results Output in Json
  -h, --help                        display help for command

```

## $ `coa` refresh [options]

Refresh is used to run data transformations, typically DML commands (Merge, Insert, Truncate, etc). See [Deployment and Refresh][] for information on refresh. That deployment must happen on a environment before refresh is possible.

To re-run a failed refresh use:  
`coa rerun <runID>`

This only works on failed refreshes, and only the nodes that previously failed will be run.

**Refresh results**

`coa` will return information about your run, including the following:

- `runType` - Returns the type of run. `deploy` or `refresh`.
- `runStatus` - The status of the run.
  - `canceled`
  - `completed`
  - `failed`
  - `running`
  - `waitingToRun`
- `runResults` - Includes the SQL executed, any error details, and other environment information.

**Flags**

`coa refresh -h`

```bash
runs a Coalesce pipeline

Options:
  --domain <value>                  Coalesce domain name to use for API requests (default:
                                    "https://app.coalescesoftware.io")
  --environmentID <value>           Environment ID
  --exclude <value>                 Coalesce Node Selector
  --include <value>                 Coalesce Node Selector
  --jobID <value>                   Coalesce JobID
  --parallelism <value>             Parallelism level to use
  --parameters <value>              Parameters
  --platformKind <value>            Platform Kind (Snowflake, Databricks) (default: "Snowflake")
  --profile <value>                 Profile To Use
  --token <value>                   Coalesce Refresh Token
  --databricksAuthType <value>      Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>      Databricks Client ID
  --databricksClientSecret <value>  Databricks Client Secret
  --databricksPath <value>          Databricks Path
  --databricksToken <value>         Databricks Token
  --snowflakeAuthType <value>       Snowflake Auth Type (Basic, KeyPair)
  --snowflakeKeyPairKey <value>     Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>    Snowflake Key Pair Pass
  --snowflakePassword <value>       Snowflake Password
  --snowflakeRole <value>           Snowflake Role
  --snowflakeUsername <value>       Snowflake Username
  --snowflakeWarehouse <value>      Snowflake Warehouse
  --snowflakeKeyPairPath <value>    Snowflake Key Pair Path (deprecated, please use --snowflakeKeyPairKey)
  -o, --out <output json>           Run Results Output in Json
  --forceIgnoreEnvironmentStatus    Ignore the environment status and refresh even if there is a failed
                                    deploy. This may cause refresh failures!
  -h, --help                        display help for command

```

## $ `coa` rerun [options] `<runID>`

Rerun a failed pipeline. This only works on failed refreshes, and only the nodes that previously failed will be run.

`coa` will return information about your run, including the following:

- `runType` - Returns the type of run. `deploy` or `refresh`.
- `runStatus` - The status of the run.
  - `canceled`
  - `completed`
  - `failed`
  - `running`
  - `waitingToRun`
- `runResults` - Includes the SQL executed, any error details, and other environment information.

```bash
re-runs a failed Coalesce pipeline. Runtime parameters from the previous run will not be preserved.

Arguments:
  runID                             The ID of the failed refresh to re-run

Options:
  --domain <value>                  Coalesce domain name to use for API requests (default:
                                    "https://app.coalescesoftware.io")
  --environmentID <value>           Environment ID
  --parameters <value>              Parameters
  --platformKind <value>            Platform Kind (Snowflake, Databricks) (default: "Snowflake")
  --profile <value>                 Profile To Use
  --token <value>                   Coalesce Refresh Token
  --databricksAuthType <value>      Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>      Databricks Client ID
  --databricksClientSecret <value>  Databricks Client Secret
  --databricksPath <value>          Databricks Path
  --databricksToken <value>         Databricks Token
  --snowflakeAuthType <value>       Snowflake Auth Type (Basic, KeyPair)
  --snowflakeKeyPairKey <value>     Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>    Snowflake Key Pair Pass
  --snowflakePassword <value>       Snowflake Password
  --snowflakeRole <value>           Snowflake Role
  --snowflakeUsername <value>       Snowflake Username
  --snowflakeWarehouse <value>      Snowflake Warehouse
  --snowflakeKeyPairPath <value>    Snowflake Key Pair Path (deprecated, please use --snowflakeKeyPairKey)
  -o, --out <output json>           Run Results Output in Json
  --forceIgnoreEnvironmentStatus    Ignore the environment status and refresh even if there is a failed
                                    deploy. This may cause refresh failures!
  -h, --help                        display help for command
```

## $ `coa` environments [options] [command]

See information about your Environments.

**Flags**

`coa environments -h`

```bash
Environments

Options:
  -h, --help      display help for command

Commands:
  list [options]  List Environments
  get [options]   Get Environment
  help [command]  display help for command
```

## $ `coa` nodes [options] [command]

Returns information about the Nodes in the Environment added to your `.config` file.

**Flags**

`coa nodes -h`

```bash
Nodes

Options:
  -h, --help      display help for command

Commands:
  list [options]  List Nodes
  get [options]   Get Node
  help [command]  display help for command
```

## $ `coa` runs [options] [command]

See information about your Runs.

**Flags**

`coa runs -h`

```bash
Runs

Options:
  -h, --help              display help for command

Commands:
  list [options]          List Runs
  get [options]           Get Run
  list-results [options]  List Run Results
  help [command]          display help for command
```

## $ `coa` cancel [options] `<runID>`

Cancel a running pipeline.

**Flags**

`coa cancel -h`

```bash
Arguments:
  runID                    The ID of the run to cancel.

Options:
  --environmentID <value>  Environment ID (default: "")
  -h, --help               display help for command
```

## $ `coa` workspace-nodes [options] [command]

Create, update, and see information about Nodes in a specific workspace. For a list of available fields see:

- [Create Node][]
- [Set Node][]

**Flags**

`coa workspace-nodes -h`

```bash
Options:
  -h, --help        display help for command

Commands:
  list [options]    List Workspace Nodes
  create [options]  Create Node
  get [options]     Get Node
  put [options]     Set Node
  delete [options]  Delete Node
  help [command]    display help for command
```

## Parallelism Limits

The parallelism parameter determines how many nodes are processed simultaneously (in parallel) by the Snowflake compute warehouse. The API allows a maximum limit of 64, while `coa` is uncapped.

## Overwriting Defaults

You can overwrite the default `coa` file information by passing in the flags to the CLI or entering the information in the API request.

## Using Include and Exclude

Include and Exclude use the [Selector][] format.

```shell title="coa refresh example with include flag"
coa refresh --include '{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION } OR { location: SAMPLE name: ORDERS } OR { location: SAMPLE name: PART } OR { location: SAMPLE name: PARTSUPP } OR { location: SAMPLE name: REGION } OR { location: SAMPLE name: SUPPLIER } OR { location: QA name: STG_PARTSUPP } OR { location: PROD name: STG_PARTSUPP }'
```

[Deployment and Refresh]: /docs/deploy-and-refresh/deploy/
[Selector]: /docs/reference/selector
[Set Node]: /docs/api/coalesce/set-workspace-node
[Create Node]: /docs/api/coalesce/create-node
[speed up your deployments]: /docs/deploy-and-refresh/deploy/caching-deployment-plan

---

## CLI Setup

## CLI System Requirements
<!-- vale off -->

<!-- vale on -->

:::tip[Looking For the Config File]

Just need to review the config file? Go to [Profiles][].

:::

## Prerequisites

- Install [Node.js version 20.x][]
- Have a Coalesce project with a pipeline that has been committed. Take a look at the quick start for your [data provider][].
- Clone the repository connected to your Coalesce project. You'll use this when making requests.
  - [GitHub][]
  - [GitLab][]
  - [Bitbucket][]
  - [Azure DevOps][]

## Installation

- To install Coalesce CLI globally, run: `npm install -g @coalescesoftware/coa`.
- If you need to update, run: `npm upgrade -g @coalescesoftware/coa`.

## Authentication Methods

Each platform supports different authentication methods.

- Databricks
  - [Token][]
  - [OAuthM2M][]
- Snowflake
  - [Key Pair Authentication][]
  - [Basic Auth][]

:::info

Snowflake OAuth is not supported for the CLI.

:::

### Snowflake Key Pair Authentication

1. Go through Snowflake’s [key pair authentication][] steps to generate your keys.
2. Make sure to generate a private key and save it for later use in the Coalesce app and the CLI. You'll need the file path.
3. Generate your public key using the private key created for Coalesce.
4. Assign the public key to your Snowflake user.
5. Add your KeyPair to your Environment.
6. Move on to [Get Your Access Token][].

:::warning[Adding Your Private Key]

When entering your private key, make sure it's formatted properly. It must include the full private key including the lines BEGIN ENCRYPTED PRIVATE KEY and END ENCRYPTED PRIVATE KEY. If you are using special characters they must be escaped with a `\`. For example:

Original passphrase: `hnw6a#n`
Escaped passphrase: `hnw6a\#n`

Be sure to escape `#` pound sign and `:` semi-colons.

**For Example:**

- Original passphrase: `hnw6a#n`
- Escaped passphrase: `hnw6a\#n`

```text
-----BEGIN ENCRYPTED PRIVATE KEY-----
...hnw6a\#n....
-----END ENCRYPTED PRIVATE KEY-----
```

:::

### Snowflake Basic Authentication

Basic authentication only requires your Snowflake username and password. This method is not recommended for automation. If you use OAuth to log into Snowflake, you'll need to use Key Pair for the CLI.

1. Follow the instructions in [Snowflake Username and Password][].
2. Save your credentials for the CLI config file.
3. Make sure to add the credentials to the Environment.
4. Move on to [Get Your Access Token][].

### Databricks Token Authentication

Personal Access Tokens (PAT) are used in Databricks to authenticate at the workspace level. They are good for development.

1. Follow the instructions in [Databricks Connection Guide][].
2. Save your credentials for the CLI config file.
3. Make sure to add the credentials to the Environment.
4. Move on to [Get Your Access Token][].

### Databricks OAuth Machine-to-Machine

Machine-to-Machine OAuth is used for automation. You'll need to create a Service Principal in Databricks.

1. Follow the instructions in [Databricks Connection Guide][].
2. Save your credentials for the CLI config file.
3. Make sure to add the credentials to the Environment.
4. Move on to [Get Your Access Token][].

## Get Your Access Token

No matter the authentication method you use, you'll need an access token.

Go to the **Deploy tab** and click **Generate Access Token**.

:::info[SSO Users and Environments]

SSO users will need a new access token for each new Environment created.

:::

## Get Your Environment ID

There are many places to get your [Environment ID][]. A quick method is to look into the Coalesce app.

Go to **Build Settings > Environments**.

## Get Your Domain URL

Your URL is what you use to log in to Coalesce. The following are examples.

- `https://<company-name>.coalescesoftware.io`
- `https://<company-name>.pp.us-east-2.aws.coalescesoftware.io`
- `https://app.eu.coalescesoftware.io`
- `https://<your-private-link-url>`

## Creating Your CLI Config File

Before creating your config file, ensure you have:

- Access token from Coalesce
- Environment ID from Coalesce
- Authentication credentials for your data platform
  - Location of your private key file if using Key Pair
- Domain URL

1. In your operating system's home directory, create a hidden folder `.coa` and an empty file with no extension called `config` inside.

    ```txt
    ~/
    └── .coa/
        └── config
    ```

2. Copy and paste either the Key Pair or Basic information into your `config` file.

<Tabs>
  <TabItem value="keypair" label="Snowflake Key Pair" default>

  ```txt
  [default]
  token=access_token
  domain=your_domain_url
  snowflakeAuthType=KeyPair
  snowflakeKeyPairKey=file_path_to_private_key
  snowflakeKeyPairPass=private key passphrase (if using KeyPair and if applicable. Can leave out if not using passphrase)
  snowflakeRole=snowflake_user_role
  snowflakeUsername=snowflake_user_name
  snowflakeWarehouse=snowflake_warehouse
  environmentID=environment_ID
  ```

  </TabItem>
  <TabItem value="basic" label="Snowflake Basic">

  Not recommended.

```txt
[default]
token=access_token
domain=you_domain_url
snowflakeAuthType=Basic
snowflakeUsername=username
snowflakePassword=password
snowflakeRole=SYSADMIN (or other role)
snowflakeWarehouse=snowflake_warehouse
environmentID=envID
```

  </TabItem>

<TabItem value="databricks-oauth" label="Databricks OAuth M2M">

```txt
[databricksOAuthM2M]
platformKind=Databricks
databricksClientID=<Databricks Client ID>
databricksClientSecret=<Databricks Client Secret>
databricksAuthType=OAuthM2M
databricksPath=<Databricks SQL Warehouse Path>
environmentID=<Coalesce ENV ID>
```

  </TabItem>

<TabItem value="databricks-token" label="Databricks Token">

```txt
[databrickstoken]
platformKind=Databricks
databricksAuthType=Token
databricksToken=<Databricks Token>
environmentID=<Coalesce ENV ID>
```

  </TabItem>

</Tabs>

## Create a Plan

To check your credentials, you'll create a plan.

A plan details the target environment, Nodes, Jobs, and other information needed deploy or refresh. Create your plan in the same repo, branch, and commit.

```bash
coa plan

# if using multiple profiles 
coa plan --profile <profile-name>
```

This will generate a `coa-plan.json`. You can review this file, but we don't recommend editing directly. If you want to make changes, make them using the API or the Coalesce app. Then commit your changes and regenerate your plan.

## HTTP Proxy

The Coalesce CLI will work with proxy environments. Set your proxy variables by using:

```bash
# For Windows (Command Prompt)
set HTTP_PROXY=http://your-proxy-address:port
set HTTPS_PROXY=http://your-proxy-address:port

# For Windows (PowerShell)
$env:HTTP_PROXY = "http://your-proxy-address:port"
$env:HTTPS_PROXY = "http://your-proxy-address:port"

# For macOS/Linux
export HTTP_PROXY=http://your-proxy-address:port
export HTTPS_PROXY=http://your-proxy-address:port

# Auth
export HTTP_PROXY=http://username:password@your-proxy-address:port
export HTTPS_PROXY=http://username:password@your-proxy-address:port
```

You can test your connection by using a tool like curl. For example:

```bash
curl https://app.coalescesoftware.io
curl -x http://your-proxy-address:port https://app.coalescesoftware.io
```

## Profiles

In Coalesce, profiles allow you to run commands against different configurations, making it possible to tailor settings for various environments, such as testing or production. Each CLI command includes the `--profile` option, which enables you to specify a profile for the scenario you're working with. If no profile is specified, the system defaults to using the `[default]` profile. While you must have a `[default]` profile, additional profiles can be named and configured as needed for different scenarios.

`coa plan --profile databricksOAuthM2M`

:::info[Default Profile]

The `[default]` profile is required. It must contain the `token` and `environmentID`.

:::

```txt
[default] <REQUIRED>
token=<Coalesce API Token> REQUIRED
environmentID=<Coalesce ENV ID> REQUIRED
domain=<Coalesce APP URL>

[databrickstoken]
platformKind=Databricks
databricksAuthType=Token
databricksToken=<Databricks Token>
environmentID=<Coalesce ENV ID>

[databricksOAuthM2M]
platformKind=Databricks
databricksClientID=<Databricks Client ID>
databricksClientSecret=<Databricks Client Secret>
databricksAuthType=OAuthM2M
databricksPath=<Databricks SQL Warehouse Path>
environmentID=<Coalesce ENV ID>

[snowflake-key-pair]
platformKind=Snowflake
snowflakeAuthType=KeyPair
snowflakeKeyPairKey=<Snowflake Key Pair Key or File Path>
snowflakeRole=<Snowflake Role>
snowflakeUsername=<Snowflake User Name>
snowflakeWarehouse=<Snowflake Warehouse>
environmentID=<Coalesce ENV ID>

[snowflake-basic]
platformKind=Snowflake
snowflakeAuthType=Basic
snowflakeUsername=<Snowflake Account Username>
snowflakePassword=<Snowflake Account Password>
snowflakeWarehouse=<Snowflake Warehouse>
snowflakeRole=<Snowflake User Role>
environmentID=<Coalesce ENV ID>
```

---

## What's Next?

- Some things you can use the CLI for is [Deploy Using the CLI][] and [Refreshing an Environment][].
- Read the available [CLI Commands][].

---

[Node.js version 20.x]: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
[GitHub]: https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository
[GitLab]: https://docs.gitlab.com/ee/user/project/repository/#clone-a-repository
[Bitbucket]: https://support.atlassian.com/bitbucket-cloud/docs/clone-a-git-repository/
[Azure DevOps]: https://docs.microsoft.com/en-us/azure/devops/repos/git/clone
[Refreshing an Environment]: /docs/deploy-and-refresh/refresh/
[CLI Commands]: /docs/coa/version-732-and-below/coa-commands
[Deploy Using the CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli
[data provider]: /docs/get-started/quick-start
[Environment ID]:/docs/reference/whats-my-id/
[Basic Auth]:/docs/coa/version-732-and-below/coa-setup#snowflake-basic-authentication
[Key Pair Authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[Profiles]:/docs/coa/version-732-and-below/coa-setup#profiles
[Get Your Access Token]: /docs/coa/version-732-and-below/coa-setup#get-your-access-token
[Databricks Connection Guide]: /docs/setup-your-project/connection-guides/databricks
[Snowflake Username and Password]: /docs/setup-your-project/connection-guides/snowflake/snowflake-username-and-password
[Token]: /docs/setup-your-project/connection-guides/databricks
[OAuthM2M]: /docs/setup-your-project/connection-guides/databricks

---

## CLI Support Policy & Usage Recommendations

## Minimum Support Version

[Version 7.24][]

We release new versions of the CLI periodically to introduce new product features, enhancements, and bug fixes. New CLI versions are compatible with existing Coalesce features, but we do not guarantee that older versions of the CLI are compatible with new features.

To avoid issues when connecting to and using Coalesce, we recommend using CLI version 7.17.1 or newer, and actively monitoring and maintaining the versions of your CLI as you use it.

To update the CLI, run:

`npm upgrade -g @coalescesoftware/coa`

[version 7.24]: https://www.npmjs.com/package/@coalescesoftware/coa?activeTab=versions

---

## Command Line Interface(Version-732-and-below)

:::warning[`coa` Version 7.32 and below]

These docs cover COA for versions 7.32 and below.

:::

:::warning[Nodejs 18 End of Life]

Starting with the **7.24 release**, we are updating the supported Node.js versions for the `coa` CLI to ensure you are working in a stable and secure environment.

## What's Changing?

The `coa` CLI will now enforce the use of the following [Node.js](https://nodejs.org/en) versions:

- 20
- 22
- 24

**Version 18 will no longer be supported.**

### Why We're Making This Change

We're removing support for version 18 because it reaches its [end-of-life in April 2025](https://nodejs.org/en/about/previous-releases). Support for version 24, a Long-Term Support (LTS) version, has been added to ensure continued stability and security.

### Action Required

If you are running Node.js version 18, **you must upgrade to a supported version** to continue using the `coa` CLI. After updating to release 7.24, running the CLI with Node.js version 18 will cause the CLI to exit with the following error message:

`Invalid node version 18 not supported`

:::

Coalesce provides a CLI (Command Line Interface, or COA) to enable automated CI/CD workflows for Coalesce projects. COA is intended for users with stringent on-premise data requirements and connects directly to your data platform from your local environment instead of using the Coalesce cloud backend to route SQL commands and data.

The CLI and SaaS app can be used in tandem. For example, users may prefer our SaaS app for development and switch to our CLI for production, or choose to deploy using the GUI and refresh the data warehouse via the CLI.

:::warning[CLI Outbound Connection]

Coalesce CLI uses an outbound connection to Coalesce metadata for tracking environment state and execution results.

:::

## How Does Coalesce CLI Work?

1. The user commits their Coalesce project to a Git repository using the SaaS app.
2. The Git repository is cloned to a local machine on which `coa` is installed.
3. The user executes code locally with `coa` commands. `coa` reads the files from the locally cloned Git commit and deploys the metadata configuration to an environment.
4. The deploy saves the metadata configuration to the environment, so that future refreshes use the current state of the data warehouse stored in the metadata.

The team will run the refresh every hour, only issuing a deploy when a structural change to the data warehouse is needed.

## REST API

Coalesce also offers a rest API for some of the functionality provided by `coa`, the CLI tool. The rest API can be a useful option for those that do not want to maintain on-premise infrastructure, but it does connect directly to your Data Platform environment from the Coalesce Cloud backend.

---

## Build Pipelines With the Coalesce CLI

In this guide, you'll clone your repo, use the Coalesce CLI with the `coa` command, and create and run Nodes locally.

## Before You Begin

Make sure to complete [CLI setup][] so you have a valid `~/.coa/config` file, credentials, and project structure.

## The Core Development Loop

Local development with COA is separate from Coalesce Cloud. Rather than going through a full plan and deploy cycle every time you change a query, you work directly against your warehouse using the files in your version control checkout. This lets you move fast: edit a SQL file, see the generated DDL, create the table, check the data, and adjust, all without touching your Coalesce Environment.

This loop is for local iteration. Once your pipeline is working the way you want, you push to your repository and move to `coa plan`, `coa deploy`, and `coa refresh` to promote it to a Coalesce Environment and run it under cloud operations.

| Step | What it does | Command |
| --- | --- | --- |
| Define | Write a `.sql` file in `nodes/` with your transformation SQL | Author the file as described in the next section |
| Preview | See the generated SQL without executing it | `coa create --dry-run --verbose` |
| Create | Run DDL to create the table or view in your warehouse | `coa create --include "{ NODE }"` |
| Run | Run DML to populate the table with data | `coa run --include "{ NODE }"` |
| Verify | Check the results in your warehouse | Use your SQL client, for example a Snowflake worksheet or another client |
| Iterate | Edit, re-create, re-run | Repeat |

There is no `coa fetch` command to query results from the CLI. Use your preferred SQL client to verify data after running.

## 1. Clone Your Coalesce Repo

Clone the repository that contains your Coalesce Project metadata.

```bash
git clone <your-coalesce-project-repo-url>
cd my-project
```

Your project should have this structure:

```text
my-project/
├── data.yml              # Root metadata (fileVersion, platformKind)
├── locations.yml         # Storage location manifest
├── nodes/                # Pipeline nodes (.yml for V1, .sql for V2)
├── nodeTypes/            # Node type definitions with templates
├── environments/         # Environment configs with storage mappings
├── macros/               # Reusable SQL macros
├── jobs/                 # Job definitions
└── subgraphs/            # Subgraph definitions
```

## 2. Set Up Project Files

You need a `workspaces.yml` file at the project root so local commands know which database and schema each storage location uses. For a new Snowflake project, use `coa init`. For an existing repository, create the file yourself or use `coa doctor --fix`.

### New Projects: Run `coa init`

From an empty project directory, run:

```bash
coa init
```

The interactive setup asks for your project name, Coalesce domain, Snowflake credentials with basic auth or key pair authentication, and `SRC` and `TARGET` storage mappings. It writes `data.yml`, `workspaces.yml`, and updates `~/.coa/config`, installs selected Node types, and runs `coa doctor` when it finishes.

:::warning[Existing configuration files]

If `workspaces.yml`, `data.yml`, or related files already exist, `coa init` prompts before overwriting them. Back up your project first when you are not starting from scratch.

:::

### Existing Projects: Create workspaces.yml Manually

Create `workspaces.yml` at the project root, at the same level as `data.yml`. This file is not usually committed to version control, so add it to `.gitignore`.

Align the location keys, for example `SRC` and `TARGET`, with the names in your project's `locations.yml`.

```yaml
workspaces:
  default:
    SRC:
      database: SNOWFLAKE_SAMPLE_DATA
      schema: TPCH_SF1
    TARGET:
      database: MY_DATABASE
      schema: MY_SCHEMA
```

## 3. Verify Your Setup

Run `coa doctor` to confirm project files, credentials, and warehouse connectivity:

```bash
coa doctor
```

A healthy setup looks like:

```text
Project:
  ✓ data.yml
  ✓ locations.yml
  ✓ workspaces.yml
      SRC     ->  SNOWFLAKE_SAMPLE_DATA.TPCH_SF1
      TARGET  ->  MY_DATABASE.MY_SCHEMA
  ✓ connection profile  "default" (from ~/.coa/config)

Connection: Basic
  ✓ account  your-account
  ✓ user  your-user
  ✓ warehouse  connected (snowflake)
```

Fix any issues before continuing. Run `coa describe config` if you need help with credential format.

`coa doctor` validates file structure and connectivity but does not verify that the databases and schemas in your `workspaces.yml` actually exist in your warehouse. You will discover those issues on your first `coa create`.

## 4. Create a Node Type for SQL Transformations

With COA, you can create Nodes without needing the Coalesce App. Before writing `.sql` transformation Nodes, you need at least one Node type configured for [Node Type V2][]. This defines the DDL and DML templates that `coa` uses to create and populate tables.

Node type directories follow the pattern `<Name>-<UUID>/`. The name can be any name you want. The UUID should match what the Coalesce App generates. For quick UUID values, use [UUID Generator][].

```bash
mkdir -p nodeTypes/Stage-a1b2c3d4-e5f6-7890-abcd-ef1234567890
```

### Create the Node Template

Add `definition.yml`, `create.sql.j2`, and `run.sql.j2` under your node type directory.

<Tabs>
  <TabItem value="definition-yml" label="definition.yml" default>

    Set `fileVersion: 2` in `definition.yml`. This tells `coa` that this node type supports `.sql` files. See [Node Type V2][] for how SQL-first Nodes work in Coalesce.

        ```yaml
        fileVersion: 2
        id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
        isDisabled: false
        metadata:
        defaultStorageLocation: null
        error: null
        nodeMetadataSpec: |-
            capitalized: SQL Stage
            short: SQL
            plural: SQLStages
            tagColor: '#2EB67D'

            config:
            - groupName: Options
            items:
            - type: materializationSelector
                default: table
                options:
                - table
                - view
                isRequired: true

            - type: multisourceToggle
                enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}"

            - type: overrideSQLToggle
                enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"

            - displayName: Multi Source Strategy
                attributeName: insertStrategy
                type: dropdownSelector
                default: INSERT
                options:
                - "INSERT"
                - "UNION"
                - "UNION ALL"
                isRequired: true
                enableIf: "{% if node.isMultisource %} true {% else %} false {% endif %}"

            - displayName: Truncate Before
                attributeName: truncateBefore
                type: toggleButton
                default: true

            - displayName: Enable Tests
                attributeName: testsEnabled
                type: toggleButton
                default: true

            - displayName: Pre-SQL
                attributeName: preSQL
                type: textBox
                syntax: sql
                isRequired: false

            - displayName: Post-SQL
                attributeName: postSQL
                type: textBox
                syntax: sql
                isRequired: false
        name: Stage V2
        type: NodeType
        ```

  </TabItem>
  <TabItem value="create-sql" label="create.sql.j2 DDL Template">

        ```sql
        {% if node.override.create.enabled %}

            {{ node.override.create.script }}

        {% elif node.materializationType == 'table' %}
            {{ stage('Create Stage Table') }}

            CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
            (
                {% for col in columns %}
                    "{{ col.name }}" {{ col.dataType }}
                    {%- if not col.nullable or col.notNull %} NOT NULL
                        {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
                    {% endif %}
                    {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                    {%- if not loop.last -%}, {% endif %}
                {% endfor %}
            )
            {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

        {% elif node.materializationType == 'view' %}
            {{ stage('Create Stage View') }}

            CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
            (
                {% for col in columns %}
                    "{{ col.name }}"
                    {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                    {%- if not loop.last -%}, {% endif %}
                {% endfor %}
            )
            {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
            AS
            {% for source in sources %}
                SELECT
                {% for col in source.columns %}
                    {{ get_source_transform(col) }} AS "{{ col.name }}"
                    {%- if not loop.last -%}, {% endif %}
                {% endfor %}

                {{ source.join }}

                {% if not loop.last %}
                    {% if config.insertStrategy in ['UNION', 'UNION ALL'] %}
                        {{ config.insertStrategy }}
                    {% else %}
                        UNION
                    {% endif %}
                {% endif %}
            {% endfor %}

        {% endif %}
        ```

  </TabItem>
  <TabItem value="run-sql" label="run.sql.j2 DML Template">

        ```sql
        {# --- Pre-SQL --- #}
        {% if config.preSQL is defined and config.preSQL.parameters is defined -%}
            {% for sql in config.preSQL.parameters -%}
                {{ stage('Pre-SQL ' + loop.index|string) }}
                {{ sql }}
            {% endfor %}
        {%- endif %}

        {# --- Insert Strategy --- #}
        {% if config.insertStrategy is defined and config.insertStrategy.parameters is defined %}
            {% if config.insertStrategy.parameters[0] == 'MERGE' %}
                {{ stage('Merge Data') }}
                MERGE INTO {{ ref_no_link(node.location.name, node.name) }} TGT
                USING (
                {% for source in sources %}
                    {{ source.cteString }}
                    SELECT
                    {% for col in source.columns %}
                        {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                    {{ source.join }}
                {% endfor %}
                ) SRC
                ON {% for col in columns if col.isBusinessKey %}{% if not loop.first %} AND {% endif %}TGT."{{ col.name }}" = SRC."{{ col.name }}"{% endfor %}
                WHEN MATCHED THEN UPDATE SET
                    {% for col in columns if col.isChangeTracking %}
                        TGT."{{ col.name }}" = SRC."{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                WHEN NOT MATCHED THEN INSERT (
                    {% for col in columns %}
                        "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                ) VALUES (
                    {% for col in columns %}
                        SRC."{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                );

            {% elif config.insertStrategy.parameters[0] == 'TRUNCATE' %}
                {{ stage('Truncate Table') }}
                TRUNCATE TABLE {{ ref_no_link(node.location.name, node.name) }};

                {{ stage('Insert Data') }}
                INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                )
                {% for source in sources %}
                    {{ source.cteString }}
                    SELECT
                    {% for col in source.columns %}
                        {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                    {{ source.join }}
                    {% if not loop.last %}UNION ALL{% endif %}
                {% endfor %};

            {% else %}
                {{ stage('Insert Data') }}
                INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                )
                {% for source in sources %}
                    {{ source.cteString }}
                    SELECT
                    {% for col in source.columns %}
                        {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
                    {% endfor %}
                    {{ source.join }}
                    {% if not loop.last %}UNION ALL{% endif %}
                {% endfor %};

            {% endif %}
        {% else %}
            {{ stage('Insert Data') }}
            INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
            (
                {% for col in columns %}
                    "{{ col.name }}"{% if not loop.last %}, {% endif %}
                {% endfor %}
            )
            {% for source in sources %}
                {{ source.cteString }}
                SELECT
                {% for col in source.columns %}
                    {{ get_source_transform(col) }} AS "{{ col.name }}"{% if not loop.last %}, {% endif %}
                {% endfor %}
                {{ source.join }}
                {% if not loop.last %}UNION ALL{% endif %}
            {% endfor %};

        {% endif %}

        {# --- Post-SQL --- #}
        {% if config.postSQL is defined and config.postSQL.parameters is defined -%}
            {% for sql in config.postSQL.parameters -%}
                {{ stage('Post-SQL ' + loop.index|string) }}
                {{ sql }}
            {% endfor %}
        {%- endif %}

        {# --- Tests --- #}
        {% if config.testsEnabled -%}
            {% if config.tests is defined and config.tests.parameters is defined -%}
                {% for test in config.tests.parameters -%}
                    {{ test_stage(test) }}
                    {{ test }}
                {% endfor %}
            {%- endif %}

            {% for column in columns -%}
                {% if column.tests is mapping and column.tests.parameters is defined -%}
                    {% for test in column.tests.parameters -%}
                        {{ test_stage(column.name + ": Test " + loop.index|string) }}
                        {{ test }}
                    {%- endfor %}
                {%- endif %}
            {%- endfor %}
        {%- endif %}
        ```

  </TabItem>
</Tabs>

Run `coa describe node-types` for the full reference on template variables, config options, and advanced patterns like MERGE.

## 5. Write and Run Your First Transformation Node

Now that you have a Node Type, you can create a Node. Create a `.sql` file in `nodes/`. The filename pattern is `<STORAGE_LOCATION>-<NodeName>.sql`.

Create `nodes/TARGET-STG_CUSTOMER.sql`:

```sql
@id("8195ddbe-6034-4bc3-821b-8f479c5b1cdf")
@nodeType("a1b2c3d4-e5f6-7890-abcd-ef1234567890")
SELECT
  C_CUSTKEY AS CUSTOMER_KEY,
  C_NAME AS CUSTOMER_NAME,
  C_ADDRESS AS ADDRESS,
  C_NATIONKEY AS NATION_KEY,
  C_PHONE AS PHONE,
  C_ACCTBAL AS ACCOUNT_BALANCE
FROM {{ ref('SRC', 'CUSTOMER') }}
```

The `@id` annotation is auto-generated and should not be changed once set. The `@nodeType` annotation must map to a valid node type ID in your `nodeTypes/` directory.

Always use single quotes inside `ref()`, for example `ref('SRC', 'CUSTOMER')`. Double quotes silently break lineage resolution and produce `UNKNOWN` column types with no error message.

### Preview First With a Dry Run

Always preview before executing:

```bash
coa create --include "{ STG_CUSTOMER }" --dry-run --verbose
```

The command prints generated DDL without executing it. Sample output includes a header like this:

```text
##{ Node Name }
```

Check that:

- The table name resolves correctly and is not blank.
- Column types are not `UNKNOWN`. If they are, recheck your `ref()` targets.
- The SQL looks correct overall.

If dry-run shows `CREATE TABLE ()` with no columns, your SQL likely has a parse error: a typo or missing keyword. The SQL parser for [Node Type V2][] silently produces zero columns when it cannot parse the SELECT. Check your SQL syntax carefully; dry-run will not warn you about this directly.

### Create and Run

```bash
# Create the table structure (DDL)
coa create --include "{ STG_CUSTOMER }"

# Populate with data (DML)
coa run --include "{ STG_CUSTOMER }"
```

Verify the results in your SQL client.

## Example: Build a Multi-Layer Pipeline

Add downstream nodes that reference upstream nodes using `ref()`.

Create `nodes/TARGET-STG_ORDERS.sql`:

```sql
@id("a295ddbe-6034-4bc3-821b-8f479c5b1cdf")
@nodeType("a1b2c3d4-e5f6-7890-abcd-ef1234567890")
SELECT
  O_ORDERKEY AS ORDER_KEY,
  O_CUSTKEY AS CUSTOMER_KEY,
  O_ORDERSTATUS AS ORDER_STATUS,
  O_TOTALPRICE AS TOTAL_PRICE,
  O_ORDERDATE AS ORDER_DATE
FROM {{ ref('SRC', 'ORDERS') }}
```

Create `nodes/TARGET-DIM_CUSTOMER.sql` to join two staging tables:

```sql
@id("8432ddbe-6034-4bc3-821b-8f479c5b1cdg")
@nodeType("a1b2c3d4-e5f6-7890-abcd-ef1234567890")
SELECT
  c.CUSTOMER_KEY,
  c.CUSTOMER_NAME,
  c.PHONE,
  c.ACCOUNT_BALANCE,
  COUNT(o.ORDER_KEY) AS TOTAL_ORDERS
FROM {{ ref('TARGET', 'STG_CUSTOMER') }} c
LEFT JOIN {{ ref('TARGET', 'STG_ORDERS') }} o
  ON c.CUSTOMER_KEY = o.CUSTOMER_KEY
GROUP BY c.CUSTOMER_KEY, c.CUSTOMER_NAME, c.PHONE, c.ACCOUNT_BALANCE
```

## Run the Full Pipeline

`coa` executes nodes in dependency order when you use `--all`: sources first, then staging, then downstream nodes.

```bash
# Create all tables (respects dependency order)
coa create --all

# Populate all tables
coa run --all
```

## Node Selectors

Most commands accept `--include` and `--exclude` flags with a selector syntax for targeting specific nodes.

```bash
# Select a node by name (shorthand)
--include "{ STG_ORDERS }"

# Select by exact name (value must be quoted)
--include '{ name: "STG_ORDERS" }'

# Select by storage location
--include '{ location: "SRC" }'

# Select by node type
--include '{ nodeType: "Stage" }'

# Combine multiple nodes with OR
--include '{ STG_ORDERS } || { STG_CUSTOMER }'

# Select everything
--all

# List all available nodes
coa create --list-nodes
```

### Common Selector Mistakes

These patterns often return no matched nodes without a clear error:

- **Separate braces for OR** - Use `{ A } || { B }`, not `{ A || B }`. Putting both selectors inside one pair of braces returns a "no nodes matched" result and no error message.
- **Quoted selector values** - `{ location: "SRC" }` works; `{ location: SRC }` returns no match.
- **`coa refresh` and the word OR** - Cloud refresh commands use the word `OR`, not `||`. For example, `--include '{ name: "STG_CUSTOMER" } OR { name: "STG_ORDERS" }'`. Run `coa describe selectors` to confirm syntax for your installed version.

For the full selector reference, run `coa describe selectors`.

## Validation

Run `coa validate` to check your workspace for structural issues before running:

```bash
coa validate
coa validate --verbose    # Details on each scanner
coa validate --json       # Structured output
```

This checks YAML schemas, storage locations, column references, data types, and more.

:::warning[Known Column Reference Scanner Issue]

The Column References scanner produces false positives on SQL Nodes when you use [Node Type V2][]. Aliased columns may be flagged as "references missing source columns." These warnings do not block `coa create` or `coa run`, but they produce a non-zero exit code and may block `coa plan` in some environments. Run `coa validate --verbose` to distinguish real errors from these false positives before troubleshooting further.

:::

---

## What's Next?

Continue with the rest of the CLI documentation when you need the full setup walkthrough, deploy and refresh workflows, policy detail, or troubleshooting help.

- **Start here:** [Deploy Pipelines Using the Coalesce CLI][] for plan, deploy, refresh, and managed Environments.
- [Command Line Interface][] for installation, authentication, profiles, and proxy configuration.
- [CLI Commands][] for `coa` command reference and flags.
- [CLI Support Policy & Usage Recommendations][] for supported versions and upgrade practices.
- [Troubleshoot the Coalesce CLI][] for common errors and diagnostic steps.
- [Node Type V2][] for SQL-first Nodes, annotations, and the editor.

---

[CLI setup]: /docs/coa
[Command Line Interface]: /docs/coa
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Deploy Pipelines Using the Coalesce CLI]: /docs/coa/version-733-and-above/coa-deploying-pipelines
[CLI Support Policy & Usage Recommendations]: /docs/coa/version-733-and-above/coa-support-policy-usage-recommendations
[Troubleshoot the Coalesce CLI]: /docs/coa/version-733-and-above/coa-troubleshooting
[Node Type V2]: /docs/build-your-pipeline/v2-node-types
[UUID Generator]: https://www.uuidgenerator.net/

---

## CLI Commands(Version-733-and-above)

<!-- markdownlint-disable MD033 -->

<!-- vale Coalesce.Headings = NO -->

Run `coa --help` at any time to see the current command list and global options for your installed version. Run `coa <command> --help` for flags specific to a subcommand. Use `coa describe` for built-in documentation that always matches your installed build.

## $ `coa` [options] [command]

```bash
Usage: coa [options] [command]

Options:
  -v, --version                   output the version number
  --verbose                       Show additional detail in command output
  --config <coa-config-location>  coa config file location
  --json                          Output machine-readable JSON instead of human-formatted text
  --no-color                      Disable colored output
  -h, --help                      display help for command

Local Development Commands:
  init      Interactive setup for a new local project
  run       Execute DML (insert/merge) for selected nodes
  create    Execute DDL (create/replace) for selected nodes
  install   Fetch and cache package contents
  validate  Validate YAML schemas and scan workspace for configuration problems
  describe  Explore CLI commands, schemas, and concepts
  doctor    Check config, auth, and warehouse access
  sources   Discover warehouse tables and scaffold source nodes

Cloud Operations:
  deploy           Apply a deployment plan to an environment
  plan             Diff workspace state against deployed state
  refresh          Run DML for selected nodes in a deployed environment
  rerun            Re-run a failed pipeline from the point of failure
  environments     List and manage deployment environments
  nodes            List and inspect deployed nodes
  jobs             List and inspect deployment jobs
  gitAccounts      Manage Git account connections
  workspace-nodes  List workspace node metadata
  projects         List and manage Coalesce projects
  runs             List and inspect pipeline runs
  cancel           Cancel a running pipeline
```

<details>
<summary>Local Development Commands Summary</summary>
  
  These commands work against files in your Git checkout and execute SQL in your warehouse using `workspaces.yml` and `~/.coa/config`. They do not replace environment-scoped `coa refresh`.
  
- **`coa init`**

  Walks you through creating a new local project: project name, Coalesce domain, Snowflake credentials, Storage Location mappings, and default V2 Node scaffolding. Updates `~/.coa/config`, `data.yml`, and `workspaces.yml`, runs `coa install` for selected Node Types, then runs `coa doctor` to report remaining issues.

  ```bash
  coa init
  ```

  If configuration files already exist, the command prompts before overwriting them. New projects created with `coa init` use SQL-first Node Types and do not install legacy default YAML Node Types, except source Nodes.

- **`coa doctor`**
  
  Sanity-checks your config, connectivity, and project files. Run this after setting up or changing `workspaces.yml` or `~/.coa/config`.
  
  ```bash
  coa doctor
  ```
  
- **`coa validate`**
  
  Runs workspace validation and scanners against your project files. Use this to catch structural issues before running.
  
  ```bash
  coa validate
  ```

- **`coa describe`**
  
  Displays built-in documentation that matches your installed build. Topics include selectors, SQL annotations, Node Types, and more. For SQL-first Node authoring in the product, see [Node Type V2][].
  
  ```bash
  coa describe
  coa describe selectors
  ```
  
  Use `coa describe` as the authoritative reference for your version rather than copying examples from external sources.

- **`coa create`**
  
  Generates and applies DDL to create or update tables and views in your warehouse for selected Nodes.
  
  ```bash
  coa create
  coa create --all
  coa create --include <selector> --dry-run
  ```
  
  Supports `--dry-run`, `--include`, `--exclude`, `--all`, and `--list-nodes` where applicable. Selector syntax for local commands is documented in `coa describe selectors` and differs from the `OR`-keyword style used in `coa refresh --include`.

- **`coa run`**
  
  Generates and applies DML to populate tables with data for selected Nodes.
  
  ```bash
  coa run
  coa run --include <selector>
  ```
  
  Uses the same selector style as `coa create`. See `coa describe selectors` for your version's syntax.

- **`coa install`**
  
  Installs or syncs Coalesce Marketplace or packaged content when your workflow uses that command.
  
  ```bash
  coa install
  ```
  
  See `coa install --help` for options specific to your build.

- **`coa sources`**

  Discovers warehouse tables for your Storage Locations and scaffolds source Node YAML under `nodes/`. Use `list` to inspect what is already imported, `add` to create source Nodes from the warehouse, and `remove` to delete files created by scaffolding.

  ```bash
  coa sources list
  coa sources list --location SRC
  coa sources add --location SRC
  coa sources add --location SRC --include 'ORDER*' --dry-run
  coa sources remove --location SRC --include 'OLD_*'
  ```

  Requires Coalesce CLI **7.35** or later. Database and schema on each source Node come from the Storage Mapping in `workspaces.yml`.

</details>

<details>
<summary>Cloud Operations Summary</summary>

  These commands authenticate to Coalesce over HTTPS and require `token`, `domain`, and `environmentID` in your profile:

- **`coa plan`**

    Generates `coa-plan.json` from the current directory or from `--dir` and your profile. Run this from the commit you intend to deploy.

    ```bash
    coa plan
    coa plan --profile databricksOAuthM2M
    coa plan --dir /path/to/repo
    ```

    The plan file captures Environment, Nodes, Jobs, and related metadata. Review it, but avoid hand-editing; change metadata in the Coalesce App or files, commit, and regenerate.

    For large projects, use `--enableCache` to speed up plan generation. Keep caching off unless plan time is a problem, and review how to [speed up your deployments][] before enabling it.

    Common options: `--profile`, `--environmentID`, `--platformKind`, `--parameters`, `--out`, `--gitsha`, and warehouse credential overrides. Run `coa plan -h` for the full list.

- **`coa deploy`**

    Applies an existing plan file to a Coalesce Environment. Defaults to `./coa-plan.json` unless you pass `--plan`.

    ```bash
    coa deploy
    coa deploy --plan ./coa-plan.json --out deploy-results.json
    ```

    Returns run metadata including `runType`, `runStatus`, and `runResults`. Use `--out` to write JSON results to disk.

- **`coa refresh`**

    Runs a Coalesce-managed refresh, typically DML, for an Environment that already has a successful deploy.

    ```bash
    coa refresh
    coa refresh --include '{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM }'
    coa refresh --jobID <id> --parallelism 8 --out refresh-results.json
    ```

    Supports `--include`, `--exclude`, `--jobID`, `--parallelism`, `--parameters`, `--forceIgnoreEnvironmentStatus`, and `--out`.

- **`coa rerun`**

    Retries the failed Nodes from a previous refresh run. Pass the `runID` of the failed run. Runtime parameters from the original run are not preserved automatically; pass `--parameters` when you need a new map.

    ```bash
    coa rerun --runID <id>
    coa rerun --runID <id> --parameters '{"key":"value"}'
    ```

- **`coa cancel`**

    Cancels an in-flight run by `runID`.

    ```bash
    coa cancel --runID <id>
    ```

</details>

<details>

<summary>API-Style Command Groups</summary>

  These groups wrap REST-style operations for scripting and automation. Each supports member commands such as `list`, `get`, `create`, `put`, and `delete` where applicable.
  
  | Command Group | Common Subcommands |
  | --- | --- |
  | `coa environments` | `list`, `get`, `create`, `update`, `delete` |
  | `coa nodes` | `list`, `get` |
  | `coa runs` | `list`, `get`, `list-results` |
  | `coa workspace-nodes` | `list`, `create`, `get`, `put`, `delete` |
  
  Additional groups ship with the product over time. Run `coa --help` to see your build's full list.

</details>

## $ `coa` create [options]

Execute DDL that creates or replaces objects for selected Nodes in your checked-out project. This runs against the warehouse configured for the Workspace in `workspaces.yml`, not through Coalesce cloud.

`coa create -h`

```bash
Usage: coa create [options]

Execute DDL (create/replace) for selected nodes

Options:
  -d, --dir <dir>          Coalesce pipeline Yaml file path (default: ".")
  --include <selector>     Node selector syntax (e.g., '{ ORDERS }+', '{ name:
                           "CUSTOMER" }', '{ location: "SRC" }+')
  --exclude <selector>     Node selector syntax for excluding nodes (e.g., '{
                           TEST }', '{ location: "TEMP" }')
  --workspace <workspace>  Workspace name from workspaces.yml (e.g., dev,
                           prod). If not specified, defaults to 'dev'
  --list-nodes             List all available nodes (names and IDs) and exit
  --list-workspaces        List all available workspaces from workspaces.yml
                           and exit
  --dry-run                Preview which nodes would be affected without
                           executing. Use --verbose to see generated SQL
  -h, --help               display help for command
```

## $ `coa` run [options]

Execute DML that inserts or merges data for selected Nodes. Same Workspace and selector model as `coa create`.

`coa run -h`

```bash
Usage: coa run [options]

Execute DML (insert/merge) for selected nodes

Options:
  -d, --dir <dir>          Coalesce pipeline Yaml file path (default: ".")
  --include <selector>     Node selector syntax (e.g., '{ ORDERS }+', '{ name:
                           "CUSTOMER" }', '{ location: "SRC" }+')
  --exclude <selector>     Node selector syntax for excluding nodes (e.g., '{
                           TEST }', '{ location: "TEMP" }')
  --workspace <workspace>  Workspace name from workspaces.yml (e.g., dev,
                           prod). If not specified, defaults to 'dev'
  --list-nodes             List all available nodes (names and IDs) and exit
  --list-workspaces        List all available workspaces from workspaces.yml
                           and exit
  --dry-run                Preview which nodes would be affected without
                           executing. Use --verbose to see generated SQL
  -h, --help               display help for command
```

## $ `coa` validate [options]

Validate YAML schemas and run Workspace scanners for graph and structural checks.

`coa validate -h`

```bash
Usage: coa validate [options]

Validate YAML schemas and scan workspace for configuration problems

Options:
  -d, --dir <dir>          Root directory to scan for YAML files (default: ".")
  --include <selector>     Node selector to scope graph scanner results (e.g.,
                           '{ STG_ORDERS }', '{ location: "WORK" }'). Schema
                           validation always runs on all files.
  --exclude <selector>     Node selector to exclude from graph scanner results
  --workspace <workspace>  Workspace name from workspaces.yml (default: dev)
                           (default: "dev")
  -h, --help               display help for command
```

## $ `coa` doctor [options]

Check configuration, authentication, and warehouse access for a workspace.

`coa doctor -h`

```bash
Usage: coa doctor [options]

Check config, auth, and warehouse access

Options:
  -d, --dir <dir>          Coalesce pipeline YAML file path (default: ".")
  --workspace <workspace>  Workspace to check (default: dev)
  --fix                    Fix issues where possible (e.g., bootstrap missing
                           workspaces.yml)
  -h, --help               display help for command
```

## $ `coa` init [options]

Set up a new local Coalesce project through an interactive questionnaire.

`coa init -h`

```bash
Usage: coa init [options]

Interactive setup for a new local project

Options:
  -d, --dir <dir>          Project directory (default: ".")
  -h, --help               display help for command
```

The questionnaire collects:

- **Project name**
- **Coalesce domain** - Choose a standard regional URL, for example North America, EMEA, or Canada, or enter a custom domain for PrivateLink or new regions
- **Snowflake credentials** - Basic authentication or key pair, plus warehouse and role. The command tests connectivity before continuing.
- **Storage Location mappings** - Database and schema for `SRC` and `TARGET` locations

When setup finishes, `coa init` writes project files, runs `coa install` for the Node Types you select, and runs `coa doctor`. For an existing repository, create `workspaces.yml` manually or use `coa doctor --fix` instead. See [Build Pipelines With the Coalesce CLI][].

## $ `coa` install [options]

Download and cache Package contents for the Workspace. This uses Coalesce API credentials.

`coa install -h`

```bash
Usage: coa install [options]

Fetch and cache package contents

Options:
  --token <value>          Coalesce Refresh Token
  --domain <value>         Coalesce domain name to use for API requests
                           (default: "https://app.coalescesoftware.io")
  --profile <value>        Profile To Use
  -d, --dir <dir>          Workspace directory (default: ".")
  --workspace <workspace>  Workspace name from workspaces.yml (e.g., dev,
                           prod). If not specified, defaults to 'dev'
  -h, --help               display help for command
```

## $ `coa` describe [options] [topic] [subtopic]

Print long-form help for commands, selectors, schemas, workflows, SQL annotations, and Node Types. Topics include `commands`, `command`, `selectors`, `schemas`, `schema`, `workflow`, `structure`, `concepts`, `sql-format`, and `node-types`. For topic `command` or `schema`, pass the command or schema name as `subtopic`. See [Node Type V2][] for documentation on SQL-first Node Types in Coalesce.

`coa describe -h`

```bash
Usage: coa describe [options] [topic] [subtopic]

Explore CLI commands, schemas, and concepts

Arguments:
  topic       Topic: commands, command, selectors, schemas, schema, workflow,
              structure, concepts, sql-format, node-types
  subtopic    Subtopic: command name (for 'command') or schema type (for
              'schema')

Options:
  -h, --help  display help for command
```

## $ `coa` sources [options] [command]

Discover warehouse tables for your Storage Locations and scaffold source Node YAML files under `nodes/`. Use `list` to inspect what exists, `add` to import, and `remove` to delete files created by scaffolding. These commands require Coalesce CLI **7.35** or later. They connect to your warehouse using the profile and Workspace configuration in `~/.coa/config` and `workspaces.yml`.

`coa sources -h`

```bash
Usage: coa sources [options] [command]

Discover warehouse tables and scaffold source nodes

Options:
  -h, --help        display help for command

Commands:
  list [options]    Show locations and how many tables are already imported as
                    sources
  add [options]     Scaffold source node YAML files from warehouse tables
  remove [options]  Delete scaffolded source node YAML files
```

### `coa sources list`

Show Storage Locations and how many warehouse tables are already imported as source Nodes. Pass `--location` for per-table detail, including whether a source Node already exists.

`coa sources list -h`

```bash
Usage: coa sources list [options]

Show locations and how many tables are already imported as sources

Options:
  -d, --dir <dir>    Workspace directory (default: ".")
  --profile <value>  Profile from ~/.coa/config
  --location <name>  Scope to one storage location (show per-table detail)
  --all              Show every table across every location (can be large)
  --include <glob>   Filter tables by glob pattern (e.g. 'ORDER*')
  -h, --help         display help for command
```

```bash
coa sources list
coa sources list --location SRC
coa sources list --all --json
coa sources list --location SRC --include 'ORDER*'
```

### `coa sources add`

Scaffold source Node YAML from warehouse tables and write metadata under `nodes/`. Database and schema on each source Node come from the Storage Mapping for that location in `workspaces.yml`. On a TTY, running with no scope flags starts an interactive prompt. In non-interactive runs, pass `--all` or `--location`.

`coa sources add -h`

```bash
Usage: coa sources add [options]

Scaffold source node YAML files from warehouse tables

Options:
  -d, --dir <dir>    Workspace directory (default: ".")
  --profile <value>  Profile from ~/.coa/config
  --all              Import from every location (cannot combine with
                     --location)
  --location <name>  Import from one storage location
  --include <glob>   Filter tables by glob pattern (e.g. 'ORDER*')
  --overwrite        Also re-import tables that already have source nodes
                     (default: skip)
  --dry-run          List what would be imported without writing files
  -h, --help         display help for command
```

```bash
coa sources add
coa sources add --all
coa sources add --location SRC
coa sources add --location SRC --include 'ORDER*'
coa sources add --all --overwrite
coa sources add --all --dry-run
```

Use `--dry-run` to list the tables that would be imported without writing files. Use `--overwrite` only when you intend to re-import tables that already have source Nodes.

### `coa sources remove`

Delete source Node YAML files previously created by scaffolding from `nodes/`. Only files that match the scaffolding naming convention are removed. Hand-renamed files are left alone.

`coa sources remove -h`

```bash
Usage: coa sources remove [options]

Delete scaffolded source node YAML files

Options:
  -d, --dir <dir>    Workspace directory (default: ".")
  --all              Remove every scaffolded source node (cannot combine with
                     --location)
  --location <name>  Remove source nodes from one storage location
  --include <glob>   Filter tables by glob pattern (e.g. 'OLD_*')
  --yes              Skip confirmation prompt (non-TTY runs always skip)
  -h, --help         display help for command
```

```bash
coa sources remove
coa sources remove --location SRC --include 'OLD_*'
coa sources remove --all --yes
```

## $ `coa` plan [options]

Compare Workspace state in your Git checkout to deployed state and write a deployment plan file, `coa-plan.json` by default. Run `coa plan` from the directory that contains your Coalesce project files, or pass `--dir`.

During plan generation, Coalesce reports **warnings** for undefined `{{ parameters... }}` references. Warnings do not block writing `coa-plan.json` or running `coa deploy`. See [Overwriting at Deploy][] for how the check works and how to fix warnings.

If you have a large project, `--enableCache` can [speed up your deployments][].

`coa plan -h`

```bash
Usage: coa plan [options]

Diff workspace state against deployed state

Options:
  --token <value>                      Coalesce Refresh Token
  --domain <value>                     Coalesce domain name to use for API
                                       requests (default:
                                       "https://app.coalescesoftware.io")
  --profile <value>                    Profile To Use
  --environmentID <value>              Environment ID
  --include <selector>                 Coalesce Node Selector
  --exclude <selector>                 Coalesce Node Selector
  --jobID <value>                      Coalesce JobID
  --parallelism <value>                Parallelism level to use
  --parameters <value>                 Parameters
  --platformKind <value>               Platform Kind (Snowflake, Databricks)
                                       (default: "Snowflake")
  --snowflakeAuthType <value>          Snowflake Auth Type (Basic, KeyPair)
  --snowflakeAccount <value>           Snowflake Account
  --snowflakeKeyPairKey <value>        Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>       Snowflake Key Pair Pass
  --snowflakePassword <value>          Snowflake Password
  --snowflakeRole <value>              Snowflake Role
  --snowflakeUsername <value>          Snowflake Username
  --snowflakeWarehouse <value>         Snowflake Warehouse
  --snowflakeKeyPairPath <value>       Snowflake Key Pair Path (deprecated,
                                       please use --snowflakeKeyPairKey)
  --databricksAuthType <value>         Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>         Databricks Client ID
  --databricksClientSecret <value>     Databricks Client Secret
  --databricksHost <value>             Databricks Host URL
  --databricksPath <value>             Databricks Path
  --databricksToken <value>            Databricks Token
  --bigQueryAuthType <value>           BigQuery Auth Type (ServiceAccount)
  --bigQueryServiceAccountKey <value>  BigQuery Service Account Key Path
  --bigQueryClientEmail <value>        BigQuery Client Email
  -d, --dir <dir>                      Coalesce pipeline Yaml file path
                                       (default: ".")
  --out <plan-location>                Coalesce plan location (default:
                                       "./coa-plan.json")
  --gitsha <sha>                       Sets the git commit SHA shown in the
                                       plan manifest and UI. This does not
                                       affect which files are used—those come
                                       from the current directory or are set
                                       with the --dir parameter.
  --enableCache                        Enable caching to improve deploy plan
                                       generation times. Coalesce recommends
                                       keeping caching disabled unless you’re
                                       experiencing very long plan generation
                                       times. (default: false)
  -h, --help                           display help for command
```

## $ `coa` deploy [options]

Apply an existing `coa-plan.json` to an Environment. You must run `coa plan` first.

### Deploy Results

`coa` returns information about your run, including the following:

- `runType` - Returns the type of run. `deploy` or `refresh`.
- `runStatus` - The status of the run.
  - `canceled`
  - `completed`
  - `failed`
  - `running`
  - `waitingToRun`
- `runResults` - Includes the SQL executed, any error details, and other environment information.

`coa deploy -h`

```bash
Usage: coa deploy [options]

Apply a deployment plan to an environment

Options:
  --token <value>                      Coalesce Refresh Token
  --domain <value>                     Coalesce domain name to use for API
                                       requests (default:
                                       "https://app.coalescesoftware.io")
  --profile <value>                    Profile To Use
  --environmentID <value>              Environment ID
  --platformKind <value>               Platform Kind (Snowflake, Databricks)
                                       (default: "Snowflake")
  --snowflakeAuthType <value>          Snowflake Auth Type (Basic, KeyPair)
  --snowflakeAccount <value>           Snowflake Account
  --snowflakeKeyPairKey <value>        Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>       Snowflake Key Pair Pass
  --snowflakePassword <value>          Snowflake Password
  --snowflakeRole <value>              Snowflake Role
  --snowflakeUsername <value>          Snowflake Username
  --snowflakeWarehouse <value>         Snowflake Warehouse
  --snowflakeKeyPairPath <value>       Snowflake Key Pair Path (deprecated,
                                       please use --snowflakeKeyPairKey)
  --databricksAuthType <value>         Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>         Databricks Client ID
  --databricksClientSecret <value>     Databricks Client Secret
  --databricksHost <value>             Databricks Host URL
  --databricksPath <value>             Databricks Path
  --databricksToken <value>            Databricks Token
  --bigQueryAuthType <value>           BigQuery Auth Type (ServiceAccount)
  --bigQueryServiceAccountKey <value>  BigQuery Service Account Key Path
  --bigQueryClientEmail <value>        BigQuery Client Email
  -p, --plan <plan-location>           Coalesce plan location (default:
                                       "./coa-plan.json")
  -o, --out <output json>              Run Results Output in Json
  -h, --help                           display help for command
```

## $ `coa` refresh [options]

Run DML for selected Nodes in a deployed Coalesce Environment. A successful deploy must exist before refresh.

To re-run a failed refresh, use `coa rerun <runID>`. That only retries failed Nodes from the prior run.

### Refresh Results

`coa` returns information about your run, including the following:

- `runType` - Returns the type of run. `deploy` or `refresh`.
- `runStatus` - The status of the run.
  - `canceled`
  - `completed`
  - `failed`
  - `running`
  - `waitingToRun`
- `runResults` - Includes the SQL executed, any error details, and other environment information.

`coa refresh -h`

```bash
Usage: coa refresh [options]

Run DML for selected nodes in a deployed environment

Options:
  --token <value>                      Coalesce Refresh Token
  --domain <value>                     Coalesce domain name to use for API
                                       requests (default:
                                       "https://app.coalescesoftware.io")
  --profile <value>                    Profile To Use
  --environmentID <value>              Environment ID
  --include <selector>                 Coalesce Node Selector
  --exclude <selector>                 Coalesce Node Selector
  --jobID <value>                      Coalesce JobID
  --parallelism <value>                Parallelism level to use
  --parameters <value>                 Parameters
  --platformKind <value>               Platform Kind (Snowflake, Databricks)
                                       (default: "Snowflake")
  --snowflakeAuthType <value>          Snowflake Auth Type (Basic, KeyPair)
  --snowflakeAccount <value>           Snowflake Account
  --snowflakeKeyPairKey <value>        Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>       Snowflake Key Pair Pass
  --snowflakePassword <value>          Snowflake Password
  --snowflakeRole <value>              Snowflake Role
  --snowflakeUsername <value>          Snowflake Username
  --snowflakeWarehouse <value>         Snowflake Warehouse
  --snowflakeKeyPairPath <value>       Snowflake Key Pair Path (deprecated,
                                       please use --snowflakeKeyPairKey)
  --databricksAuthType <value>         Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>         Databricks Client ID
  --databricksClientSecret <value>     Databricks Client Secret
  --databricksHost <value>             Databricks Host URL
  --databricksPath <value>             Databricks Path
  --databricksToken <value>            Databricks Token
  --bigQueryAuthType <value>           BigQuery Auth Type (ServiceAccount)
  --bigQueryServiceAccountKey <value>  BigQuery Service Account Key Path
  --bigQueryClientEmail <value>        BigQuery Client Email
  -o, --out <output json>              Run Results Output in Json
  --forceIgnoreEnvironmentStatus       Ignore the environment status and
                                       refresh even if there is a failed
                                       deploy. This may cause refresh failures!
  -h, --help                           display help for command
```

## $ `coa` rerun [options] `<runID>`

Re-run a failed pipeline from the point of failure. Runtime parameters from the previous run are not preserved.

`coa` returns the same run fields as `coa refresh`: run type, status, and results.

`coa rerun -h`

```bash
Usage: coa rerun [options] <runID>

Re-run a failed pipeline from the point of failure

Arguments:
  runID                                The ID of the failed refresh to re-run

Options:
  --token <value>                      Coalesce Refresh Token
  --domain <value>                     Coalesce domain name to use for API
                                       requests (default:
                                       "https://app.coalescesoftware.io")
  --profile <value>                    Profile To Use
  --environmentID <value>              Environment ID
  --include <selector>                 Coalesce Node Selector
  --exclude <selector>                 Coalesce Node Selector
  --jobID <value>                      Coalesce JobID
  --parallelism <value>                Parallelism level to use
  --parameters <value>                 Parameters
  --platformKind <value>               Platform Kind (Snowflake, Databricks)
                                       (default: "Snowflake")
  --snowflakeAuthType <value>          Snowflake Auth Type (Basic, KeyPair)
  --snowflakeAccount <value>           Snowflake Account
  --snowflakeKeyPairKey <value>        Snowflake Key Pair Path
  --snowflakeKeyPairPass <value>       Snowflake Key Pair Pass
  --snowflakePassword <value>          Snowflake Password
  --snowflakeRole <value>              Snowflake Role
  --snowflakeUsername <value>          Snowflake Username
  --snowflakeWarehouse <value>         Snowflake Warehouse
  --snowflakeKeyPairPath <value>       Snowflake Key Pair Path (deprecated,
                                       please use --snowflakeKeyPairKey)
  --databricksAuthType <value>         Databricks Auth Type (OAuthM2M, Token)
  --databricksClientID <value>         Databricks Client ID
  --databricksClientSecret <value>     Databricks Client Secret
  --databricksHost <value>             Databricks Host URL
  --databricksPath <value>             Databricks Path
  --databricksToken <value>            Databricks Token
  --bigQueryAuthType <value>           BigQuery Auth Type (ServiceAccount)
  --bigQueryServiceAccountKey <value>  BigQuery Service Account Key Path
  --bigQueryClientEmail <value>        BigQuery Client Email
  -o, --out <output json>              Run Results Output in Json
  --forceIgnoreEnvironmentStatus       Ignore the environment status and
                                       refresh even if there is a failed
                                       deploy. This may cause refresh failures!
  -h, --help                           display help for command
```

## $ `coa` environments [options] [command]

List and manage deployment Environments through the Coalesce cloud API. Use `create`, `update`, and `delete` to provision and maintain Environments for `coa plan` and `coa deploy`. These commands require Coalesce **7.36** or later.

`coa environments -h`

```bash
Usage: coa environments [options] [command]

List and manage deployment environments

Options:
  -h, --help      display help for command

Commands:
  list [options]    List Environments
  get [options]     Get Environment
  create [options]  Create Environment
  update [options]  Update Environment
  delete [options]  Delete Environment
  help [command]    display help for command
```

### `coa environments list`

`coa environments list -h`

```bash
Usage: coa environments list [options]

List Environments

Options:
  --detail                 Include the full detail of the environments.
                           (default: false)
  --project <string>       Filter results to Environments in this Project ID
  --limit <integer>        The maximum number of environments to return.
                           (default: 100)
  --startingFrom <string>  The cursor point for paging the query results.
  --orderBy <string>       The field to order the results by. (default: "id")
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --paging                 Enable interactive paging
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

Use `--project` when you work in a multi-Project Org and need only the Environments for one Project. The server applies the filter before paging, so `total` and `next` match the filtered set.

### `coa environments get`

`coa environments get -h`

```bash
Usage: coa environments get [options]

Get Environment

Options:
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --environmentID <value>  Environment ID
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa environments create`

`coa environments create -h`

```bash
Usage: coa environments create [options]

Create Environment

Options:
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

An Environment belongs to a Coalesce Project. If you do not have a Project yet, run `coa projects create` first. Use `coa environments list` to read the new Environment ID for `coa plan` and `coa deploy`.

Pass a JSON request file with `--inputFile`. Required fields are `project`, `name`, and `oauthEnabled`. For Snowflake Projects, include `connectionAccount`. For Databricks Projects, include `accessUrl`.

:::info[Credentials and mappings]

Create and update requests configure connection metadata only. Add warehouse passwords or OAuth refresh tokens in the Coalesce App under **Build Settings > Environments > User Credentials** after creation. Storage Mappings on deployed Environments are normally set when you deploy from a Workspace.

:::

Example Snowflake create request file:

```json title="create-environment.json"
{
  "project": "your-project-id",
  "name": "QA",
  "oauthEnabled": false,
  "connectionAccount": "xy12345.us-east-1",
  "description": "QA deployment target",
  "runTimeParameters": {
    "env": "qa"
  }
}
```

### `coa environments update`

`coa environments update -h`

```bash
Usage: coa environments update [options]

Update Environment

Options:
  --environmentID <value>     Environment ID
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

`coa environments update` sends a partial update to the Environment. Omitted fields stay unchanged. Set a field to `null` to clear nullable values such as `accessUrl`.

Example patch request file:

```json title="update-environment.json"
{
  "name": "QA Updated",
  "description": "Updated QA deployment target",
  "runTimeParameters": {
    "env": "qa-updated"
  }
}
```

Run:

```bash
coa environments update --environmentID 6 --inputFile update-environment.json
```

See [Update Environment][] in the API reference for the full request schema.

### `coa environments delete`

`coa environments delete -h`

```bash
Usage: coa environments delete [options]

Delete Environment

Options:
  --environmentID <value>  Environment ID
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

The command prompts for confirmation before it calls the API. Pass `--skipConfirm` to skip the prompt in automation. Ensure the Environment has no active runs or dependent Job Schedules before you delete it.

## $ `coa` nodes [options] [command]

List and inspect deployed Nodes for an Environment.

`coa nodes -h`

```bash
Usage: coa nodes [options] [command]

List and inspect deployed nodes

Options:
  -h, --help      display help for command

Commands:
  list [options]  List Nodes
  get [options]   Get Node
  help [command]  display help for command
```

### `coa nodes list`

`coa nodes list -h`

```bash
Usage: coa nodes list [options]

List Nodes

Options:
  --detail                 Include the full detail of the nodes. (default:
                           false)
  --limit <integer>        The maximum number of nodes to return. (default:
                           100)
  --startingFrom <string>  The cursor point for paging the query results.
  --orderBy <string>       The field to order the results by. (default: "id")
  --skipParsing            Skip parsing column references and updating sources
                           for the nodes. (default: false)
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --environmentID <value>  Environment ID
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --paging                 Enable interactive paging
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa nodes get`

`coa nodes get -h`

```bash
Usage: coa nodes get [options]

Get Node

Options:
  --nodeID <string>        The node ID.
  --skipParsing            Skip parsing column references and updating sources
                           for the nodes. (default: false)
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --environmentID <value>  Environment ID
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

## $ `coa` jobs [options] [command]

Inspect deployment jobs.

`coa jobs -h`

```bash
Usage: coa jobs [options] [command]

List and inspect deployment jobs

Options:
  -h, --help      display help for command

Commands:
  get [options]   Get Job
  help [command]  display help for command
```

### `coa jobs get`

`coa jobs get -h`

```bash
Usage: coa jobs get [options]

Get Job

Options:
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --environmentID <value>  Environment ID
  --jobID <value>          Coalesce JobID
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

## $ `coa gitAccounts` [options] [command]

Manage Git account connections for your organization.

`coa gitAccounts -h`

```bash
Usage: coa gitAccounts [options] [command]

Manage Git account connections

Options:
  -h, --help        display help for command

Commands:
  list [options]    List all org git accounts
  create [options]  Create a git account
  get [options]     Get a single git account
  delete [options]  Delete a single git account
  update [options]  Update a single git account
  help [command]    display help for command
```

### `coa gitAccounts list`

`coa gitAccounts list -h`

```bash
Usage: coa gitAccounts list [options]

List all org git accounts

Options:
  --accountOwner <string>  The owner of the git accounts. If not provided will
                           fallback to the requester's accounts.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa gitAccounts create`

`coa gitAccounts create -h`

```bash
Usage: coa gitAccounts create [options]

Create a git account

Options:
  --accountOwner <string>     The owner of the git accounts. If not provided
                              will fallback to the requester's accounts.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

### `coa gitAccounts get`

`coa gitAccounts get -h`

```bash
Usage: coa gitAccounts get [options]

Get a single git account

Options:
  --gitAccountID <string>  The git account ID.
  --accountOwner <string>  The owner of the git accounts. If not provided will
                           fallback to the requester's accounts.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa gitAccounts delete`

`coa gitAccounts delete -h`

```bash
Usage: coa gitAccounts delete [options]

Delete a single git account

Options:
  --gitAccountID <string>  The git account ID.
  --accountOwner <string>  The owner of the git accounts. If not provided will
                           fallback to the requester's accounts.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa gitAccounts update`

`coa gitAccounts update -h`

```bash
Usage: coa gitAccounts update [options]

Update a single git account

Options:
  --gitAccountID <string>     The git account ID.
  --accountOwner <string>     The owner of the git accounts. If not provided
                              will fallback to the requester's accounts.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

## $ `coa workspace-nodes` [options] [command]

List and mutate workspace Node metadata through the Coalesce API. For field shapes used by create and update, see [Create Node][] and [Set Node][].

`coa workspace-nodes -h`

```bash
Usage: coa workspace-nodes [options] [command]

List workspace node metadata

Options:
  -h, --help        display help for command

Commands:
  list [options]    List Workspace Nodes
  create [options]  Create Node
  get [options]     Get Node
  put [options]     Set Node
  delete [options]  Delete Node
  help [command]    display help for command
```

### `coa workspace-nodes list`

`coa workspace-nodes list -h`

```bash
Usage: coa workspace-nodes list [options]

List Workspace Nodes

Options:
  --workspaceID <string>   The workspace ID.
  --detail                 Include the full detail of the nodes. (default:
                           false)
  --limit <integer>        The maximum number of nodes to return. (default:
                           100)
  --startingFrom <string>  The cursor point for paging the query results.
  --orderBy <string>       The field to order the results by. (default: "id")
  --skipParsing            Skip parsing column references and updating sources
                           for the nodes. (default: false)
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --paging                 Enable interactive paging
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa workspace-nodes create`

`coa workspace-nodes create -h`

```bash
Usage: coa workspace-nodes create [options]

Create Node

Options:
  --workspaceID <string>      The workspace ID.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

### `coa workspace-nodes get`

`coa workspace-nodes get -h`

```bash
Usage: coa workspace-nodes get [options]

Get Node

Options:
  --workspaceID <string>   The environment ID.
  --nodeID <string>        The node ID.
  --skipParsing            Skip parsing column references and updating sources
                           for the nodes. (default: false)
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa workspace-nodes put`

`coa workspace-nodes put -h`

```bash
Usage: coa workspace-nodes put [options]

Set Node

Options:
  --workspaceID <string>      The environment ID.
  --nodeID <string>           The node ID.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

### `coa workspace-nodes delete`

`coa workspace-nodes delete -h`

```bash
Usage: coa workspace-nodes delete [options]

Delete Node

Options:
  --workspaceID <string>  The environment ID.
  --nodeID <string>       The node ID.
  --profile <value>       Profile To Use
  --token <value>         Coalesce Refresh Token
  --skipConfirm           Skip any confirmation prompts
  -h, --help              display help for command
```

## $ `coa` projects [options] [command]

List and manage Coalesce projects.

`coa projects -h`

```bash
Usage: coa projects [options] [command]

List and manage Coalesce projects

Options:
  -h, --help        display help for command

Commands:
  list [options]    Get Projects
  create [options]  Create Project
  get [options]     Get Project
  delete [options]  Delete Project
  update [options]  Update a project
  help [command]    display help for command
```

### `coa projects list`

`coa projects list -h`

```bash
Usage: coa projects list [options]

Get Projects

Options:
  --includeWorkspaces      Whether or not to include nested workspace data for
                           all projects.
  --includeJobs            Whether or not to include nested job data for all
                           workspaces in the projects.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa projects create`

`coa projects create -h`

```bash
Usage: coa projects create [options]

Create Project

Options:
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

### `coa projects get`

`coa projects get -h`

```bash
Usage: coa projects get [options]

Get Project

Options:
  --projectID <string>     The project ID.
  --includeWorkspaces      Whether or not to include nested workspace data for
                           all projects.
  --includeJobs            Whether or not to include nested job data for all
                           workspaces in the projects.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa projects delete`

`coa projects delete -h`

```bash
Usage: coa projects delete [options]

Delete Project

Options:
  --projectID <string>        The project ID.
  --includeWorkspaces         Whether or not to include nested workspace data
                              for all projects.
  --includeJobs               Whether or not to include nested job data for all
                              workspaces in the projects.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

### `coa projects update`

`coa projects update -h`

```bash
Usage: coa projects update [options]

Update a project

Options:
  --projectID <string>        The project ID.
  --includeWorkspaces         Whether or not to include nested workspace data
                              for all projects.
  --includeJobs               Whether or not to include nested job data for all
                              workspaces in the projects.
  --profile <value>           Profile To Use
  --token <value>             Coalesce Refresh Token
  --inputFile <request-file>  The request content
  --outputFile <fileName>     Write output to file
  --format <json|text>        Output results as JSON or text (choices: "json",
                              "text", default: "text")
  --skipConfirm               Skip any confirmation prompts
  -h, --help                  display help for command
```

## $ `coa` runs [options] [command]

List and inspect pipeline runs.

`coa runs -h`

```bash
Usage: coa runs [options] [command]

List and inspect pipeline runs

Options:
  -h, --help              display help for command

Commands:
  list [options]          List Runs
  get [options]           Get Run
  list-results [options]  List Run Results
  help [command]          display help for command
```

### `coa runs list`

`coa runs list -h`

```bash
Usage: coa runs list [options]

List Runs

Options:
  --limit <integer>                The maximum number of runs to return.
                                   (default: 25)
  --startingFrom <integer|string>  The starting run ID, runStartTime, or
                                   runEndTime (exclusive) for paging the query
                                   results.
  --orderBy <string>               The field used to sort query results.
                                   Defaults to `id`, but must be explicitly
                                   provided when also using `startingFrom`.
                                   Make sure to match data type.
  --orderByDirection <string>      The sort order for query results. (default:
                                   "desc")
  --projectID <string|array>       One or more project IDs to filter the query
                                   results.
  --runType <string|array>         One or more run types to filter the query
                                   results.
  --runStatus <string|array>       One or more status values to filter the
                                   query results.
  --detail                         Include the full detail of the run.
                                   (default: false)
  --profile <value>                Profile To Use
  --token <value>                  Coalesce Refresh Token
  --environmentID <value>        Environment ID
  --allEnvironments                Include all environments
  --outputFile <fileName>          Write output to file
  --format <json|text>             Output results as JSON or text (choices:
                                   "json", "text", default: "text")
  --paging                         Enable interactive paging
  --skipConfirm                    Skip any confirmation prompts
  -h, --help                       display help for command
```

### `coa runs get`

`coa runs get -h`

```bash
Usage: coa runs get [options]

Get Run

Options:
  --runID <integer>        The run ID.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

### `coa runs list-results`

`coa runs list-results -h`

```bash
Usage: coa runs list-results [options]

List Run Results

Options:
  --runID <integer>        The run ID.
  --profile <value>        Profile To Use
  --token <value>          Coalesce Refresh Token
  --outputFile <fileName>  Write output to file
  --format <json|text>     Output results as JSON or text (choices: "json",
                           "text", default: "text")
  --skipConfirm            Skip any confirmation prompts
  -h, --help               display help for command
```

## $ `coa` cancel [options] `<runID>`

Cancel a running pipeline.

`coa cancel -h`

```bash
Usage: coa cancel [options] <runID>

Cancel a running pipeline

Arguments:
  runID                    The ID of the run to cancel.

Options:
  --token <value>          Coalesce Refresh Token
  --domain <value>         Coalesce domain name to use for API requests
                           (default: "https://app.coalescesoftware.io")
  --profile <value>        Profile To Use
  --environmentID <value>  Environment ID
  -h, --help               display help for command
```

## Parallelism Limits

The `--parallelism` parameter on `coa refresh` controls how many Nodes run concurrently against your warehouse. REST refresh APIs cap parallelism at **64**; the CLI refresh path is not capped the same way. Choose a value your warehouse can sustain.

## Overwriting Defaults

You can override values from your `coa` config by passing flags on the command line or by setting the same fields on API requests you send yourself.

## Using Include and Exclude

Cloud commands such as `coa plan`, `coa refresh`, and `coa rerun` use the same [Selector Queries][] model as the Coalesce App. Local `coa create`, `coa run`, and `coa validate` accept selector strings on `--include` and `--exclude`. Run `coa describe selectors` for the exact syntax your build supports.

```shell title="coa refresh example with include flag"
coa refresh --include '{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION } OR { location: SAMPLE name: ORDERS } OR { location: SAMPLE name: PART } OR { location: SAMPLE name: PARTSUPP } OR { location: SAMPLE name: REGION } OR { location: SAMPLE name: SUPPLIER } OR { location: QA name: STG_PARTSUPP } OR { location: PROD name: STG_PARTSUPP }'
```

## Selectors

### `coa refresh --include` and `--exclude`

The `--include` and `--exclude` flags for `coa refresh` use the **`OR`** keyword, not `||`, to separate selector expressions:

```bash
coa refresh --include '{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION }'
```

Full selector syntax is documented in [Selector Queries][].

### Local Commands for Create and Run

Selector syntax for local commands differs from the `OR`-keyword style used in `coa refresh`. Open `coa describe selectors` and follow the examples for your installed version rather than copying snippets across contexts.

## What's Next?

- [Command Line Interface][] for install, config, profiles, and Snowflake auth
- [Build Pipelines With the Coalesce CLI][] for local `create`, `run`, `validate`, and source Nodes
- [Deploy Pipelines to the Coalesce App Using COA][] for end-to-end cloud deploy
- [Troubleshoot the Coalesce CLI][] when commands fail or validate is noisy

[Selector Queries]: /docs/reference/selector
[Update Environment]: /docs/api/coalesce/update-environment
[Set Node]: /docs/api/coalesce/set-workspace-node
[Create Node]: /docs/api/coalesce/create-node
[speed up your deployments]: /docs/deploy-and-refresh/deploy/caching-deployment-plan
[Overwriting at Deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy
[Command Line Interface]: /docs/coa
[Build Pipelines With the Coalesce CLI]: /docs/coa/version-733-and-above/coa-building-pipelines
[Deploy Pipelines to the Coalesce App Using COA]: /docs/coa/version-733-and-above/coa-deploying-pipelines
[Troubleshoot the Coalesce CLI]: /docs/coa/version-733-and-above/coa-troubleshooting
[Node Type V2]: /docs/build-your-pipeline/v2-node-types

---

## Deploy Pipelines Using the Coalesce CLI

This guide walks you through promoting a pipeline you built locally with `coa` so it runs as a managed Environment in the Coalesce App. You push metadata to your repository, configure deploy-time mappings, sync the Coalesce App, and use cloud commands for plan, deploy, and refresh.

`coa` local development commands `create` and `run` execute SQL directly against your warehouse. They don't go through Coalesce Cloud. To see your pipeline in the Coalesce App and run it as a managed environment, push your code to your repository and deploy it.

## Prerequisites

- A Coalesce Project linked to a version control repository, created in the Coalesce App.
- An Environment for that Project, created in the Coalesce App.
- Cloud credentials in `~/.coa/config`, including `domain`, `token`, and `environmentID`.

## Step 1: Push Your Code to Your Repository

From your project root, initialize version control if you need to, ignore local-only files, and push your first commit:

```bash
git init
echo "workspaces.yml" >> .gitignore
echo "coa-plan.json" >> .gitignore
git add .
git commit -m "Initial pipeline"
git remote add origin <your-project-git-repo-url>
git push -u origin main
```

## Step 2: Configure Environment Storage Mappings

Your Environment needs storage location mappings, meaning the database and schema pairs for each location at deploy time. Add a YAML file under `environments/` using a name like `environments/<ENVIRONMENT_NAME>-<ID>.yml`, then use the following structure:

```yaml
fileVersion: 1
id: "<your-environment-id>"
mappingDefinitions:
  SRC:
    database: <MY_DATABASE>
    schema: <MY_SOURCE_SCHEMA>
  TARGET:
    database: <MY_DATABASE>
    schema: <MY_TARGET_SCHEMA>
name: <My Environment>
type: Environment
```

Get your Environment ID with `coa environments list`. Commit and push this file.

:::info[Configure storage in the App and in YAML]

You also need to configure **Workspace** storage locations in the Coalesce App separately. The Environment YAML configures deploy-time mappings. Workspace mappings are configured in the Coalesce App for the build-time experience.

:::

## Step 3: Pull Into the Coalesce App

In the Coalesce App:

1. Open your Project's Workspace.
2. Set the branch to match what you pushed.
3. Click **Force Checkout** to load the pipeline from your repository.

## Step 4: Plan and Deploy

Generate a plan, then apply it from your machine:

```bash
# Generate a deployment plan
coa plan --environmentID <id>

# Apply the plan
coa deploy --environmentID <id> --plan ./coa-plan.json
```

After a successful deploy, your Nodes appear in the DAG view in the Coalesce App, and you can run them from the Environment.

## Refresh a Deployed Environment

To run DML on already-deployed Nodes without re-deploying, use:

```bash
coa refresh --environmentID <id> --include '{ name: "STG_CUSTOMER" }'
```

## What's Next?

- Read [CLI Commands][] for flags and options on `plan`, `deploy`, and `refresh`.
- Return to [Build Pipelines With the Coalesce CLI][] for the local authoring loop.
- See [Troubleshoot the Coalesce CLI][] when commands or validation behave unexpectedly.

---

[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Build Pipelines With the Coalesce CLI]: /docs/coa/version-733-and-above/coa-building-pipelines
[Troubleshoot the Coalesce CLI]: /docs/coa/version-733-and-above/coa-troubleshooting

---

## CLI Support Policy & Usage Recommendations(Version-733-and-above)

## Minimum Support Version

[Version 7.24][]

We release new versions of the CLI periodically to introduce new product features, enhancements, and bug fixes. New CLI versions are compatible with existing Coalesce features, but we do not guarantee that older versions of the CLI are compatible with new features.

To avoid issues when connecting to and using Coalesce, we recommend using CLI version 7.17.1 or newer, and actively monitoring and maintaining the versions of your CLI as you use it.

To update the CLI, run:

`npm upgrade -g @coalescesoftware/coa`

[version 7.24]: https://www.npmjs.com/package/@coalescesoftware/coa?activeTab=versions

---

## Troubleshoot the Coalesce CLI

Use this page when a `coa` command or local workflow behaves unexpectedly. Each section names a situation you might hit, then explains what is going on and what to try next.

These patterns come up often while you author pipelines locally or run validation before cloud deploy.

## Database or Schema Missing When You Run Create

You might see messaging that you ran `create` but the database doesn't exist. `coa` doesn't create databases or schemas. Create them in your warehouse first, then map them in `workspaces.yml`.

## Ref Columns Show as UNKNOWN Type

If `ref()` columns show as `UNKNOWN` type, check the following:

1. You're using single quotes inside `ref()`. Double quotes silently break resolution.
2. The target Node name and location are spelled correctly.
3. If you use aggregations through CTEs or JOINs, add explicit casts, for example `MAX(col)::TIMESTAMP_TZ`.

## Validate Shows Column Reference Errors on SQL Nodes

The Column References scanner produces false positives on SQL Nodes when you use [Node Type V2][]. Aliased columns are flagged as missing source columns. These don't block local `create` or `run` but may block `coa plan` in some environments. Run `coa validate --verbose` to see which errors are real.

## Plan Fails Without Showing Specific Errors

When `coa plan` fails with a message like `issues preventing deployment` and the output doesn't show what is wrong, run `coa validate --verbose` separately to see specific errors. Common causes include Column References false positives when you author with [Node Type V2][], or pre-existing reference errors on V1 Nodes. Try a fresh environment. Plan behavior can vary between environments with and without existing deploy state.

## Plan Reports Undefined Parameter Warnings

When `coa plan` lists warnings about undefined parameter references, a Node or Node Type uses a `{{ parameters... }}` path that is not defined for that plan. Warnings do not block writing the plan file or running `coa deploy`. See [Overwriting at Deploy][] for how to fix them.

## V2 Nodes Work in the CLI but Fail in the Coalesce App

When you author with [Node Type V2][], the CLI doesn't enforce `isRequired` config fields from your Node type definition, for example `insertStrategy`. The Coalesce App enforces them. Add config annotations to your `.sql` file, for example `@insertStrategy('INSERT')`. Run `coa describe sql-format` for the full annotation reference.

## UNION ALL Returns Only the First SELECT in a V2 SQL Node

With [Node Type V2][], the SQL parser currently captures only the first source block. `UNION ALL` written as plain SQL is silently dropped. Use the `insertStrategy: UNION ALL` config approach instead of writing `UNION ALL` directly in your SQL.

## Preview SQL Before You Execute

Always use `--dry-run --verbose`, for example `coa create --include "{ NODE }" --dry-run --verbose`. Check that the table name isn't blank and column types aren't `UNKNOWN`.

## Dry-Run Shows CREATE TABLE With No Columns

Your SQL likely has a parse error, such as a typo or missing keyword. The SQL parser for [Node Type V2][] silently produces zero columns when it can't parse the `SELECT`. Check your SQL syntax carefully.

## Validate Reports Error Parsing Repo Files

Check that `data.yml` has `fileVersion: 3`, not `fileVersion: 1`. Also check for malformed `@id` or `@nodeType` annotations in your `.sql` files.

## What's Next?

Continue with the rest of the CLI documentation when you need setup steps, command syntax, local authoring, deploy workflows, or policy detail.

- [Command Line Interface][] for installation, authentication, profiles, and proxy configuration.
- [CLI Commands][] for `coa` command reference and flags.
- [Build Pipelines With the Coalesce CLI][] for workspaces, Nodes, validation, and local runs.
- [Deploy Pipelines to the Coalesce App Using COA][] for plan, deploy, refresh, and managed Environments.
- [CLI Support Policy & Usage Recommendations][] for supported versions and upgrade practices.
- [Node Type V2][] for SQL-first Nodes, annotations, and the editor.

[Command Line Interface]: /docs/coa
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Build Pipelines With the Coalesce CLI]: /docs/coa/version-733-and-above/coa-building-pipelines
[Deploy Pipelines to the Coalesce App Using COA]: /docs/coa/version-733-and-above/coa-deploying-pipelines
[CLI Support Policy & Usage Recommendations]: /docs/coa/version-733-and-above/coa-support-policy-usage-recommendations
[Node Type V2]: /docs/build-your-pipeline/v2-node-types
[Overwriting at Deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy

---

## Command Line Interface(Version-733-and-above)

This page walks you through installing and configuring the Coalesce CLI, `coa`, for Snowflake: system requirements, authentication, access tokens, Environment details, local config, and how source Nodes fit into your workflow.

:::info[Supported Platforms]

Currently Snowflake is the only supported platform for `coa` version 7.33 and above.

:::

## Local Development vs. Cloud Operations

`coa` has two distinct modes of operation, and understanding the difference will help you know which steps in this guide apply to you.

### Local Development

Local development is how you build and iterate on pipelines. You work directly against your warehouse using the files from your version control checkout. You do not need a connection to Coalesce cloud. You edit SQL, preview the generated DDL, create tables, run DML, and check results in your SQL client. This loop is fast because you skip the full plan and deploy cycle entirely while you're still building.

### Cloud Operations

Cloud operations are how you promote a finished pipeline and keep it running. Once your pipeline works the way you want locally, you generate a deployment plan, apply it to a Coalesce Environment, and schedule refreshes through Coalesce cloud. These commands require a Coalesce access token, your domain URL, and an Environment ID.

Most of the setup steps below apply to both modes. Steps that are only needed for cloud operations are marked with a **☁️ Cloud only** label.

## System Requirements

Use the minimum table to confirm your workstation can install and run the CLI. Use the recommended table when you want more headroom for larger Projects or parallel work.

### Minimum System Requirements

| Component | Requirement |
| --- | --- |
| OS | 64-bit Windows 10, macOS 11, Linux kernel version 5.0+ |
| Processor | AWS t2.medium 4X vCPU or equivalent, Apple M1, Intel Core i3-10100 4 Core or better, or AMD Ryzen 3 1200 4 Core or better |
| RAM | 8 GB+ |
| Storage | 1 GB+ free space |
| Network | Access to Snowflake instance; access to Coalesce cloud for cloud operations |
| Node version | v20.x |

### Recommended System Requirements

| Component | Requirement |
| --- | --- |
| OS | 64-bit Windows 10, macOS 11, Linux kernel version 5.0+ |
| Processor | Apple M1 or newer, AWS t2.medium 8X vCPU or equivalent, Intel i3 8X Core or higher, or AMD Ryzen 3 8X Core or better. Choose Apple M1 or newer when you can. |
| RAM | 16 GB+ |
| Storage | 1 GB+ free space |
| Network | Access to Snowflake instance; access to Coalesce cloud for cloud operations |
| Node version | v20.x |

## Before You Begin

What you need depends on how you use the CLI.

- **Local development** - You need your warehouse credentials and a Coalesce Project on your machine. For a new Snowflake project, run `coa init` to generate `workspaces.yml` and related files. See [Build Pipelines With the Coalesce CLI][].

When you deploy pipelines through Coalesce cloud, you need a [Project][], a [Workspace][], and an [Environment][] before you run `coa plan` or `coa deploy`. You can create the Project and Environment in the Coalesce App or with `coa projects create` and `coa environments create`. Then add a [data source][], commit your project to your repo, and continue with deploy.

Existing customers can use an existing repository.

## Installation

Use `npm` to install or update the global CLI package, then confirm the binary is on your PATH:

- To install: `npm install -g @coalescesoftware/coa`
- To update: `npm upgrade -g @coalescesoftware/coa`
- To verify: `coa --version`

## Authentication

The Coalesce CLI supports Snowflake [Key Pair Authentication][] and [Basic Auth][]. You'll also need a Coalesce access token for cloud operations.

### Snowflake

<Tabs>
  <TabItem value="snowflake-key-pair" label="Snowflake Key Pair Authentication" default>

1. Go through the steps in [Key Pair Authentication][] to generate your keys.
2. Save the private key file path. You'll need it for the CLI config and the Coalesce App.
3. Generate your public key using the private key.
4. Assign the public key to your Snowflake user.
5. **☁️ Cloud only** - Add your key pair to your [Environment][].

:::warning[Adding Your Private Key]

When entering your private key, include the full PEM block including the `BEGIN ENCRYPTED PRIVATE KEY` and `END ENCRYPTED PRIVATE KEY` lines. Escape special characters with `\`. For example:

- Original passphrase: `hnw6a#n`
- Escaped passphrase: `hnw6a\#n`

Be sure to escape `#` pound signs and `:` semicolons.

```text
-----BEGIN ENCRYPTED PRIVATE KEY-----
...hnw6a\#n....
-----END ENCRYPTED PRIVATE KEY-----
```

:::

  </TabItem>
  <TabItem value="snowflake-basic-auth" label="Snowflake Basic Authentication">

Basic authentication only requires your Snowflake username and password. This method is not recommended for automation. If you use OAuth to log into Snowflake, you'll need to use Key Pair for the CLI.

1. Follow the instructions in [Snowflake Username and Password][].
2. Save your credentials for the CLI config file.
3. **☁️ Cloud only** - Add the credentials to your [Environment][].

  </TabItem>
</Tabs>

## ☁️ Cloud Only: Coalesce Access Token

You need a Coalesce access token to run `coa plan`, `coa deploy`, `coa refresh`, and any API-style commands. Local development commands (`coa init`, `coa create`, `coa run`, `coa validate`, `coa doctor`) do not require a token.

Go to the **Deploy tab** and click **Generate Access Token**.

If you sign in to the Coalesce App with SSO, plan on generating a new access token for each new Environment you create.

## ☁️ Cloud Only: Get Your Environment ID

Get your Environment ID with `coa environments list`.

## ☁️ Cloud Only: Get Your Domain URL

Your domain is the base URL you use to sign in to Coalesce. The value is the same one you use in your CLI `domain` field. Examples:

- `https://<company-name>.coalescesoftware.io`
- `https://<company-name>.pp.us-east-2.aws.coalescesoftware.io`
- `https://app.eu.coalescesoftware.io`
- `https://<your-private-link-url>`

## Creating Your CLI Config File

Create `~/.coa/config` without a file extension inside a `.coa` folder in your home directory:

```txt
~/
└── .coa/
    └── config
```

For local development, you only need your warehouse credentials. The `token`, `domain`, and `environmentID` fields are only required when you run cloud commands.

<Tabs>
  <TabItem value="keypair" label="Snowflake Key Pair" default>

```txt
[default]
snowflakeAuthType=KeyPair
snowflakeAccount=your-account.region.cloud
snowflakeKeyPairKey=file_path_to_private_key
snowflakeKeyPairPass=private_key_passphrase_if_applicable
snowflakeRole=snowflake_user_role
snowflakeUsername=snowflake_user_name
snowflakeWarehouse=snowflake_warehouse

# ☁️ Cloud only: add these when you're ready to deploy
environmentID=your_environment_id
token=your_access_token
domain=your_domain_url
```

  </TabItem>
  <TabItem value="basic" label="Snowflake Basic">

Not recommended for automation.

```txt
[default]
snowflakeAuthType=Basic
snowflakeAccount=your-account.region.cloud
snowflakeUsername=username
snowflakePassword=password
snowflakeRole=SYSADMIN
snowflakeWarehouse=snowflake_warehouse

# ☁️ Cloud only: add these when you're ready to deploy
environmentID=your_environment_id
token=your_access_token
domain=your_domain_url
```

  </TabItem>
</Tabs>

With warehouse credentials in place, you're ready to build pipelines locally. Add the cloud fields when you're ready to deploy.

## Profiles

Profiles let you run commands against different configurations, for example, a local development warehouse, a staging environment, and production. Pass `--profile <name>` on any command to select a profile. If no profile is specified, `coa` uses `[default]`.

```bash
coa plan --profile snowflakeAdmin
```

The `[default]` profile is required. For cloud operations it must contain `token` and `environmentID`.

```txt title="Example config with multiple profiles"
[default]
token=<Coalesce API Token>
environmentID=<Coalesce ENV ID>
domain=<Coalesce APP URL>

[snowflake-key-pair]
platformKind=Snowflake
snowflakeAuthType=KeyPair
snowflakeAccount=your-account.region.cloud
snowflakeKeyPairKey=<Snowflake Key Pair Key or File Path>
snowflakeRole=<Snowflake Role>
snowflakeUsername=<Snowflake User Name>
snowflakeWarehouse=<Snowflake Warehouse>
environmentID=<Coalesce ENV ID>

[snowflake-basic]
platformKind=Snowflake
snowflakeAuthType=Basic
snowflakeAccount=your-account.region.cloud
snowflakeUsername=<Snowflake Account Username>
snowflakePassword=<Snowflake Account Password>
snowflakeWarehouse=<Snowflake Warehouse>
snowflakeRole=<Snowflake User Role>
environmentID=<Coalesce ENV ID>
```

## Configure HTTP and HTTPS Forward Proxies

The CLI configures its default HTTP client for Coalesce API traffic using standard proxy environment variables when the process starts. Set variables in the same shell or job definition you use to run `coa`, or set them at the machine level if your organization requires it.

### Set `HTTP_PROXY` and `HTTPS_PROXY`

<Tabs groupId="coa-proxy-http-env">
  <TabItem value="windows-cmd" label="Windows Command Prompt" default>

```bat
set HTTP_PROXY=http://your-proxy-address:port
set HTTPS_PROXY=http://your-proxy-address:port
```

  </TabItem>
  <TabItem value="windows-ps" label="Windows PowerShell">

```powershell
$env:HTTP_PROXY = "http://your-proxy-address:port"
$env:HTTPS_PROXY = "http://your-proxy-address:port"
```

  </TabItem>
  <TabItem value="mac-linux" label="macOS and Linux">

```bash
export HTTP_PROXY=http://your-proxy-address:port
export HTTPS_PROXY=http://your-proxy-address:port
```

  </TabItem>
</Tabs>

If specific hosts must bypass the proxy, set `NO_PROXY` to a comma-separated list of host names or CIDR ranges your organization uses for direct routing. Some shells expect the lowercase variable name instead, so use the spelling your environment documents.

### Optional Proxy Credentials

Embed a username and password directly in the proxy URL:

<Tabs groupId="coa-proxy-http-env-credentials">
  <TabItem value="windows-cmd" label="Windows Command Prompt" default>

```bat
set HTTP_PROXY=http://username:password@your-proxy-address:port
set HTTPS_PROXY=http://username:password@your-proxy-address:port
```

  </TabItem>
  <TabItem value="windows-ps" label="Windows PowerShell">

```powershell
$env:HTTP_PROXY = "http://username:password@your-proxy-address:port"
$env:HTTPS_PROXY = "http://username:password@your-proxy-address:port"
```

  </TabItem>
  <TabItem value="mac-linux" label="macOS and Linux">

```bash
export HTTP_PROXY=http://username:password@your-proxy-address:port
export HTTPS_PROXY=http://username:password@your-proxy-address:port
```

  </TabItem>
</Tabs>

Proxy URLs that include passwords can appear in process listings and logs on some systems. Prefer proxy accounts designed for automation, URL-encode special characters in credentials when required, and use secret stores your platform supports.

### Test Connectivity Before You Run `coa`

Confirm HTTPS connectivity through the proxy with `curl`, substituting your Coalesce domain:

```bash
curl -x http://your-proxy-address:port https://your-coalesce-domain.example/
```

If `curl` succeeds but `coa` does not, compare environment variables between the two sessions and review proxy or TLS policies with your network team.

### What Proxy Settings Apply To

Proxy environment variables configure the standard HTTP client the CLI uses for Coalesce API calls. Other libraries or drivers your workload touches may use different networking stacks. Validate full deploy and refresh paths in your own environment when proxies, custom certificate authorities, or TLS inspection are in use.

## Network Connectivity

Which services `coa` connects to depends on the command you run. Treat network requirements as **path-specific**: allow Coalesce cloud endpoints for cloud operations, and allow your warehouse endpoints for all commands that execute SQL.

| Command | Connects to Coalesce Cloud | Connects to Warehouse |
| --- | --- | --- |
| `coa init`, `coa create`, `coa run` | No (most workflows) | Yes |
| `coa validate`, `coa doctor` | No (most workflows) | Yes |
| `coa plan` | Yes | Yes |
| `coa deploy` | Yes | Yes |
| `coa refresh`, `coa rerun`, `coa cancel` | Yes | Yes |
| `coa environments`, `coa nodes`, `coa runs` | Yes | No |
| `coa workspace-nodes` | Yes | No |

---

## What's Next?

Continue with the rest of the CLI documentation when you need workflows, detailed command syntax, or troubleshooting help.

- **Start here: [Build Pipelines With the Coalesce CLI][]** for local Node authoring, workspaces configuration, and validation.
- [CLI Commands][] for `coa` command reference and flags.
- [Deploy Pipelines to the Coalesce App Using COA][] for plan, deploy, refresh, and managed Environments.
- [CLI Support Policy & Usage Recommendations][] for supported versions and upgrade practices.
- [Troubleshoot the Coalesce CLI][] for common errors and diagnostic steps.

---

[Project]: /docs/setup-your-project/create-your-project
[Workspace]: /docs/setup-your-project/create-a-workspace
[Environment]: /docs/setup-your-project/create-your-environments
[data source]: /docs/setup-your-project/add-a-data-source
[Key Pair Authentication]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[Basic Auth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-username-and-password
[Snowflake Username and Password]: /docs/setup-your-project/connection-guides/snowflake/snowflake-username-and-password
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Build Pipelines With the Coalesce CLI]: /docs/coa/version-733-and-above/coa-building-pipelines
[Deploy Pipelines to the Coalesce App Using COA]: /docs/coa/version-733-and-above/coa-deploying-pipelines
[CLI Support Policy & Usage Recommendations]: /docs/coa/version-733-and-above/coa-support-policy-usage-recommendations
[Troubleshoot the Coalesce CLI]: /docs/coa/version-733-and-above/coa-troubleshooting

---

## Copilot Best Practices and Limitations

<!-- vale off -->

This guide covers recommended practices for working with Coalesce Copilot effectively, along with its current limitations. Following these guidelines helps you get the best results while avoiding common pitfalls.

## Copilot Is Evolving

As with any AI-powered tool, Copilot is under active development. We regularly ship improvements based on feedback and testing, but you may occasionally encounter:

- Unexpected Node configurations or SQL outputs.
- Intermittent service interruptions.
- Changes to available capabilities as we refine features.

## Always Review and Verify

:::warning Review Before Deployment
Examine all generated SQL, Node configurations, and transformations before deploying to production. Test changes in a development environment first.
:::

**Before deploying Copilot-generated work, verify:**

- Business logic matches your requirements
- Generated code follows your organization's standards
- Column mappings and transformations are accurate
- Join conditions and relationships are correct
- SQL syntax works in your target data platform
- Node Types match intended use (such as Dimension, Fact, or View)
- Business keys are properly set
- System columns are preserved

### Use Version Control Checkpoints for Safety

- Create a checkpoint before starting Copilot sessions
- Review all changes in version control before deployment
- Roll back easily if Copilot makes unwanted changes

## Write Effective Prompts

**Be specific about requirements:**

| Instead of | Try |
| ------------ | ----- |
| "Create a table" | "Create a dimension table for customers with customer_id as the business key" |
| "Add columns" | "Add columns for first_name, last_name, and email to the dim_customer Node" |

**Break complex tasks into smaller steps:**

Start with basic structure, then add complexity iteratively. For example:

1. First create the base dimension
2. Then add calculated columns
3. Then configure slowly changing dimension logic

**Provide context and examples:**

- Reference existing Nodes: "Create a Node like dim_product but for stores"
- Include domain-specific terminology
- Specify Node Types when relevant (Dimension, Fact, Persistent Stage)
- Include your data platform specifics when needed (Snowflake-specific functions, Databricks syntax)

**Use iterative refinement:**

1. "Create a customer dimension"
2. Review the generated Node
3. "Add loyalty_tier and lifetime_value columns"
4. "Set customer_id as the business key"

## What Copilot Does Well

- Generating boilerplate transformations and staging layers
- Converting SQL queries into Node chains
- Adding columns and updating Node configurations
- Explaining existing Node logic and dependencies
- Building standard Coalesce patterns quickly
- Rapid prototyping and iteration

## Where You Should Be Cautious

:::caution Exercise Extra Care
The following scenarios require additional review and testing:
:::

- Complex multi-node dependencies with intricate business logic
- Migrations from legacy systems (always validate thoroughly)
- Production-critical transformations without testing
- Interpreting domain-specific business rules

## Copilot Limitations

### Workspace-Level Operations

Copilot **cannot**:

- Create source Nodes
- Create or delete entire Subgraphs (it can only add or remove Nodes from existing Subgraphs)
- Modify Workspace settings or configurations
- Manage Environments or deployments

### External Data Operations

Copilot **cannot**:

- Query your data warehouse to preview actual data
- Create new Storage Locations or database connections
- Run actual data transformations (it only builds the DAG)

### Advanced Customization

Copilot **cannot**:

- Set advanced Node configuration options
- Create Custom Node Types
- Modify Package definitions or install new Packages
- Change Workspace-level templates or Macros

### Data and Metadata Awareness

Copilot:

- Knows your table schemas and metadata
- **Doesn't** know the actual data in your tables
- **Can't** automatically detect data quality issues without your description
- **Can't** recommend optimizations based on query performance

## Team Collaboration

- Share Copilot-generated Nodes for peer review
- Document business context that Copilot can't know
- Use Copilot for scaffolding and the Coalesce UI for fine-tuning
- Track time saved on repetitive tasks
- Monitor accuracy of generated transformations
- Identify patterns where Copilot excels or needs guidance

## Security and Data Governance

### Understand Copilot's Scope

| Copilot Can Access | Copilot Cannot Access |
| -------------------- | ---------------------- |
| Your Workspace metadata (Node names, columns, schemas) | Actual data in your tables |
| Your Workspace permissions | External systems or databases directly |
| Your Storage Locations | |

### Avoid Sharing Sensitive Information in Prompts

- Don't paste proprietary business logic that should remain private
- Use generic examples when asking questions about patterns

## Learn With Copilot

Ask questions to deepen your understanding:

- "How should I structure slowly changing dimensions?"
- "What's the difference between Persistent Stage and View for this use case?"
- "Can you explain the join logic you created?"
- "What columns are in sales_fact?"
- "Which Nodes depend on customer_dim?"

## Troubleshooting

If you encounter issues:

- If Copilot stops responding or produces an error, wait 1-2 minutes before trying again
- Start a new conversation if the issue persists. Sometimes a fresh context helps
- Complex requests may take longer to process; simpler, focused prompts often work better
- If results aren't what you expected, rephrase your prompt with more specificity or context
- Report persistent issues to our support team so we can investigate

---

## What's Next?

- [Migrating SQL to Coalesce with Copilot](/docs/guides/migrating-sql-to-coalesce-with-copilot)
- [Coalesce Copilot][]
- [AI Policy][]

---

[AI Policy]: https://coalesce.io/company/ai-policy/
[Coalesce Copilot]: /docs/coalesce-ai/copilot

---

## Coalesce Copilot

<!-- vale off -->

Coalesce Copilot is an AI assistant integrated into the Coalesce platform. It helps you build and modify data transformation pipelines using natural language. Instead of manually configuring Nodes through the Coalesce UI, you can describe your requirements in plain language. Copilot translates them into working DAG Nodes with transformations, columns, and relationships.

Think of it as a proficient Coalesce teammate. You describe your data needs in plain English or paste SQL, and Copilot translates that into actual DAG Nodes with proper transformations, columns, and relationships.

:::note Copilot Is Evolving
As with any AI-powered tool, Copilot is under active development. We regularly ship improvements based on feedback and testing, but you may occasionally encounter unexpected Node configurations, intermittent service interruptions, or changes to available capabilities as we refine features.

Always review generated Nodes and SQL before deployment. If Copilot becomes unresponsive, wait a few moments and start a new conversation in a fresh chat window.
:::

<Wistia id="eb9yzln3gz" />

## Prerequisites

Before using Copilot, ensure you have:

- Access to Coalesce version 7.27 or later
- Active [Workspace permissions][] to create and modify Nodes
- Version control checkpoint created before starting work with Copilot (recommended for easy rollback)
- Available Storage Locations (databases and schemas) configured in your Environment

## Enable Copilot

Org administrators can go to **Org Settings > Preferences** and turn the features on or off for the whole organization. These features are not enabled by default. Reach out to your Coalesce account manager to sign up.

On your build tab, click the AI button .

### Organization-Wide Instructions

Set organization wide settings such as Node and column name standards and other standards.

<DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>Set organization level instructions</figcaption>
    </figure>
    <figure>
    
    <figcaption>Organization instructions will apply to all users</figcaption>
    </figure>

</DocCarousel>

### Project Level Instructions

Set project level settings such as Node and column name standards and other standards.

1. Click on **Project Settings > Copilot Instructions**.
  

:::warning[Copilot Instructors Order]

Organization level and project level work together. If there is a conflict **Project** level instructions take precedence.

:::

## Core Capabilities

Copilot can perform the following operations:

### Build and Modify Nodes

- Create Nodes from natural language descriptions
- Add, remove, or update columns in existing Nodes
- Configure joins and relationships between Nodes
- Set business keys and system columns

### Import SQL

- Convert existing SQL queries into Coalesce DAG Nodes
- Automatically create predecessor Nodes and dependencies
- Parse complex joins and aggregations into proper Node structures

### Answer Questions

- Explain Workspace structure and Node configurations
- Provide guidance on Coalesce best practices
- Describe Node dependencies and column lineage

### Multi-Step Operations

- Execute complex workflows requiring multiple Nodes
- Build complete pipelines from high-level requirements

### Platform Awareness

- Adapts SQL syntax for Snowflake or Databricks
- Understands Coalesce-specific Node Types such as Stage, Dimension, Fact, View, and Persistent Stage
- Respects your Workspace context and existing Nodes

## Operating Modes

Copilot has two operating modes that determine whether it modifies your Workspace or just provides information.

### Edit Mode

Copilot makes actual changes to your Workspace. Use this mode when you want to build or modify Nodes.

**What it does:**

- Creates new Nodes and updates existing ones
- Imports SQL and executes multi-step operations
- Makes actual changes to your Workspace

**Example prompts:**

- "Create a customer dimension with these columns..."
- "Add a lifetime_value column to dim_customer"
- [Paste SQL query]
- "Build a complete customer 360 pipeline..."

### Non-Edit Mode

Copilot provides answers, explanations, and recommendations without modifying your Workspace. This mode is safe for exploration and learning.

**What it does:**

- Answers questions about your Workspace
- Explains configurations and best practices
- Provides guidance without making changes

**Example prompts:**

- "What columns are in the sales_fact table?"
- "How should I structure a slowly changing dimension?"
- "Which Nodes depend on customer_dim?"
- "What's the difference between Persistent Stage and View?"

## Communicating With Copilot

Copilot appears as a chat window in the Coalesce builder. Type your request naturally—no special syntax needed. The conversation is contextual, so you can have back-and-forth exchanges:

> **You:** "Create a customer dimension"
>
> **Copilot:** **Creates dim_customer Node** "I've created a customer dimension table. What columns would you like?"
>
> **You:** "Add customer_id, name, email, and signup_date"
>
> **Copilot:** **Adds the columns** "I've added those columns. Would you like me to set customer_id as the business key?"

## Basic Workflow

1. Create a version control checkpoint before starting with Copilot
2. Open the Copilot chat in the Coalesce builder
3. Type what you want to build or ask a question
4. Review Copilot's actions in your Workspace
5. Refine through follow-up messages if needed
6. Deploy when ready

### Example Prompt Workflows

Copilot adapts to different working styles. Choose the approach that fits your workflow.

<Tabs>
<TabItem value="incremental" label="Start Simple and Add Details">

This approach works well when you're exploring or aren't sure of all requirements upfront.

1. "Create a basic customer dimension"
2. Review the Node Copilot creates
3. "Add loyalty_tier and lifetime_value columns"
4. "Set customer_id as the business key"

</TabItem>
<TabItem value="copy" label="Copy Existing Structures">

Use this when you want consistency across similar Nodes or need to replicate a pattern.

1. "Create a Node like dim_product but for stores"
2. Copilot copies the structure
3. "Update the columns to match store attributes"

</TabItem>
<TabItem value="goal" label="Describe Your End Goal">

This is fastest when you know exactly what you need and want Copilot to figure out the implementation.

1. "I need to calculate monthly recurring revenue by customer segment"
2. Copilot builds the entire pipeline
3. Review and approve or request changes

</TabItem>
<TabItem value="sql" label="Import Existing SQL">

Use this to migrate existing queries or logic into Coalesce.

1. Paste your existing SQL query into the chat
2. Copilot analyzes and creates necessary Nodes
3. Review the generated DAG structure
4. Adjust join logic or transformations as needed

</TabItem>
</Tabs>

### Example 1: Creating a Dimension Node

**Prompt:**

"Create a dimension table for customers with customer_id as the business key. Include columns for customer_id, first_name, last_name, email, and signup_date."

**What Copilot does:**

- Creates a Dimension Node Type
- Adds specified columns with appropriate data types
- Sets customer_id as the business key
- Configures necessary predecessor relationships

### Example 2: Importing SQL

You can add SQL directly into Copilot.

**Prompt:**

```sql
SELECT 
  c.customer_id,
  c.name,
  SUM(o.amount) as total_spent
FROM customers c
JOIN orders o ON c.id = o.customer_id
GROUP BY c.customer_id, c.name
```

**What Copilot does:**

- Analyzes the SQL query
- Creates predecessor Nodes for customers and orders
- Builds a transformation Node with the join
- Sets up the aggregation logic
- Configures column mappings

### Example 3: Multi-Step Pipeline

**Prompt:**

"Create a complete customer 360 view starting with raw customer data, add their order history, calculate lifetime value, and join with marketing campaign responses."

**What Copilot does:**

- Creates staging Nodes for raw data sources
- Builds dimension tables for customers
- Creates fact tables for orders and campaign responses
- Adds calculated columns for lifetime value
- Wires all Nodes together with proper relationships

### Example 4: Modifying Existing Nodes

**Prompt:**

"Add a column called lifetime_value to dim_customer that calculates the sum of all order amounts for each customer."

**What Copilot does:**

- Finds the dim_customer Node
- Adds the new column
- Writes the aggregation logic
- References the appropriate predecessor Node for order data

## What Copilot Knows About Your Workspace

### Copilot Has Awareness Of

- **Your current Workspace:** All existing Nodes, their columns, and relationships
- **Your data platform:** Whether you're using Snowflake or Databricks, and adapts SQL syntax accordingly
- **Available Node Types:** How to build Dimension, Fact, View, and Persistent Stage
- **Your Storage Locations:** Which databases and schemas are available
- **Your viewing context:** Understands what you're currently looking at

### What Copilot Doesn't Know

- Copilot doesn't see the actual data in your tables. It only sees metadata like Node names, columns, and schemas
- Data quality issues unless you describe them
- Query performance metrics or optimization opportunities based on runtime data

### Getting Better Results

- Keep relevant Nodes visible in your Workspace
- Reference specific Node names using clear identifiers
- Mention target Subgraphs when organizing Nodes
- Ask questions about existing Nodes: "What columns are in sales_fact?"
- Request explanations: "Why did you choose a Persistent Stage instead of a View?"
- Get recommendations: "What's the best Node Type for aggregated metrics?"

## When to Use Copilot Versus the Coalesce UI

| Use Copilot When | Use the Coalesce UI When |
| ------------------ | -------------------------- |
| Building new Nodes from scratch | Fine-tuning individual column transformations with autocomplete |
| Converting existing SQL to Coalesce | Visual exploration of complex DAGs |
| Making bulk updates to multiple Nodes | Precise drag-and-drop column mapping |
| Learning Coalesce or exploring new features | Detailed configuration of advanced Node properties |
| Rapid prototyping and iteration | You prefer visual interfaces over conversation |
| You have clear requirements and want to save time | Making small tweaks where clicking is faster than typing |
| Working with standard Coalesce patterns | |

:::tip Best Approach
Use both. Let Copilot build the structure quickly, then use the Coalesce UI for fine-tuning and visual validation. Teams often describe it as "Copilot for speed, Coalesce UI for polish."
:::

---

## What's Next?

- [Migrating SQL to Coalesce with Copilot](/docs/guides/migrating-sql-to-coalesce-with-copilot)
- [Copilot Best Practices and Limitations](/docs/coalesce-ai/copilot/copilot-best-practices)
- [AI Policy][]

---

[Workspace permissions]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[AI Policy]: https://coalesce.io/company/ai-policy/

---

## Transform AI

## Coalesce Copilot

Coalesce Copilot is an AI assistant that accelerates data pipeline development through natural language interaction. Instead of manually configuring nodes through the UI, simply describe your requirements in plain English or paste existing SQL—Copilot translates your intent into working DAG nodes with proper transformations, columns, and relationships.
This intelligent assistant empowers teams to:

* Build and modify nodes through conversational commands.
* Import existing SQL queries into structured Coalesce pipelines.
* Execute multi-step workflows from high-level requirements.
* Receive guidance on best practices and workspace configurations.

By combining the speed of AI-generated scaffolding with the precision of Coalesce's visual interface, Copilot enables both experienced developers and SQL users to build production-ready data transformations faster than ever before.

<ButtonLink to="/docs/coalesce-ai/copilot/">
  Learn more about Coalesce Copilot
</ButtonLink>

## Intelligent Documentation Assistant

Coalesce's AI-powered documentation feature revolutionizes how teams document their data transformations. By analyzing lineage and transformation metadata, our AI automatically generates natural language descriptions for:

* Nodes and columns within your data pipeline.
* Git commits and changes.

This intelligent assistant significantly reduces documentation overhead while ensuring consistency and clarity across your data projects. Data stakeholders can now easily understand the business context and transformation logic without manual documentation effort.

<ButtonLink to="/docs/coalesce-ai/intelligent-documentation">
  Learn more about the Intelligent Documentation Assistant
</ButtonLink>

<Wistia id="uv6ghzm3xx" subtitle="Simplify documentation using Coalesce's AI features"/>

## Seamless Snowflake Cortex Integration

Coalesce integrates natively with Snowflake Cortex, bringing [machine learning capabilities][] directly to your data pipeline. This integration:

* Enables SQL users to leverage ML functions without specialized expertise.
* Facilitates rapid development through pattern-based approaches.
* Provides access to Snowflake's fully managed AI services.
* Streamlines the implementation of ML functionality.

<ButtonLink to="/docs/marketplace/package/coalesce_snowflake_cortex">
  Learn more about the Snowflake Cortex
</ButtonLink>

<Wistia id="q34plnfnfi" subtitle="Operationalize Snowflake Cortex LLM Functions With Coalesce"/>

## Transform MCP

Connect MCP-capable AI assistants to your Coalesce account through the hosted Transform Model Context Protocol server. Your assistant can list **Projects** and **Environments**, inspect runs and **Nodes**, and trigger refreshes within your existing permissions. No local install is required.

<ButtonLink to="/docs/coalesce-ai/mcp/about-mcp">
  Learn more about Transform MCP
</ButtonLink>

## Enterprise-Scale AI Operations

Successfully deploying AI and ML models across an organization requires robust infrastructure and standardized processes. Coalesce delivers:

* Automated pipeline development and maintenance.
* Collaborative development environments.
* Enterprise-grade reliability and performance.
* Comprehensive troubleshooting capabilities.

Our automation capabilities ensure your AI initiatives can scale effectively while maintaining consistency and reliability across all data projects.

---

## What's Next?

* [About Transform MCP](/docs/coalesce-ai/mcp/about-mcp)
* [Migrating SQL to Coalesce with Copilot](/docs/guides/migrating-sql-to-coalesce-with-copilot)
* [AI Policy][]

---

[machine learning capabilities]:/docs/marketplace/package/coalesce_snowflake_cortex
[AI Policy]: https://coalesce.io/company/ai-policy/

---

## Intelligent Documentation Assistant

Coalesce's AI Documentation Assistant automatically generates clear, consistent documentation for your data workflows, including node descriptions, column explanations, and Git commit messages, to save time and improve clarity across your team.

## Enabling Intelligent Documentation Assistant

Organization administrators can go to **Org Settings > Preferences** and turn the features on or off for the whole organizations. These features are not enabled by default, reach out to your Coalesce account manager to sign up.

## How To Use Coalesce's Intelligent Documentation Assistant

To find AI enabled featured in Coalesce, look for the .

* **Column Descriptions** - In the Node Editor, go to the Description heading.
* **Node Description** - In the Node Editor, go to the Node description.
* **Git Commit** - Create a Git commit and look for it in the commit message box.

:::info[AI Column Descriptions Node Limitation]

Coalesce only supports Nodes with 60 columns or less for AI column descriptions.

:::

---

## About Transform MCP

The Transform Model Context Protocol, or MCP, server lets MCP-capable AI clients operate your Coalesce **Environments**, **Projects**, **Runs**, and **Nodes** through your existing Coalesce account. Point your client at your Coalesce App URL, authenticate with your API, and the Transform MCP assistant can list metadata, manage **Environments** and **Projects**, inspect runs, and trigger refreshes.

This section covers hosted setup, available tools, client integrations, and troubleshooting. For Catalog search and lineage through MCP, see [MCP Integration][].

## When to Use Transform MCP

Transform MCP is the right choice when:

- You want AI assistants to operate Coalesce **Environments**, **Projects**, runs, and **Nodes** without installing software locally.
- You do not need MCP to deploy **Workspaces** or author **Node** SQL. Those workflows stay in COA and the Coalesce App.

## What Transform MCP Is

Transform MCP is a hosted, stateless server run by Coalesce on your regional app host. It is a thin adapter over the Coalesce cloud API: it holds no project graph, cache, session store, or warehouse secrets. Every action runs as you, scoped to your organization and existing Coalesce permissions.

- **Transport** - HTTP at `POST https://<your-app-host>/api/v1/mcp`. Each tool call is one stateless request.
- **Authentication** - `Authorization: Bearer <refreshToken>`. This is the same credential as COA and the **Deploy** tab access token.
- **Tenancy** - Resolved from your token. Use the app host you sign in to, including vanity SSO subdomains.

## What Transform MCP Does Not Do

Transform MCP does not replace the Coalesce App or COA for these workflows:

- **Deploy** - Deploying a **Workspace** stays in COA or the Coalesce App.
- **Authoring** - Editing **Node** SQL, subgraphs, or **Job** definitions stays in COA or the Coalesce App.
- **Warehouse credentials** - Refreshes use the **Workspace** stored connection. You do not pass Snowflake or other warehouse secrets through MCP.

For pipeline authoring with AI inside the Coalesce App, see [Coalesce Copilot][].

## Get Started

Use this table to pick the right starting page for your goal.

| Goal | Start here |
| --- | --- |
| Connect an AI assistant to a Coalesce account | [Connect to Transform MCP][] |
| Configure the endpoint URL, token, and regional hosts in detail | [Configure Transform MCP][] |
| See every tool name, input, and REST equivalent | [Transform MCP Available Tools][] |
| Find **Environment**, **Project**, run, and **Node** IDs | [Find Coalesce IDs][] |
| Set up Claude Desktop or Claude Code | [Integrate Claude with Transform MCP][] |
| Set up Cursor | [Integrate Cursor with Transform MCP][] |
| Set up VS Code | [Integrate VS Code with Transform MCP][] |

## Capabilities

Transform MCP exposes 21 tools across five areas:

| Area | Tools | What you can do |
| --- | --- | --- |
| Environments | `environments_list`, `environments_get`, `environments_create`, `environments_update`, `environments_delete` | List, inspect, create, update, and delete deployment **Environments** |
| Projects | `projects_list`, `projects_get`, `projects_create`, `projects_update`, `projects_delete` | List, inspect, create, update, and delete **Projects** |
| Runs, read-only | `runs_list`, `runs_get`, `runs_results` | Inspect run history and per-run results |
| Nodes, read-only | `nodes_list_environment`, `nodes_get_environment`, `nodes_list_workspace`, `nodes_get_workspace` | Inspect deployed and in-progress **Nodes** |
| Run actions | `runs_start`, `runs_retry`, `runs_cancel`, `runs_status` | Start, retry, cancel, and poll refresh runs |

Read and CRUD tools apply immediately. Run action tools trigger real warehouse work on the target **Environment**.

For the full tool reference with parameters, see [Transform MCP Available Tools][].

## Authentication

Transform MCP accepts your Coalesce **refresh token** in the `Authorization: Bearer` header. This is the same credential COA stores after `coa login` and the token shown on the **Deploy** tab.

1. Open the **Deploy** tab in the Coalesce App and copy your access token, or read the `token=` value from your active COA profile in `~/.coa/config`.
2. Configure your MCP client with `Authorization: Bearer <your-token>`.
3. Set the server URL to `https://<your-app-host>/api/v1/mcp`, where `<your-app-host>` is the host you use to sign in.

See [Getting Your Token][] and [Configure Transform MCP][] for regional hosts and SSO vanity subdomains.

## MCP Client Integrations

Step-by-step configuration for common clients:

- [Integrate Claude with Transform MCP][]
- [Integrate Cursor with Transform MCP][]
- [Integrate VS Code with Transform MCP][]

## Data Handling

Transform MCP does not store production data or run results. It reads metadata and triggers actions against the Coalesce API in real time when a tool is called. Your organization's data retention policies govern how long runs and artifacts remain available in Coalesce.

## What's Next?

- [Connect to Transform MCP][]
- [Transform MCP Available Tools][]
- [Transform MCP Troubleshooting][]

[MCP Integration]: /docs/catalog/developer/mcp-integration
[Coalesce Copilot]: /docs/coalesce-ai/copilot/
[Connect to Transform MCP]: /docs/coalesce-ai/mcp/mcp-quickstart
[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Getting Your Token]: /docs/api/authentication
[Find Coalesce IDs]: /docs/reference/whats-my-id
[Integrate Claude with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-claude
[Integrate Cursor with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-cursor
[Integrate VS Code with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-vscode
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting

---

## Configure Transform MCP

Transform MCP runs on your Coalesce regional app host. You connect over HTTP with a bearer refresh token. No local MCP package install is required.

For a minimal first connection, see [Connect to Transform MCP][].

## Endpoint URL

Transform MCP accepts stateless HTTP `POST` requests at a single path on your app host.

| Component | Value |
| --- | --- |
| Method | `POST`, HTTP, stateless |
| Path | `/api/v1/mcp` |
| Full URL | `https://<your-app-host>/api/v1/mcp` |

Use `<your-app-host>` from the host you sign in to. This matches how COA and the REST API target your deployment.

### Regional and SSO Hosts

Coalesce offers multiple regional app hosts. Use the base URL that matches your deployment. See the **Base URLs** table in [Getting Your Token][].

If you use SSO, prefix the regional host with your company subdomain. For example, for Europe primary:

```text
https://mycompany.app.eu.coalescesoftware.io/api/v1/mcp
```

Cross-region isolation is enforced at authentication: a token minted in one region cannot be used against another region's app host.

## Authentication

Transform MCP uses **token-based authentication only**. OAuth is not supported for this server.

### Required Header

| Header | Required | Description |
| --- | --- | --- |
| `Authorization` | Required | `Bearer <refreshToken>`: your Coalesce refresh token |

No environment ID or user ID header is required. Your organization and permissions are resolved server-side from the verified token.

### Obtain a Refresh Token

1. Sign in to the Coalesce App.
2. Open the **Deploy** tab and copy your access token, or generate a new one.
3. Alternatively, read the `token=` value from your active COA profile in `~/.coa/config`.

Treat the token as a secret. Do not commit it to version control or store it in a Coalesce **Environment** variable.

### Token Lifecycle

Access tokens follow the same lifecycle as REST API tokens documented in [Getting Your Token][]. Tokens remain valid until your account is deleted, disabled, or you change your password or email, or similar account events occur.

## Permission Model

Transform MCP does not add a separate permission layer. The server can only perform actions your Coalesce account can perform.

- Read tools return resources your account can access.
- `environments_create`, `projects_update`, and similar CRUD tools require the same rights you would need in the Coalesce App or REST API.
- `runs_start`, `runs_retry`, and `runs_cancel` trigger real warehouse work and require run permissions on the target **Environment**.

## Reads and Writes

Tool calls fall into three behavior groups depending on whether they read metadata or trigger warehouse work.

| Tool type | Behavior |
| --- | --- |
| List and get tools | Read metadata immediately; no warehouse compute |
| Environment and Project CRUD | Apply changes through the Coalesce API |
| Run action tools | Trigger refresh or DML runs using the **Workspace** stored connection |

## Full-Environment Refresh Guardrail

`runs_start` refreshes every **Node** in an **Environment** when you omit both `jobID` and `includeNodesSelector`. The tool requires an explicit confirmation:

- Pass `jobID` or `includeNodesSelector` to scope the run, **or**
- Set `confirmRunAllNodes: true` to confirm an intentional full-environment refresh.

`excludeNodesSelector` alone does not scope the run. Excluding nodes from an otherwise full refresh still requires `confirmRunAllNodes: true`.

## Test Your Configuration

After configuring your client, ask a read-only question such as:

> List environments in this Coalesce organization.

Then try a scoped run only if you intend to trigger warehouse work:

> Start a refresh for job `<job-id>` in environment `<environment-id>`.

## Related Docs

These pages cover setup, tool reference, and client-specific configuration.

- [Connect to Transform MCP][]
- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]
- [Integrate Claude with Transform MCP][]
- [Integrate Cursor with Transform MCP][]
- [Integrate VS Code with Transform MCP][]

## What's Next?

- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]
- [Transform MCP Troubleshooting][]

[Connect to Transform MCP]: /docs/coalesce-ai/mcp/mcp-quickstart
[Getting Your Token]: /docs/api/authentication
[Integrate VS Code with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-vscode
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Find Coalesce IDs]: /docs/reference/whats-my-id
[Integrate Claude with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-claude
[Integrate Cursor with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-cursor
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting

---

## Integrate Claude with Transform MCP

Claude Desktop and Claude Code can connect to the hosted Transform MCP server over HTTP. After setup, Claude can list **Projects**, inspect runs, and perform other actions your Coalesce account allows.

For token and endpoint details, see [Configure Transform MCP][].

## Prerequisites

You need:

- A Coalesce refresh token from the **Deploy** tab or COA `~/.coa/config`
- Your Coalesce App host URL, the same host you sign in to
- Claude Desktop or Claude Code with MCP support

Transform MCP uses token authentication only. OAuth is not supported.

## Claude Desktop

Claude Desktop reads MCP servers from `claude_desktop_config.json`.

### Open the Configuration File

1. In Claude Desktop, go to **Settings**.
2. Select the **Developer** tab.
3. Click **Edit Config**.

File locations:

- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
- **Linux:** `~/.config/Claude/claude_desktop_config.json`

### Add Transform MCP

Add a `coalesce-transform` entry under `mcpServers`. Replace `<your-app-host>` and `<your-refresh-token>`.

```json title="Claude Desktop configuration"
{
  "mcpServers": {
    "coalesce-transform": {
      "type": "http",
      "url": "https://<your-app-host>/api/v1/mcp",
      "headers": {
        "Authorization": "Bearer <your-refresh-token>"
      }
    }
  }
}
```

### Restart and Verify

1. Save the file and quit Claude Desktop completely.
2. Reopen Claude Desktop. An MCP indicator should appear when the server connects.
3. Ask: *List Coalesce projects in this organization.*

## Claude Code

Claude Code reads MCP servers from `.mcp.json` at your project root, or from user-level config when you use the CLI.

### Option 1: CLI Registration

With Claude Code open, run:

```bash
claude mcp add --transport http coalesce-transform "https://<your-app-host>/api/v1/mcp" --header "Authorization: Bearer <your-refresh-token>"
```

### Option 2: Project `.mcp.json`

Create or edit `.mcp.json` at your repository root:

```json title="Claude Code project configuration"
{
  "mcpServers": {
    "coalesce-transform": {
      "type": "http",
      "url": "https://<your-app-host>/api/v1/mcp",
      "headers": {
        "Authorization": "Bearer <your-refresh-token>"
      }
    }
  }
}
```

Claude Code picks up `.mcp.json` on startup. Ask a Coalesce question to confirm the connection.

:::info[Project Versus User Config]

The `claude mcp add` command may write to user-level config at `~/.claude.json` rather than the project `.mcp.json`. For team-shared setup, commit `.mcp.json` without secrets and inject the token from your environment.

:::

## Troubleshooting

| Symptom | What to check |
| --- | --- |
| Authentication error | Re-copy the token from the **Deploy** tab. Confirm the header is `Bearer <token>`. |
| Server not listed | Restart Claude after editing the config file. |
| Tools not called | Ask explicitly: *Use the coalesce-transform MCP server to list environments.* |

For more errors, see [Transform MCP Troubleshooting][].

## What's Next?

- [Integrate Cursor with Transform MCP][]
- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]

[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting
[Integrate Cursor with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-cursor
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Find Coalesce IDs]: /docs/reference/whats-my-id

---

## Integrate Cursor with Transform MCP

Cursor can connect to the hosted Transform MCP server so Agent can list **Projects**, inspect runs, and call other tools within your Coalesce permissions.

For token and endpoint details, see [Configure Transform MCP][].

## Prerequisites

You need:

- A Coalesce refresh token from the **Deploy** tab or COA `~/.coa/config`
- Your Coalesce App host URL
- Cursor with MCP support enabled

## Configure Transform MCP in Cursor

### Step 1: Open MCP Settings

In Cursor, go to **Settings** > **Tools & MCP** > **New MCP server**.

### Step 2: Edit `mcp.json`

Create or edit `~/.cursor/mcp.json`. Add a `coalesce-transform` entry. Replace `<your-app-host>` and `<your-refresh-token>`.

```json title="Cursor MCP configuration"
{
  "mcpServers": {
    "coalesce-transform": {
      "url": "https://<your-app-host>/api/v1/mcp",
      "headers": {
        "Authorization": "Bearer <your-refresh-token>"
      }
    }
  }
}
```

### Step 3: Restart Cursor

Restart Cursor so it loads the new MCP configuration.

### Step 4: Verify in Agent

Open Cursor Agent and ask:

> List Coalesce projects using Transform MCP.

You should see **Projects** from your organization. If the assistant answers from general knowledge instead of calling tools, name the server explicitly in your prompt.

## Project-Level Configuration

Some teams commit a project-level `.cursor/mcp.json` for shared server URLs. Do not commit refresh tokens. Use environment-specific secrets or local overrides for the `Authorization` header.

## Troubleshooting

| Symptom | What to check |
| --- | --- |
| Failed to connect | Confirm the URL is `https://<your-app-host>/api/v1/mcp` with the host you sign in to. |
| 401 Unauthorized | Regenerate or re-copy the token from the **Deploy** tab. |
| Tools not used | Reference the server by name: *Search Coalesce Transform MCP for…* |

For more errors, see [Transform MCP Troubleshooting][].

## What's Next?

- [Integrate VS Code with Transform MCP][]
- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]

[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting
[Integrate VS Code with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-vscode
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Find Coalesce IDs]: /docs/reference/whats-my-id

---

## Integrate VS Code with Transform MCP

Visual Studio Code supports MCP servers through workspace or user `mcp.json` configuration. After setup, your AI assistant in VS Code can call Transform MCP tools against your Coalesce account.

For token and endpoint details, see [Configure Transform MCP][].

## Prerequisites

You need:

- A Coalesce refresh token from the **Deploy** tab or COA `~/.coa/config`
- Your Coalesce App host URL
- VS Code with MCP support through an MCP-compatible extension

## Configure Transform MCP

### Step 1: Open MCP Configuration

1. Open the Command Palette with **Cmd+Shift+P** on macOS or **Ctrl+Shift+P** on Windows.
2. Run **MCP: Open Workspace Folder MCP Configuration**. If your extension offers a user-level command instead, use that equivalent.

### Step 2: Add the Server

VS Code uses the `servers` key, not `mcpServers`. Replace `<your-app-host>` and `<your-refresh-token>`.

```json title="VS Code MCP configuration"
{
  "servers": {
    "coalesce-transform": {
      "type": "http",
      "url": "https://<your-app-host>/api/v1/mcp",
      "headers": {
        "Authorization": "Bearer <your-refresh-token>"
      }
    }
  }
}
```

### Step 3: Reload and Verify

Reload the VS Code window if prompted. Ask your assistant:

> List Coalesce environments in this organization.

## Configuration Notes

- Use `https://<your-app-host>/api/v1/mcp` as the full URL.
- The `Authorization` header must be `Bearer <your-refresh-token>`.
- No additional Coalesce headers are required.

Exact MCP UI and command names depend on your VS Code MCP extension. Refer to your extension documentation if the palette command differs.

## Troubleshooting

| Symptom | What to check |
| --- | --- |
| Server not found | Confirm you used `servers`, not `mcpServers`. |
| Authentication failed | Re-copy the token from the **Deploy** tab. |
| Wrong host | Use the same app host you sign in to, including SSO vanity subdomains. |

For more errors, see [Transform MCP Troubleshooting][].

## What's Next?

- [Integrate Claude with Transform MCP][]
- [Integrate Cursor with Transform MCP][]
- [Transform MCP Available Tools][]

[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting
[Integrate Claude with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-claude
[Integrate Cursor with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-cursor
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools

---

## Transform MCP Available Tools

Transform MCP exposes 21 tools grouped into five families. Tool names use underscores, for example `environments_list`, to satisfy MCP client naming rules.

List tools return a `{ data, next, limit }` envelope where pagination applies. Pass `startingFrom` from a previous response's `next` field to fetch the next page.

For setup, see [Connect to Transform MCP][]. For ID formats, see [Find Coalesce IDs][].

## Environment Tools

Manage deployment **Environments** in your organization.

| Tool | Purpose | REST equivalent |
| --- | --- | --- |
| `environments_list` | List **Environments** with optional pagination | [Get Environments][] |
| `environments_get` | Get one **Environment** by ID | [Get Environment][] |
| `environments_create` | Create a new **Environment** | [Create Environment][] |
| `environments_update` | Partially update an **Environment** | [Update Environment][] |
| `environments_delete` | Delete an **Environment** | [Delete Environment][] |

### `environments_list`

List deployment **Environments** in your organization.

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `detail` | boolean | No | Include full detail, including **Storage Locations** and health, rather than summaries |
| `limit` | integer | No | Page size, 1–500 (default 100) |
| `startingFrom` | string | No | Pagination cursor from a previous response's `next` |
| `project` | string | No | Filter to a single **Project** ID |

### `environments_get`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | The **Environment** ID |

### `environments_create`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | string | Yes | **Environment** name |
| `project` | string | Yes | **Project** ID the **Environment** belongs to |
| `oauthEnabled` | boolean | Yes | Whether OAuth is enabled for the **Environment** connection |
| `description` | string | No | **Environment** description |
| `connectionAccount` | string | No | Account name for the **Environment** connection |
| `accessUrl` | string | No | Optional access URL for the **Environment** connection |
| `defaultStorageMapping` | string | No | Default **Storage Mapping** name |
| `mappings` | object | No | **Storage Location** mappings keyed by location name |

### `environments_update`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | The **Environment** ID to update |
| `name` | string | No | **Environment** name |
| `description` | string | No | **Environment** description |
| `connectionAccount` | string | No | Account name for the **Environment** connection |
| `accessUrl` | string | No | Access URL for the **Environment** connection |
| `defaultStorageMapping` | string | No | Default **Storage Mapping** name |
| `oauthEnabled` | boolean | No | Whether OAuth is enabled |
| `mappings` | object | No | **Storage Location** mappings |

### `environments_delete`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | The **Environment** ID to delete |

## Project Tools

Manage **Projects**, which group **Environments** and **Workspaces**.

| Tool | Purpose | REST equivalent |
| --- | --- | --- |
| `projects_list` | List **Projects** in your org | [Get Projects in Org][] |
| `projects_get` | Get one **Project** by ID | [Get Project in Org][] |
| `projects_create` | Create a new **Project** | [Create Project][] |
| `projects_update` | Partially update a **Project** | [Update Project][] |
| `projects_delete` | Delete a **Project** | [Delete Project][] |

### `projects_list`

`projects_list` is not paginated upstream. The response always fits on one page. The `next` field is `null`.

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `includeWorkspaces` | boolean | No | Include nested **Workspace** data for all **Projects** |
| `includeJobs` | boolean | No | Include nested **Job** data for all **Workspaces** in the **Projects** |

### `projects_get`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `projectID` | string | Yes | The **Project** ID |
| `includeWorkspaces` | boolean | No | Include nested **Workspace** data |
| `includeJobs` | boolean | No | Include nested **Job** data for all **Workspaces** |

### `projects_create`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `platformKind` | string | Yes | `snowflake`, `databricks`, `bigquery`, or `fabric` |
| `name` | string | Yes | **Project** name |
| `description` | string | No | **Project** description |
| `gitRepoURL` | string | No | Git repository URL |
| `autoPreviewEnabled` | boolean | No | Whether sample data is fetched automatically after running a **Node** |

### `projects_update`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `projectID` | string | Yes | The **Project** ID to update |
| `platformKind` | string | No | `snowflake`, `databricks`, `bigquery`, or `fabric` |
| `name` | string | No | **Project** name |
| `description` | string | No | **Project** description |
| `gitRepoURL` | string | No | Git repository URL |
| `autoPreviewEnabled` | boolean | No | Whether sample data is fetched automatically after running a **Node** |

### `projects_delete`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `projectID` | string | Yes | The **Project** ID to delete |

## Run Read Tools

Inspect run history and results.

| Tool | Purpose | REST equivalent |
| --- | --- | --- |
| `runs_list` | List runs with filters and pagination | [Get Runs][] |
| `runs_get` | Get one run by numeric run ID | [Get Run][] |
| `runs_results` | Get per-run results for a deploy or refresh | [Get Run Results][] |

### `runs_list`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `limit` | integer | No | Maximum runs to return, 1–1000 |
| `startingFrom` | number or string | No | Pagination cursor from a previous response's `next` |
| `orderBy` | string | No | Field to sort by. Set explicitly when using `startingFrom` |
| `orderByDirection` | string | No | Sort order |
| `projectID` | string | No | Filter to a single **Project** ID |
| `runType` | string | No | Filter by run type |
| `runStatus` | string | No | Filter by run status |
| `environmentID` | string | No | Filter to a single **Environment** ID |
| `detail` | boolean | No | Include full run detail rather than summaries |

### `runs_get` and `runs_results`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `runID` | integer | Yes | Numeric run ID. The scheduler API calls this value `runCounter`. |

:::warning[Run ID is numeric]

`runID` must be the numeric run counter, for example `42`, not the UUID in a run URL. See [Find Coalesce IDs][].

:::

## Node Read Tools

Inspect **Nodes** in an **Environment** or **Workspace**.

| Tool | Purpose | REST equivalent |
| --- | --- | --- |
| `nodes_list_environment` | List **Nodes** deployed in an **Environment** | [Get Nodes][] |
| `nodes_get_environment` | Get one **Environment** **Node** | [Get Node][] |
| `nodes_list_workspace` | List **Nodes** in a **Workspace** | [Get Workspace Nodes][] |
| `nodes_get_workspace` | Get one **Workspace** **Node** | [Get Workspace Node][] |

### `nodes_list_environment`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | The **Environment** ID |
| `detail` | boolean | No | Include full **Node** detail rather than summaries |
| `limit` | integer | No | Page size, 1–500 (default 100) |
| `startingFrom` | string | No | Pagination cursor from a previous response's `next` |
| `skipParsing` | boolean | No | Skip parsing column references and updating sources |

### `nodes_get_environment`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | The **Environment** ID |
| `nodeID` | string | Yes | The **Node** ID |
| `skipParsing` | boolean | No | Skip parsing column references and updating sources |

### `nodes_list_workspace`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `workspaceID` | string | Yes | The **Workspace** ID |
| `detail` | boolean | No | Include full **Node** detail rather than summaries |
| `limit` | integer | No | Page size, 1–500 (default 100) |
| `startingFrom` | string | No | Pagination cursor from a previous response's `next` |
| `skipParsing` | boolean | No | Skip parsing column references and updating sources |

### `nodes_get_workspace`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `workspaceID` | string | Yes | The **Workspace** ID |
| `nodeID` | string | Yes | The **Node** ID |
| `skipParsing` | boolean | No | Skip parsing column references and updating sources |

## Run Action Tools

Trigger and manage refresh runs. These tools call the scheduler API and can start warehouse compute.

| Tool | Purpose | REST equivalent |
| --- | --- | --- |
| `runs_start` | Start a refresh run | [Start Run][] |
| `runs_retry` | Re-run a previous run | [Retry Failed Run][] |
| `runs_cancel` | Cancel an in-progress run | [Cancel Run][] |
| `runs_status` | Poll run status | [Run Status][] |

### `runs_start`

Trigger a refresh or DML run on a deployed **Environment** using the **Workspace** stored connection.

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `environmentID` | string | Yes | Target **Environment** ID |
| `jobID` | string | No | **Job** to scope the refresh to |
| `includeNodesSelector` | string | No | Selector for **Nodes** to include |
| `excludeNodesSelector` | string | No | Selector for **Nodes** to exclude |
| `parallelism` | integer | No | Number of **Nodes** to process in parallel |
| `forceIgnoreWorkspaceStatus` | boolean | No | Run even if **Workspace** status would normally block it |
| `confirmRunAllNodes` | boolean | No | Must be `true` when neither `jobID` nor `includeNodesSelector` is set |

### `runs_retry`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `runID` | integer | Yes | Numeric run ID to re-run |
| `forceIgnoreWorkspaceStatus` | boolean | No | Re-run even if **Workspace** status would normally block it |

### `runs_cancel`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `runID` | integer | Yes | Numeric run ID to cancel |
| `environmentID` | string | Yes | **Environment** ID the run belongs to |
| `runType` | string | No | Type of run to cancel |

### `runs_status`

| Input | Type | Required | Description |
| --- | --- | --- | --- |
| `runID` | integer | Yes | Numeric run ID to look up |

Terminal statuses are `completed`, `failed`, and `canceled`. Non-terminal statuses are `waitingToRun` and `running`.

## What's Next?

- [Find Coalesce IDs][]
- [Configure Transform MCP][]
- [Transform MCP Troubleshooting][]

[Connect to Transform MCP]: /docs/coalesce-ai/mcp/mcp-quickstart
[Find Coalesce IDs]: /docs/reference/whats-my-id
[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting
[Get Environments]: /docs/api/coalesce/get-environments
[Get Environment]: /docs/api/coalesce/get-environment
[Create Environment]: /docs/api/coalesce/create-environment
[Update Environment]: /docs/api/coalesce/update-environment
[Delete Environment]: /docs/api/coalesce/delete-environment
[Get Projects in Org]: /docs/api/coalesce/get-projects-in-org
[Get Project in Org]: /docs/api/coalesce/get-project-in-org
[Create Project]: /docs/api/coalesce/create-project
[Update Project]: /docs/api/coalesce/update-project
[Delete Project]: /docs/api/coalesce/delete-project
[Get Runs]: /docs/api/coalesce/get-runs
[Get Run]: /docs/api/coalesce/get-run
[Get Run Results]: /docs/api/coalesce/get-run-results
[Get Nodes]: /docs/api/coalesce/get-nodes
[Get Node]: /docs/api/coalesce/get-node
[Get Workspace Nodes]: /docs/api/coalesce/get-workspace-nodes
[Get Workspace Node]: /docs/api/coalesce/get-workspace-node
[Start Run]: /docs/api/runs/startrun
[Retry Failed Run]: /docs/api/runs/retry-failed-run
[Cancel Run]: /docs/api/runs/cancelrun
[Run Status]: /docs/api/runs/run-status

---

## Connect to Transform MCP

Connect your MCP-capable AI client to Transform MCP over HTTP. No local install is required. You configure your client with your Coalesce App URL and refresh token instead of running a sidecar process.

For endpoint details, permissions, and regional hosts, see [Configure Transform MCP][].

## Before You Begin

You need:

1. A Coalesce account with permission to access the **Environments**, **Projects**, and runs you want the assistant to use.
2. Your Coalesce **refresh token**, the same credential COA uses.
3. An MCP client that supports HTTP transport with custom headers, such as Claude Desktop, Claude Code, Cursor, or VS Code.

## Step 1: Get Your Token

Your refresh token is the access token COA and the REST API use.

From the Coalesce App:

1. Sign in to the Coalesce App at your usual URL, for example `https://app.coalescesoftware.io` or your custom domain.
2. Open the **Deploy** tab.
3. Click **Generate Access Token**.

Tenancy is resolved from the token, so the subdomain in the URL does not matter. See [Getting Your Token][] for token lifecycle notes and regional base URLs.

## Step 2: Build Your MCP URL

Form the MCP endpoint from the host you sign in to:

```text
https://<your-app-host>/api/v1/mcp
```

Examples:

- US primary: `https://app.coalescesoftware.io/api/v1/mcp`
- Custom subdomain: `https://mycompany.app.eu.coalescesoftware.io/api/v1/mcp`

## Step 3: Add the Server to Your Client

Add the Transform MCP server to your MCP client with HTTP transport and an `Authorization: Bearer <your-refresh-token>` header. Replace `<your-app-host>` with the host you sign in to.

For step-by-step configuration by client, see:

- [Integrate Claude with Transform MCP][]
- [Integrate Cursor with Transform MCP][]
- [Integrate VS Code with Transform MCP][]

To register from Claude Code CLI:

```bash
claude mcp add --transport http coalesce-transform "https://<your-app-host>/api/v1/mcp" --header "Authorization: Bearer <your-refresh-token>"
```

## Step 4: Verify the Connection

Ask your assistant to list your Coalesce **Projects**, for example:

> List Coalesce projects in this organization.

A successful response returns the **Projects** in your organization. If you see an authentication error, re-copy the token from the **Deploy** tab and confirm the URL uses the same app host you sign in to.

## What's Next?

- [Configure Transform MCP][]
- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]
- [Transform MCP Troubleshooting][]

[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Getting Your Token]: /docs/api/authentication
[Integrate Claude with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-claude
[Integrate Cursor with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-cursor
[Integrate VS Code with Transform MCP]: /docs/coalesce-ai/mcp/integrate-mcp-vscode
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Find Coalesce IDs]: /docs/reference/whats-my-id
[Transform MCP Troubleshooting]: /docs/coalesce-ai/mcp/mcp-troubleshooting

---

## Transform MCP Troubleshooting

This page covers common issues when connecting AI clients to the hosted Transform MCP server. For setup steps, see [Connect to Transform MCP][] and [Configure Transform MCP][].

## Authentication Errors

### 401 Unauthorized

The server returns `401` when the `Authorization` header is missing, malformed, or the refresh token is invalid for the regional app host.

Check the following:

1. The header format is `Authorization: Bearer <your-refresh-token>`.
2. The token is current. Re-copy it from the **Deploy** tab or your COA `~/.coa/config` profile.
3. The MCP URL uses the app host for your region. A token from one region cannot authenticate against another region's host.
4. You restarted the MCP client after updating the config file.

See [Getting Your Token][] for token lifecycle events that invalidate existing tokens.

### Token Works for REST but Not MCP

Confirm the MCP URL path is `/api/v1/mcp`, not a REST API path such as `/api/v1/environments`. The token is the same, but the endpoint differs.

## Connection Errors

### Failed to Connect to MCP Server

| Cause | Fix |
| --- | --- |
| Wrong URL | Use `https://<your-app-host>/api/v1/mcp` |
| Network block | Allow HTTPS to your Coalesce app host from your machine |
| Client does not support HTTP MCP | Use a client with HTTP MCP support |

### Wrong Organization or Empty Results

Tenancy is resolved from your token, not the hostname subdomain alone. If results look empty:

1. Confirm you are signed in to the expected Coalesce organization in the Coalesce App.
2. Verify you have access to the **Projects** and **Environments** you are querying.
3. Call `projects_list` or `environments_list` explicitly to inspect raw responses.

## Permission Errors

Transform MCP does not grant permissions beyond your Coalesce account.

| Symptom | Likely cause |
| --- | --- |
| Create or delete denied | Your role lacks **Environment** or **Project** admin rights |
| Run start fails | You lack run permission on the target **Environment** |
| Resource not found | The ID is wrong or belongs to a **Project** you cannot access |

Use the Coalesce App or REST API with the same token to confirm whether the action succeeds outside MCP.

## Run Action Errors

### Refusing to Refresh All Nodes

`runs_start` returns an error when you omit both `jobID` and `includeNodesSelector` without confirming a full-environment refresh:

```text
Refusing to refresh all nodes: pass a jobID or includeNodesSelector to scope the run,
or set confirmRunAllNodes: true to refresh the entire environment.
```

Fix options:

- Pass `jobID` to run a specific **Job**.
- Pass `includeNodesSelector` to scope to selected **Nodes**.
- Set `confirmRunAllNodes: true` only when you intend to refresh every **Node** in the **Environment**.

See [Configure Transform MCP][] for the full guardrail description.

### Wrong Run ID Format

Run tools require a numeric `runID`, the scheduler `runCounter` value, not the UUID from a run URL.

```text
# Correct
runID: 42

# Wrong
runID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
```

See [Find Coalesce IDs][].

### Run Status Values

`runs_status` returns:

- **Terminal** - `completed`, `failed`, `canceled`
- **Non-terminal** - `waitingToRun`, `running`

Poll `runs_status` until the run reaches a terminal status.

## AI Client Not Calling Tools

If the assistant answers from general knowledge instead of calling MCP tools:

- Name the server in your prompt: *Use coalesce-transform MCP to list environments.*
- Ask for specific resource types: **Projects**, **Environments**, runs, or **Nodes**.
- Confirm the MCP server shows as connected in your client UI.
- Restart the client after config changes.

## Client-Specific Notes

### Claude Desktop

- Config file: `claude_desktop_config.json` under **Settings** > **Developer** > **Edit Config**
- **macOS logs:** `~/Library/Logs/Claude`

### Claude Code

- Project config: `.mcp.json` at the repository root
- CLI alternative: `claude mcp add --transport http …`

### Cursor

- Config file: `~/.cursor/mcp.json`
- Restart Cursor after edits

### VS Code

- Uses `servers` in `mcp.json`, not `mcpServers`
- Reload the window after configuration changes

## Still Stuck?

1. Verify setup with [Connect to Transform MCP][].
2. Review [Transform MCP Available Tools][] for required inputs.
3. Contact [Coalesce Support][] if the same token and IDs work in the REST API but fail consistently through MCP.

## What's Next?

- [Connect to Transform MCP][]
- [Transform MCP Available Tools][]
- [Find Coalesce IDs][]

[Connect to Transform MCP]: /docs/coalesce-ai/mcp/mcp-quickstart
[Configure Transform MCP]: /docs/coalesce-ai/mcp/configure-mcp
[Getting Your Token]: /docs/api/authentication
[Find Coalesce IDs]: /docs/reference/whats-my-id
[Transform MCP Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Coalesce Support]: /docs/get-started/support-information

---

## Data Platforms

Coalesce works with several major data platforms. This page summarizes how each platform fits with Coalesce and links you to connection guides and related docs.

## BigQuery

BigQuery is Google Cloud's serverless data warehouse for large-scale SQL analytics. Connect Coalesce to BigQuery with OAuth or a service account using the [BigQuery Connection Guide][].

## Databricks

Databricks is a unified analytics platform built on Apache Spark for data engineering, SQL analytics, and machine learning. Follow the [Databricks Connection Guide][] to connect your Databricks workspace to Coalesce with OAuth, a service principal, or a personal access token.

## Fivetran

Fivetran automates extraction, transformation, and loading from SaaS and database sources into your warehouse. Coalesce transforms data that Fivetran loads into Snowflake, BigQuery, and other supported platforms.

To connect Fivetran to Coalesce, work with Fivetran and review the [Fivetran documentation][].

## Snowflake

Snowflake separates compute and storage for elastic data warehousing, data lake workloads, and sharing across clouds. Snowflake is integrated natively with Coalesce. Start a free trial from our [Snowflake Marketplace listing][].

## What's Next?

- [Snowflake Quickstart][]
- [BigQuery Connection Guide][]
- [Databricks Connection Guide][]
- [Sync Back to Snowflake][]

[BigQuery Connection Guide]: /docs/setup-your-project/connection-guides/bigquery
[Databricks Connection Guide]: /docs/setup-your-project/connection-guides/databricks
[Fivetran documentation]: https://fivetran.com/docs/transformations/coalesce/setup-guide
[Snowflake Quickstart]: /docs/guides/snowflake-quickstart
[Sync Back to Snowflake]: /docs/catalog/integrations/sync-back/sync-back-to-snowflake
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce

---

## Caching Deployment Plan

The `enableCache` feature accelerates deployment **plan generation** by caching node metadata dependencies. This optimization allows the planning process to skip SQL re-rendering for unchanged nodes, dramatically reducing plan generation times for large projects.

## Caching Usage

The cache is disabled by default. It's recommended to keep caching disabled unless you are experiencing very long plan generation times.

The feature can be enabled in two ways:

* **CLI flag:** `coa plan --enableCache`
* **UI toggle:** Available in the "Parameters" step of the deployment wizard.

For troubleshooting planning or deployment issues, it is always recommended to run the plan with caching turned off to ensure a full, clean render.

## How Caching Works

When enabled, the `enableCache` flag modifies the plan generation process. The caching occurs when you run a plan; you do not need to deploy that plan for the cache to be populated.

### Plan Generation Process

1. Before rendering a Node during the planning phase, the system calculates a unique hash based on all the inputs that affect its compilation.
2. It checks a central cache for an existing entry matching that hash within the target environment.
3. If a valid entry is found, the system uses the cached information and skips the rendering step.
4. If no entry is found, the system renders the node using the current mechanism and then saves the result to the cache for future plans.

### What Is Not Cached

To manage expectations, it's important to know what this feature does not do:

* It does **not** cache the final compiled DDL (the SQL code itself).
* It does **not** apply to in-workspace actions like `Run All`, `Create All`, or `Validate`.

### Keep In Mind

* Caching only affects deployment plans (CLI and UI). It does not apply to workspace actions like *Run All* or *Validate*.
* If a cached result is unavailable or invalid, Coalesce automatically falls back to the normal rendering process.
* Cache entries are automatically evicted if unused for 90 days.
* **First-time use**: The first time you enable caching—or if the cache hasn’t been used for more than 90 days—you won’t see performance improvements immediately. Instead, Coalesce will populate the cache during that plan. Performance benefits appear on subsequent plans, where cached results can be reused.

---

## What's Next?

* [Deploy Using the CLI](/docs/deploy-and-refresh/deploy/deploy-using-the-cli)
* [CLI Commands](/docs/coa/version-733-and-above/coa-commands)

---

---

## Deploy Using the CLI

This guide goes over deploying your environment using the Coalesce `coa` CLI tool.

## Before You Begin

Make sure you have the following setup before deploying using `coa`.

- Install `coa` by following the instructions in [CLI Setup][].
- You need to have a [Git Repository][] with your pipeline information added.

### Environment Configured

Each deploy targets a Coalesce Environment. You can create and configure an Environment in the Coalesce App or from the CLI.

- [Data Platform Credentials][]
- [Storage Locations and Storage Mappings][]
- [Parameters][]

#### Manage Environments from the CLI

Environment create, update, and delete commands require Coalesce **7.36** or later. See [CLI Commands][] for flags and request file formats.

1. Authenticate with a Coalesce access token in `~/.coa/config` or your profile. See [CLI Setup][].
2. Ensure you have a Coalesce Project. If not, run `coa projects create` with a request file. See [CLI Commands][].
3. Run `coa environments create --inputFile <request-file>` with the Project ID, Environment name, `oauthEnabled`, and platform connection fields (`connectionAccount` for Snowflake or `accessUrl` for Databricks). Optional fields include `description`, `runTimeParameters`, and `mappings`.
4. Add warehouse credentials in the Coalesce App under **Build Settings > Environments > User Credentials**. Create and update requests do not accept passwords or OAuth refresh tokens.
5. Run `coa environments list` or `coa environments list --project <project-id>` and note the Environment ID for `coa plan` and `coa deploy`.

To change connection metadata or default parameters later, run `coa environments update --environmentID <id> --inputFile <patch-file>`. See [Update Environment][].

To remove an Environment you no longer need, run `coa environments delete --environmentID <id>`. The command prompts for confirmation unless you pass `--skipConfirm`.

## Step 1: Open Your Repository

You need to be in the correct repository with the commit you want to deploy before moving onto Step 2.

## Step 2: Create a Plan

This step assumed you've [installed `coa`][].

Before you can deploy, you need to create a plan. The plan details the environment, data platform credentials, and what will be deployed. You can create the plan as many times as needed before you deploy. It is based on the repo/branch and the environment configuration.

Run `coa plan`. This will generate a `coa-plan.json`. Open the file to review the plan information.

1. Review the `targetEnvironment`, this is the environment where the deploy will happen. You can see the environment ID in multiple locations:

    1. The UI by going to **Build Settings> Environments**. The environment IDs are 2 and 6 in this example.

        

    2. Run `coa environments list` or use the [List of Environments][] API.

    3. The environment ID is set by either:

        1. Hard-coding the information in the `.coa` file
        2. Setting it manually in the command line. `coa plan --environmentID 6`. You can also override the `.coa` file by using the command line.
2. You can also review `plan` object, which includes tables that have been deleted, added, or altered. You can also see jobs, parameters, macros, and other deployment information.

    1. If you want to change the `plan` object, make your edits in the Coalesce app then update your local branch. We don't recommend updating the plan file directly.

### Parameters

Parameters inherit from the **Environment** and can be overridden for this plan. Values overridden for a deploy are omitted from `runTimeParameters` in run results. Pass `--parameters` with a JSON object on `coa plan`. During plan generation, Coalesce also warns on undefined `{{ parameters... }}` references; warnings do not block the plan. See [Overwriting at Deploy][].

:::info[CLI Options]

Review the [CLI Commands][] for a full list of command line options.

:::

## Step 3: Deploy Your Plan

Run `coa deploy`.This will execute the plan you just created.

## Review the Deploy Status

[CLI Setup]: /docs/coa
[Data Platform Credentials]: /docs/setup-your-project/create-your-environments
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[Parameters]: /docs/deploy-and-refresh/parameters/
[List of Environments]: /docs/api/coalesce/get-environments
[Update Environment]: /docs/api/coalesce/update-environment
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[Overwriting at Deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy
[Git Repository]: /docs/git-integration/set-up-your-git-integration
[installed `coa`]: /docs/coa

---

## Deploy Using the Coalesce App

This guide walks you through a deployment in the Coalesce App using the **Deploy Wizard**.

## Before Deployment

Confirm the target **Environment** is configured before you open the **Deploy Wizard**.

- Review the [Deployment Overview][] to ensure you've configured your [Environment][].

## Deployment Wizard

To deploy, click **Deploy** on the target **Environment**. You need **Environment Admin** on that **Environment** to start the wizard.

The wizard opens with a three-step progress indicator: **Select Commit**, **Parameters**, and **Review Plan**. A footer bar shows the data platform connection, signed-in user, and the branch and commit you selected as you move through the steps.

### Select Commit

On **Select Commit**, choose the Git branch and commit to promote into the target **Environment**.

1. Use **Selected Branch** to pick the branch that holds the metadata you want to deploy. Click **Fetch** if you need to pull the latest commits from the remote without changing your Workspace checkout.
2. Select one commit from the table. Each row shows **Message**, **Author**, **Commit**, **Date**, **Used by Workspace**, and **Deployed In Environment**. For column definitions, see [Git Branches][].
3. Click **Next** to continue to **Parameters**.

For branch discipline and which commit to pick for production-like **Environments**, see [Deployment Overview][].

### Parameters

On **Parameters**, confirm or override runtime parameters and optional plan settings before Coalesce generates the deployment plan.

Parameters inherit from the **Environment** and can be overridden for this deploy only. Values overridden for a deploy are omitted from `runTimeParameters` in run results. See [Overwriting at Deploy][].

1. Edit the JSON editor if you need deploy-time overrides. Environment defaults appear in the editor when they are set.
2. Review optional toggles below the editor:
   - **Caching** - Disabled by default. Enable only when plan generation is very slow. See [Caching Deployment Plan][].
3. Click **Next**. Coalesce generates the deployment plan before **Review Plan** loads.

### Review Plan

On **Review Plan**, inspect every object Coalesce will create, alter, or delete in the target **Environment**. Resolve errors and warnings before you click **Deploy**.

#### Plan Tabs

The plan groups changes by object type. Open the tab that matches what you need to verify:

- **Nodes** - Tables, views, stages, and other Node objects in your pipeline
- **Subgraphs** - Subgraph definitions included in the deploy
- **Jobs** - Job definitions that will be created or updated
- **Mappings** - Storage and source mapping changes
- **Node Types** - Installed or updated Node Type definitions
- **Macros** - Macro changes
- **Packages** - Package version or configuration updates

#### Plan Table

Each tab lists affected objects in a table. Column headers include **Node Name** or the equivalent name for that object type, plus **Summary**, **Storage Location**, **Node Type**, and **View Details**.

Use **Summary** for a quick read of what will happen, for example deleted Nodes, Presync cleanup, column alters, or metadata-only updates.

#### View Details

Click **View Details** on any row to open the full plan for that object.

- **Stages and SQL** - The ordered list of stages Coalesce will run and the generated DDL. Use this view to confirm destructive steps such as **Replace Table**, **Create Table**, or **Drop Table**. For Snowflake interpretation, see [Snowflake Table Re-Creation During Deployment][].
- **Changes** - On altered Nodes, the **Changes** tab shows property-level before and after values for DDL, metadata, source mapping, and other configuration. Use it to confirm metadata-only edits before you deploy.

If the plan reports **errors**, for example missing **Storage Mappings**, fix the underlying configuration in the Coalesce App and regenerate the plan. **Deploy** stays disabled until blocking issues are resolved.

If the plan reports **warnings** about undefined parameter references, you can still continue. Fix the reference or parameter values, then regenerate the plan when you want a clean review. See [Overwriting at Deploy][].

## Review the Deploy Status

After you click **Deploy**, track progress on [The Deploy Page][]. Open the run from the **Activity Feed** to see per-Node stages, errors, and completion status.

You can also review results with the [List Run Results][] API using the run ID shown on the run, or with the CLI by adding `--out` to a deploy command.

[Environment]: /docs/get-started/coalesce-fundamentals/environments
[Git Branches]: /docs/git-integration/git-branches
[Deployment Overview]: /docs/deploy-and-refresh/deploy/
[Overwriting at Deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy
[Caching Deployment Plan]: /docs/deploy-and-refresh/deploy/caching-deployment-plan
[Snowflake Table Re-Creation During Deployment]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/snowflake-table-recreation-during-deployment
[The Deploy Page]: /docs/deploy-and-refresh/overview-of-the-deploy-interface
[List Run Results]: /docs/api/coalesce/get-run-results

---

## Excluding Nodes From Deployment

1. Go to the node you want to exclude.
2. In the **Node Properties** toggle **Deploy Enabled**.

    

3. Commit your changes. You will see `deployEnabled: false` as part of the node changes.

- If you toggle on this setting for a Node after you have previously deployed it, the Node will be dropped on your next deploy operation. This is the expected behavior for existing Nodes, as the deployment operation is bringing the existing environment up to the desired state depicted in what you are deploying; as the Node has now been excluded from the desired state (deployment), it's being assumed that it should no longer should exist and should be dropped.
- If you have any other feature branches in existence at the time your feature branch is created for this activity, and the Nodes you edit this setting in are also edited in another branch, you will need to be prepared to resolve merge conflicts when you go to merge the feature branches back into your main branch.
- If you later want to include this Nodes in deployment, you will have to reverse the process described above so "Deploy Enabled" is turned back on.

---

## Deployment Overview

Deployment creates the structural foundation of your Pipeline. When you deploy, Coalesce analyzes the differences between your current Environment and your desired configuration, then executes the necessary changes.

Once you've completed [Setting Up Your Project][] and [Building Your Pipeline][], you're ready to deploy your Pipeline to your data warehouse.

## Governance and Production Deployments

Production deployments stay predictable when you combine controls from version control, your continuous integration and delivery automation, and Coalesce. This section describes how those layers fit together, how manual and automated promotion work, and where to look when something blocks a deploy.

### Where Governance Lives

Governance is shared across three areas. You configure each one in its own system:

- **Version control** - Required reviewers, protected branches, optional merge queues, and repository rules live here.
- **Continuous integration and delivery** - Workflow triggers, job approvals, secret storage, and optional checks before a deploy runs are defined in your pipelines.
- **Coalesce** - Each Environment maps to branch and commit selection at deploy time, and [RBAC Roles and Permissions][] control who can run deploys through the Coalesce App, API, or [Deploy Using the CLI][]. Deploying requires **Environment Admin** on the target Environment.

Together, these layers decide which commits are eligible to run against which warehouse credentials, and who can start that work in Coalesce.

### Branches, Commits, and Non-Development Environments

Coalesce applies the branch and commit you select when you deploy. For non-development Environments, align with the DataOps pattern of treating one integration branch, often `main`, as the source of truth for promoted work. That keeps Workspace development flexible while production-like Environments track a single, reviewed line of history.

Environment mapping and branch discipline are covered in more detail in [DataOps Best Practices with Git and Coalesce][].

:::info[Pull requests and deploys]

Pull request review is enforced by your Git host as part of merging into protected branches. It is not a substitute for Coalesce roles: an approved merge still requires an account with **Environment Admin** or an automated principal with that role to deploy to the Environment. Coalesce does not add a second pull request approval step on the deploy action itself.

:::

### Manual Promotion in the Coalesce App

Use this pattern when a person promotes vetted commits after merges complete.

1. Confirm the commit you want is on the branch your team uses for production metadata, often `main`.
2. In the Coalesce App, open **Deploy** for the target Environment. An **Environment Admin** starts the **Deploy Wizard**.
3. On [Select Commit][], pick that branch and commit.
4. On [Deploy Wizard Parameters][], confirm deploy-time overrides and plan options, then click **Next** to generate the plan.
5. On [Review Plan][], inspect DDL, the **Changes** tab on altered Nodes, and any warnings, then click **Deploy**.

Step-by-step UI detail is in [Deploy Using the Coalesce App][].

### Automated Promotion in Pipelines

Use this pattern when merges or tags should trigger `coa plan` and `coa deploy` without someone clicking through the wizard.

1. Store the Coalesce API token and related secrets in your CI system, not in the repository.
2. Run `coa plan`, review or archive the plan artifact as your team requires, then run `coa deploy` with that plan against the target Environment.
3. Give the identity your pipeline uses **Environment Admin** on the Environment it deploys to. Many teams use a dedicated Environment for automation so production credentials stay behind a separate manual or gated job.

Command reference is in [Deploy Using the CLI][]. A full GitHub Actions example is in [Orchestrate Deploys With GitHub Actions and the CLI][].

For Git operations inside automation when a bot performs commits, clones, or pushes, use a dedicated Git account so tokens and ownership do not depend on an individual developer. See [Creating a Git Service Account][]. For the Coalesce side of automation identities, see [Service Accounts in Coalesce][].

### Example Team Patterns

These patterns are simplified sketches you can adapt to your own naming and Environment layout.

#### Small Team: Merge in Git, Then Deploy From `main`

Your Git host requires at least one approving review before merge into `main`. After the merge lands, an **Environment Admin** opens the **Deploy Wizard** for Production, selects `main` and the new commit, reviews the plan, and deploys. Lower Environments can follow the same branch or use feature branches for testing before you merge.

#### Larger Team: Protected `main`, Automation to a CI Environment, Optional Human Gate for Production

`main` stays protected with required reviewers in GitHub or your equivalent Git host, with an optional merge queue if your team uses that feature. A workflow on push to `main` runs `coa plan` and `coa deploy` into a dedicated CI or staging Environment using a service identity with **Environment Admin** on that Environment only. If you want an extra human checkpoint before Production, keep Production deploys as a separate manual **Deploy Wizard** step or a workflow that runs only after an allowed approver starts it in your CI system.

#### Automation Accounts for Git and Coalesce

A pipeline bot or service user performs Git steps and calls Coalesce with an API token. Configure that Coalesce identity using [Service Accounts in Coalesce][] so it has **Environment Admin** on the automation Environment, following [Adding Users and Setting Permissions][]. Pair it with [Creating a Git Service Account][] so Git credentials match the automation story.

### Hotfixes on a Faster Path Than the Normal Release

When you cannot wait for the standard release cadence, you still branch, test, and merge in Git before Production sees the commit. Branch from your integration branch, implement the fix, validate in a lower Environment, deploy where needed, then merge back so `main` stays authoritative. Follow the numbered hotfix flow in [Hotfix Management in the DataOps guide][].

### Optional Policy Checks in Pipelines

Some organizations add static analysis, policy engines such as Open Policy Agent, or custom gates so a job fails before `coa deploy` runs. Those checks are part of your pipeline definitions and tooling choices, not built-in Coalesce product rules. Design them where you already enforce repository and infrastructure standards.

### When Automated Deploys Fail With Permission Errors

If `coa deploy` from CI reports permission or `env-deploy` style errors, treat it as a roles and secrets problem first. Confirm the Coalesce identity has **Environment Admin** on the target Environment and that your secrets match the Environment and Project you intend. Walk through [COA Deployment Error Resolution][] and confirm data platform grants for the Environment credentials.

## Deployment Methods

You can deploy your Pipeline using:

- [The Coalesce App][]
- [CLI][]
- [Third-Party Scheduling Tools][]

## Before You Deploy

This assumes you've created your [Pipeline][]. If you don't have one, check out the [Quick Start Guide][].

- Configure your [Environment][]
- Configure your [Git Integration][]
- Optionally configure [Exclude Nodes from Deployment][]
- Optionally set your [Parameters][]
  - You can set them on the Environment level or while you deploy.
- Deploy your Pipeline. You can use the following:
  - [Deploy Using the CLI][]
  - [Deploy Using the Coalesce App][]

<DocCardList />

---

[Setting Up Your Project]: /docs/setup-your-project/
[Building Your Pipeline]: /docs/build-your-pipeline/the-build-interface/
[Pipeline]: /docs/category/build-your-pipeline
[Quick Start Guide]: /docs/get-started/quick-start
[Environment]: /docs/get-started/coalesce-fundamentals/environments
[Git Integration]: /docs/setup-your-project/setup-version-control
[Exclude Nodes from Deployment]: /docs/deploy-and-refresh/deploy/excluding-nodes-from-deployment
[Parameters]: /docs/deploy-and-refresh/parameters/
[Deploy Using the CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli
[Deploy Using the Coalesce App]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment
[Select Commit]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment#select-commit
[Deploy Wizard Parameters]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment#parameters
[Review Plan]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment#review-plan
[Third-Party Scheduling Tools]: /docs/deploy-and-refresh/third-party-devops-tools/
[The Coalesce App]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment
[CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli
[DataOps Best Practices with Git and Coalesce]: /docs/guides/dataops-and-git-best-practices
[RBAC Roles and Permissions]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Adding Users and Setting Permissions]: /docs/setup-your-project/add-users-set-permissions
[Service Accounts in Coalesce]: /docs/organization-and-accounts/organization-management/role-based-access-control/service-accounts
[Creating a Git Service Account]: /docs/git-integration/creating-a-git-service-account
[Orchestrate Deploys With GitHub Actions and the CLI]: /docs/deploy-and-refresh/third-party-devops-tools/continuous-integration/schedule-jobs-github-actions-cli
[COA Deployment Error Resolution]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/coa-deployment-error-resolution
[Hotfix Management in the DataOps guide]: /docs/guides/dataops-and-git-best-practices#hotfix-management

---

## Rollback a Deployment in Snowlake

The following approach will rollback data structures, but will not restore data to its previous state in the case of tables/columns that were restored via the rollback operation. If you are restoring a table or column as part of the rollback operation, you should leverage [Snowflake Time Travel][] to reinstate your data to its intended state after you have completed the rollback of your data structures

## How To Rollback a Deployment in Coalesce

To rollback a deployment in Coalesce and restore your Environment to its prior state in terms of data structures, you can redeploy the commit that was deployed just prior to the deployment you wish to rollback. This will rollback to what was previously deployed in terms of data structures:

- Any new tables or columns that were created in Snowflake in the deployment you are rolling back will be dropped.
- Any prior tables or columns that were dropped during the deployment will be recreated.
- Any column that was ALTERed in the deployment you are rolling back will be re-ALTERed to restore its previous state.

[Snowflake Time Travel]: https://docs.snowflake.com/en/user-guide/data-time-travel

---

## Understanding the Presync Process

When you deploy tables and views in Coalesce, a background process called **presync** runs automatically. Presync helps prevent deployment failures caused by changes made outside of Coalesce. This guide explains what presync is, why it matters, and what you’ll see when it runs.

## The Problem Presync Solves

Deployments can fail if objects are changed or deleted directly in your data platform. For example, you might deploy a table called `CUSTOMER_DATA` through Coalesce. Later, someone drops that table directly in the warehouse. When you try to redeploy changes, Coalesce attempts to modify a table that no longer exists, causing the deployment to fail.

Presync solves this by keeping Coalesce synchronized with the current state of your data platform. Common off-platform changes include:

- Tables or views being dropped directly in the warehouse
- Schema changes made through other tools
- Objects renamed or moved to different databases
- Entire databases or schemas deleted

## How Presync Works

Presync compares three sources of truth:

1. What Coalesce has stored from your last successful deployment.
2. What you’re trying to deploy.
3. What actually exists in your data platform.

When differences are found, presync decides how to reconcile them to prevent deployment failures.

### Phase 1: Discovery and Validation

Presync categorizes your changes:

- New tables or views being added
- Existing tables or views being modified
- Tables or views being removed

It then checks that all referenced databases and schemas exist. If any are missing, presync filters out the affected nodes.

### Phase 2: Conflict Resolution

For each object, presync compares intent with reality:

- **New objects**
  - If the object doesn’t exist: deployment continues normally.
  - If it already exists with the correct structure: presync adopts it.
  - If it exists with differences: presync updates Coalesce’s understanding to match.
- **Modified objects**
  - If it exists as expected: presync compares columns and updates details.
  - If it’s missing: presync creates it.
  - If it was moved or changed type: presync updates Coalesce to reflect the actual state.

### Phase 3: Column Synchronization

Presync reviews detailed table and view structures, including:

- Column names, types, and properties
- Missing columns to be added
- Unexpected columns that are preserved
- Changed column properties to be updated

### Platform-Specific Validations

On **Databricks**, presync validates that Delta Lake tables include required properties for schema evolution. If the property `delta.columnMapping.mode=name` is missing, presync halts deployment and reports which tables must be fixed.

## What You’ll See During Deployment

### Normal Operation

You’ll see a “Making metadata updates” step when presync runs. In most cases, this step completes quietly and deployment continues.

### When Presync Adjusts Objects

If presync detects differences, you’ll see log messages such as:

- **Object unexpectedly exists at location** - Presync adopted an object it didn’t expect.
- **Object missing from expected location** - Presync recreated an object.
- **Column differences detected** - Presync found unexpected column properties.
- **Location no longer exists** - A database or schema was deleted.

When your deploy plan shows **Replace Table**, **Create Table**, or **Drop Table** stages, see [Snowflake Table Re-creation During Deployment][] to interpret alter versus re-create behavior and when each stage is expected.

### When Presync Stops Deployment

Deployment is halted if:

- Required platform properties are missing
- Structural conflicts can’t be resolved automatically

## Example Scenarios

### Adopting an Existing Table

You add a new node for `SALES_SUMMARY`, but the same table already exists with the correct structure. Presync adopts the table instead of recreating it, preserving your data.

### Handling a Moved Table

You move `CUSTOMER_DATA` from `staging` to `prod`, but another `CUSTOMER_DATA` already exists in `prod`. Presync compares structures and either adopts the existing table or reports conflicts.

### Database Cleanup

If a referenced database has been deleted, presync detects the missing location and treats those nodes as new objects, preventing deployment errors.

## Benefits of Presync

- Prevents deployment failures caused by off-platform changes  
- Preserves data whenever possible  
- Provides detailed visibility into external changes  
- Makes intelligent decisions automatically  
- Validates platform requirements before deployment continues  

## Best Practices

- Review presync logs if a deployment behaves unexpectedly
- Limit off-platform changes when possible
- Remember that presync adopts existing objects rather than overwriting them
- On Databricks, ensure Delta Lake tables include `delta.columnMapping.mode=name`

## When To Investigate Further

Contact your Coalesce administrator if you see:

- Repeated warnings about unexpected object states
- Deployment failures tied to missing platform requirements
- Presync decisions that don’t align with your expectations

Presync is designed to make deployments more reliable by automatically reconciling differences between Coalesce and your data platform.

[Snowflake Table Re-creation During Deployment]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/snowflake-table-recreation-during-deployment

---

## Deploy and Refresh

:::tip[Coalesce Scheduler]
The Coalesce Scheduler is now available. Schedule refreshes directly in the app. Read more in [Scheduling Jobs in Coalesce][].
:::

## Understanding Deploy and Refresh

**Deploy** applies structural changes (DDL) such as creating or altering tables and columns. **Refresh** runs data transformations (DML) such as MERGE, INSERT, and UPDATE. You must deploy before you refresh. For concepts, when to use each operation, and Job types, see [Understanding Deploy, Refresh, and Jobs in Coalesce][].

## Getting Started

Here are the recommended steps for getting started:

1. Configure your [Environment][] settings.
2. [Deploy][] your pipeline.
3. Create and schedule refresh Jobs using the [Coalesce Scheduler][].
4. Monitor [deployment][] and [refresh][] status.

## Deploy and Refresh Methods

| Location | Deploy | Refresh and Jobs |
| --- | --- | --- |
| Coalesce App | ✔️ | ✔️ |
| CLI | ✔️ | ✔️ |
| REST API | | ✔️ |
| DevOps Platform | ✔️ | ✔️ |

<DocCardList />

[Understanding Deploy, Refresh, and Jobs in Coalesce]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh
[Scheduling Jobs in Coalesce]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Environment]: /docs/get-started/coalesce-fundamentals/environments
[Deploy]: /docs/deploy-and-refresh/deploy/
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[deployment]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment#review-the-deploy-status
[refresh]: /docs/deploy-and-refresh/refresh/managing-jobs#refresh-status

---

## Job Notifications and Monitoring

## Introduction

You can monitor and receive notifications for Jobs in Coalesce using built-in tools or through integrations with external systems. This guide covers the available options for configuring notifications, monitoring job status, and setting up retries for recovery.

## Built-in Scheduler Notifications

If you're using the [Coalesce Job Scheduler](/docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce), you can configure email notifications directly in the platform. Notifications can be set up for job successes, job failures, or both. Multiple email addresses can be added to receive alerts.

## External Orchestration Tool Integration

If you’re managing jobs with an external orchestration tool such as [Airflow](/docs/deploy-and-refresh/third-party-devops-tools/orchestration/schedule-coalesce-jobs-with-apache-airflow) or [Azure Data Factory](/docs/deploy-and-refresh/third-party-devops-tools/orchestration/coalesce-refresh-scheduling-with-azure-data-factory) you can configure these tools to send alerts. The orchestration tool can call Coalesce APIs to check job status and then trigger alerts through your configured channels such as email, Slack, or SMS.

## API-Based Monitoring

You can also use Coalesce’s APIs to programmatically monitor job status and build custom alerting solutions. With APIs, you can:

- Use the [job status API](/docs/api/runs/run-status) to check if jobs succeeded or failed.
- Retrieve detailed error messages and run results.
- Integrate with your existing monitoring and alerting platforms.

## Retry and Recovery Options

We recommend configuring retry mechanisms to handle transient failures. You can set jobs to [retry automatically](/docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes) from the point of failure and use scheduling with retry logic to ensure resilience across job runs.

---

## What's Next?

- [Coalesce Scheduler](/docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce)
- [Run/Jons API](/docs/api/runs/startrun)

---

---

## The Deploy Page

The **Deploy Page** is where you can deploy your project's metadata from the desired state in your Git repository, and see a history of your pipeline's **Deploys** and **Refreshes**.

## Filters

The **Filters** drop-down allow you to narrow down the runs currently visible. For example, if you wanted to see all failed deployments on the New Environment, you would do the following:

1. Select New Environment from **Environments**.
2. Select Deploy from the **All Types**.
3. Select Failed from the **All Statuses**.
4. Click **Apply Filters** and you'll see all the deployments failed deployments.

## Dashboard

The **Dashboard** shows a list of your currently deployable environments. From here you can configure and deploy your **Environments** and hover the colored bars with your mouse to view specific runs.

**View Documentation** allows for quick access to the generated documentation for the current state of that environment. Additionally, you can generate an access token for use in our API or CLI tools.

## Activity Feed

The **Activity Feed** shows a list of completed and currently running jobs, which include both **Refreshes** and **Deploys**.

## Individual Run Interface

Clicking on a run in the **Activity Feed** will open that individual run's details.

1. **Run Header** - This section shows the **Environment**, run ID number, and Nodes progress bar.
2. **Run Information** - This section shows many details about the run, many updated in real time. You can select **Details**, **Connection**, and **Parameters** from the left side to see the various listings.
3. **Run Results** - Each line or stage is a single SQL query for one of the Nodes in that run. There can be multiple stages per Node, so the number of stages doesn't necessarily equal the number of Nodes.

## Run Cancellation

Coalesce deployment and refresh runs can be cancelled while they're still in progress by clicking the **Cancel Run** button in the **Run Header**.  Two actions occur upon cancellation:

> Snowflake
>
> 1. Coalesce aborts all queued and running Snowflake queries.
> 2. Coalesce stops submitting queries to Snowflake.

## API Tokens

Generate an access token by selecting **Generate Access Token**. Use this token for:

- [API Access](/docs/api/)
- [CLI](/docs/coa/version-732-and-below/coa-setup)

---

## Parameters

Parameters act as read-only environment variables. Parameters are defined as a JSON blob set by the user that can be accessed in the metadata during template rendering. These values can be set for a Workspace/Environment at various points to provide flexibility of template behavior during Deploying or Refreshing.

## Where Parameters Can Be Used

- In Your Workspace or Environment to set [default parameters][].
- Overwrite default parameters at [deploy][].
- Overwrite default parameters at [refresh][].
- Overwrite default parameters when you [create or edit a Job Schedule][] in the Job Scheduler.

## Parameters in Git

Parameters are exclusively stored in Coalesce's metadata repository and will not be committed to your Git repository.

## Why Use Parameters

- **Environment Configuration**: Parameters allow you to configure your data pipeline for different environments (for example,, development, staging, production) without modifying the underlying code. For example, you can use parameters to specify different database credentials, file paths, or other environment-specific settings.
- **Dynamic Data Filtering**: Parameters can be used to dynamically filter or partition data based on certain criteria. For example, you can use a date parameter to only process data for a specific date range or a customer ID parameter to process data for a particular customer.
- **Reusable and Modularity**: By changing certain values or logic into parameters, you can make your data pipeline more modular and reusable. This promotes code reuse and maintainability, as you can easily swap out parameter values without modifying the core transformation logic.
- **Testing and Debugging**: Parameters can be helpful for testing and debugging purposes. You can use different parameter values to simulate various scenarios or edge cases, making it easier to identify and fix issues in your data pipeline.
- **Scheduling and Automation**: When scheduling or automating data pipeline runs, parameters can be used to pass dynamic values or configurations. This allows you to adapt the pipeline behavior without changing the underlying code, enabling greater flexibility and control.
- **Separation of Concerns**: By externalizing certain values or configurations as parameters, you can separate the concerns of data transformation logic from the specific values or configurations used in different contexts. This promotes better code organization and maintainability.
- **Compliance and Auditing**: In regulated industries or environments with strict data governance requirements, parameters can be used to enforce data access controls, data masking, or other compliance-related configurations.

<DocCardList />

[default parameters]: /docs/deploy-and-refresh/parameters/rtp-default
[deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy
[refresh]: /docs/deploy-and-refresh/parameters/rtp-refresh
[create or edit a Job Schedule]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce#how-to-schedule-a-job

---

## Setting Default Parameters

## Setting Default Parameters for a Workspace/Environment

To set default Parameters for an environment:

1. Open the **Build Settings Tab**, the gear icon located on the bottom left of the build tab interface.
2. Choose a **Workspace/Environment** and **Edit**.
3. Click on **Parameters** and add your Parameters in JSON format.
4. Make sure to click **Save** to save your new parameters.

## Example of Setting Parameters

1. Using CUSTOMER as the source Node.
2. Create and Run Staging Node from the CUSTOMER Source node.
3. Go to your Workspace by clicking **Build Settings > Edit** on your Development Workspace.
4. Go to Parameters and add the following:

   ```jsom
    {
        "mkt_segment": "BUILDING"
    }
    ```

5. In the Staging node, go to the JOIN tab. It should already have a ref line. Add the line `WHERE C_MKTSEGMENT = '{{ parameters.mkt_segment }}'`. Then Create and Run again.

    ```sql title="Final JOIN tab SQL"

        FROM {{ ref('PROD', 'STG_CUSTOMER') }} "STG_CUSTOMER"

        WHERE C_MKTSEGMENT = '{{ parameters.mkt_segment }}'
    ```

6. In the results pane, the Data tab will show all records where `C_MKTSEGMENT` is equal to `BUILDING`.

## Using Parameters in Transforms

The values in Parameters can be used in transforms within Nodes using the following format:  
`{{ parameters.my_var }}`  
where your parameters follow JavaScript object dot notation syntax. Below are a few examples to illustrate this.

```sql title="Parameters in Transforms Example"
{{parameters.my_number}}

CONCAT( "CUSTOMER"."C_ADDRESS", '{{parameters.my_string}}' )

CONCAT( "CUSTOMER"."C_ADDRESS", '{{parameters.accepted_values[0]}}' )
```

:::info[On Parameter Value Resolving]

`{{parameters.my_number}}` will not resolve to 1 (in the above example) until the node is actually rendered. In other words, the value in the transform field will remain `{{parameters.my_number}}`.

:::

## Using Parameters in a Node Type Template

In the Node Type Editor, Parameters are appended to the metadata. Using Jinja logic, these parameters can be accessed and rendered in the Create and Run Templates if so desired.

They can be inserted into the Jinja Templates following the same format as in transforms and will run within Stages.

## Parameters and Advanced Deploy Strategy

When using parameters as part of an Advanced Deployment Strategy, they need to be referenced with `desiredState` or `currentState`. For example, instead of `{{parameters.my_number}}`, instead use `{{desiredState.parameters.my_number}}`.

---

## Overwriting at Deploy

Learn how to override [default parameters][] when you deploy from the Coalesce App, and how to pass JSON overrides on `coa plan` when you use the CLI (including shell quoting on macOS, Linux, and Windows).

## Overwriting Default Parameters in the Coalesce App

When deploying from the Coalesce App, you'll have the option to overwrite [default parameters][] or provide new parameters specific to that Environment. If default parameters were set, they will appear here.

:::warning[Persistence of Overwrites]

Any changes to **Parameters** in the **Deploy** screen will only apply to that specific deployment. Defaults will be left unchanged for subsequent deployments. Values overridden at deploy time are omitted from `runTimeParameters` in run results.

:::

## Overwriting Default Parameters With Plan

To override [default parameters][] from the command line, pass `--parameters` with a valid JSON object on `coa plan` for that run.

:::warning[Override on Deploy Doesn't Work]

`--parameters` on `coa deploy` does nothing. Deploy does not read JSON parameters or merge them again.

:::

When you include `--parameters` on `coa plan`:

1. Planning renders your pipeline SQL with those values.
2. `coa deploy` runs the queries produced during that plan step (the same rendered SQL), not a separate parameter pass on deploy.

## Parameter Reference Warnings During Plan

When Coalesce generates a deployment plan in the **Deploy Wizard** or with `coa plan`, it checks `{{ parameters... }}` references in your Nodes and Node Types against the parameters defined for that plan. Mistyped or missing keys surface as **warnings**. Warnings do not block the plan by themselves.

Resolve warnings by correcting the reference, adding the key to your [default parameters][], or supplying it as a plan-time override on the **Parameters** step or with `--parameters`. Prefer fixing warnings before you deploy or refresh so templates do not fail later with missing values.

This check runs at plan time. It does not run when you save Nodes in the Workspace or when you use local `coa create` or `coa run`.

<Tabs>
  <TabItem value="mac-linux" label="bash" default>

    Wrap the JSON in single quotes so the shell does not interpret double quotes inside the object:

    ```bash
    coa plan --parameters '{ "foo": "bar" }'
    ```

  </TabItem>
  <TabItem value="powershell" label="Windows PowerShell">

    A single-quoted argument is treated literally, which usually matches the macOS or Linux approach:

    ```powershell
    coa plan --parameters '{ "foo": "bar" }'
    ```

    If that fails for your PowerShell version or profile, use the backslash-escaped double quotes from the **Windows Command Prompt** tab, or run the command from Command Prompt.

  </TabItem>
  <TabItem value="windows-cmd" label="Windows Command Prompt">

    The shell strips the outer double quotes and passes the JSON to the CLI. Escape each inner double quote with a backslash:

    ```bat
    coa plan --parameters "{\"foo\": \"bar\"}"
    ```

  </TabItem>
</Tabs>

[default parameters]: /docs/deploy-and-refresh/parameters/rtp-default

---

## Overwriting at Refresh

Learn how to pass runtime parameters when you refresh a pipeline with `coa refresh` or the [Start a Run][] endpoint, including how replacement works when you supply a `parameters` object.

## COA CLI

You can set parameters at refresh time by passing JSON after `--parameters` on `coa refresh`.

`coa refresh --parameters '{ "foo": "bar" }'`

When you use `--parameters`, follow the same rule as the [Start a Run][] `parameters` field: pass the full runtime parameter map your Job needs for that Run, not a partial update layered on environment defaults. If you omit `--parameters`, the Run uses the environment defaults you configured in **Build Settings > Environments** for that environment.

## REST API

You can pass runtime parameters in the JSON body of a [Start a Run][] request by including a `parameters` object with your credentials and `runDetails`.

When you include `parameters`, Coalesce treats it as **the complete set of runtime parameters for that Run**. Keys from environment-level parameters are **not** merged in automatically. If the environment defines parameters your Job still needs, you must include those keys and values in `parameters`, or omit `parameters` entirely so the Run uses environment defaults alone. Values you override at refresh time are omitted from `runTimeParameters` in run results.

This behavior trades a larger request body for predictable results: you copy or reconstruct the full map in your client when you need overrides.

Use the following pattern when you only need to change a few values for one Run:

1. Start from the full parameter map the environment would normally supply.
2. Apply your overrides.
3. Send the resulting object as `parameters` in the request.

If you send a partial map by mistake, omitted keys may be missing for that Run, which can cause Nodes or Jobs to miss values they expect.

### Example

Suppose your environment defines runtime parameters `A` and `B`. If your request body includes only `C` inside `parameters`:

```json
{
  "parameters": {
    "C": "1"
  }
}
```

then `A` and `B` are **not** carried forward for that Run unless you also list them inside `parameters`. The effective runtime map for the Run is the object you sent under `parameters`.

:::tip[When you want environment defaults only]

If you don't need run-specific overrides, omit `parameters` from the request body so the Run picks up environment defaults without you copying the map client-side.

:::

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

     ```json
     {
          "runDetails": {
               "environmentID": "4"
          },
          "userCredentials": {
               "platformKind": "databricks",
               "databricksClientSecret": "******",
               "databricksClientID": "*****",
               "databricksPath": "sql warehouse path",
               "databricksAuthType": "OAuthM2M"
          },
          "parameters": {
               "foo": "bar1"
          }
     }
     ```

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

     ```json
     {
          "runDetails": {
               "environmentID": "4"
          },
          "userCredentials": {
               "snowflakeUsername": "user",
               "snowflakePassword": "*****"
          },
          "parameters": {
               "foo": "bar1"
          }
     }

     ```
  </TabItem>
</Tabs>

For how defaults and deploy-time overrides work, see [Default Parameters][] and [Overwriting at Deploy][]. The full request schema and credential shapes for refresh are documented on [Start a Run][].

---

[Start a Run]: /docs/api/runs/startrun
[Default Parameters]: /docs/deploy-and-refresh/parameters/rtp-default
[Overwriting at Deploy]: /docs/deploy-and-refresh/parameters/rtp-deploy

---

## Query Tags in Snowflake and Coalesce

Coalesce uses JSON `query_tag` in Snowflake to track job execution. You'll learn how to view the `query_tag` and ways to use it.

## Query Tags Applied by Coalesce

Coalesce automatically adds a query tag to every SQL query that runs as part of a Job. The query tag is a JSON string with these fields:

* `jobName`: The name of the Job that's running.
* `nodeID`: The unique identifier for the Node that's running.
* `nodeName`: The name of the Node that's running.
* `runID`: The unique identifier for the Job run.
* `runType`: The type of operation, such as Refresh or Deploy.
* `stageName`: The stage in the run, such as Insert into table.
* `storageLocation`: The target Storage Location.

```json title="Query Tag Example"
{
 "coalesce": {
  "jobName": "Refreshed Job - customer refresh",
  "nodeID": "b4e70d66-0b69-48b2-95c5-3ec198467287",
  "nodeName": "DIM_CUSTOMER_LOYALTY",
  "runID": "18764",
  "runType": "Refresh",
  "stageName": "MERGE  Sources",
  "storageLocation": "STG"
 }
}
```

### Run ID

The RunID is a unique identifier assigned to each job execution in Coalesce. It allows you to track all queries and operations that were part of a specific job run, enabling you to correlate Coalesce metadata with Snowflake query history for debugging, auditing, and cost analysis

## Inspect Query Tags in Query History

This query reads recent query history and shows the query tag and SQL text.

```sql
SELECT
    start_time,
    query_tag,
FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY(
    END_TIME_RANGE_START => DATEADD('hour', -1, CURRENT_TIMESTAMP())
))
WHERE query_tag IS NOT NULL
ORDER BY start_time DESC;

-- You can also use SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY

SELECT
    start_time,
    query_tag,
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE start_time >= DATEADD('hour', -1, CURRENT_TIMESTAMP())
  AND query_tag IS NOT NULL
ORDER BY start_time DESC;
```

### How Inspect Query Works

* Call the `QUERY_HISTORY` table function

```sql
FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY(
    END_TIME_RANGE_START => DATEADD('hour', -1, CURRENT_TIMESTAMP())
))
```

* `INFORMATION_SCHEMA.QUERY_HISTORY` returns query history for the current account
* Wrapping it in `TABLE(...)` makes it usable in the `FROM` clause
* The `END_TIME_RANGE_START` argument tells Snowflake to return queries that ended after a specific time
* `DATEADD('hour', -1, CURRENT_TIMESTAMP())` moves back one hour from the current timestamp

The result is a set of queries that completed in the last hour.

---

## What's Next?

* [Snowflake - QUERY_HISTORY , QUERY_HISTORY_BY_*][]

---

[Snowflake - QUERY_HISTORY , QUERY_HISTORY_BY_*]: https://docs.snowflake.com/en/sql-reference/functions/query_history

---

## Refresh Your Pipeline

Refresh operations update your data by executing the transformations defined in your pipeline. A refresh can update your entire pipeline or specific Jobs that you've created to manage subsets of your data.

## What Is Refresh?

Refresh runs the data transformations defined in your pipeline metadata. Use it when you need to load or update data after structure is in place. For how refresh differs from deploy, when to use each, and Job types, see [Understanding Deploy, Refresh, and Jobs in Coalesce][].

## What Are Jobs?

Jobs are a subset of Nodes, created by the selector query, that are run during a refresh. To refresh only specific parts of your pipeline, create, and use Jobs.

:::info[Deploy Jobs]

Jobs can only be run if they have been deployed first. Review our Deployment Overview to learn different ways to deploy your pipeline.

:::

## Refresh Methods

The Coalesce Scheduler lets you automate refresh operations directly in the application, making it easier to maintain regular data updates without external tools. You can also refresh your pipeline using:

- [The Coalesce Scheduler][]
- [The Coalesce App][]. Only existing, deployed Jobs can be run from the Coalesce App.
- [CLI][]
- [Jobs API][]
- [Third-Party Scheduling Tools][]

## Types of Jobs

Coalesce offers three ways to refresh your data pipeline: pre-configured Jobs with specific IDs, ad-hoc Jobs for manual execution, and full pipeline refreshes.

| Name | Job ID | Method | Description |
| --- | --- | --- | --- |
| Jobs | Yes | API, CLI, Coalesce Scheduler, Coalesce App | Any Jobs you created in the Coalesce app on the Build page. They have a Job ID and are started using the Coalesce Scheduler, API, Coalesce App, or CLI. |
| Ad-Hoc | None | API or CLI | Jobs that run manually using the API or CLI. They use include and exclude syntax. They aren't created in the app and can be run in addition to existing Jobs. These are standard within Coalesce and can't be removed from the Deploy page. |
| Refreshed All Jobs | None | API or CLI | Refresh all the nodes in your pipeline. They don't use include or exclude syntax. They aren't created in the app and can be run in addition to existing Jobs. These are standard within Coalesce and can't be removed from the Deploy page. |

## Steps to Refresh or Run Jobs

Deploy and refresh jobs are triggered for individual environments. Once an environment has been deployed, it can be refreshed using the Scheduler, API, or CLI.

1. Create your [Jobs][]
2. Configure your [Environment][]
3. Configure your [Git Integration][]
4. Set your [Parameters][] (optional). You can set them on the environment level or during the deploy processes.
5. [Refresh][] your pipeline.

:::info[Deploy Before Refreshing]
You can only refresh if you've deployed your pipeline.
:::

<DocCardList />

[Understanding Deploy, Refresh, and Jobs in Coalesce]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh
[The Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[The Coalesce App]: /docs/deploy-and-refresh/refresh/refreshing-an-environment#coalesce-app
[CLI]: /docs/deploy-and-refresh/refresh/refreshing-an-environment#cli
[Jobs API]: /docs/deploy-and-refresh/refresh/refreshing-an-environment#api
[Third-Party Scheduling Tools]: /docs/deploy-and-refresh/third-party-devops-tools/
[Jobs]: /docs/deploy-and-refresh/refresh/refreshing-an-environment
[Environment]: /docs/deploy-and-refresh/refresh/refreshing-an-environment
[Git Integration]: /docs/deploy-and-refresh/refresh/refreshing-an-environment
[Parameters]: /docs/deploy-and-refresh/parameters/
[Refresh]: /docs/deploy-and-refresh/refresh/refreshing-an-environment

---

## Managing Jobs

## Edit Jobs

- Jobs can also be modified by dragging and dropping Nodes or Subgraphs into the include/exclude text boxes, while on the Graph, Node Grid, or Column Grid of a Job. For when to use Subgraphs and how they differ from Jobs, see [Using Subgraphs to Build Your Pipeline][].
- If you adjust a subgraph that changes what a Job should refresh, commit the Job to version control and deploy. The [Coalesce Scheduler][] only runs deployed Jobs.
- Nodes can only be removed from the Job by modifying the include/exclude query.
- Right click each Job for more options.

<VideoPlayer url="/img/docs-images/deploy-your-pipeline/editing_jobs.mp4" />

## Refresh Status

## Job Scheduling

Jobs can be run at set time intervals using a scheduler. See our article on [Scheduling][] for details and examples.

## Re-Run a Job

You can a rerun an existing Job using the API or CLI.

- **CLI** - Can re-run Nodes, starting at the point of failure. Use the `refresh` [commands][]. `coa rerun <runID>`.
- **API** - Can re-run Nodes, starting at the point of failure. Use the [Rerun a Job][] endpoint.
- **Coalesce App** - Starts a new run each time. Doesn't run previously failed job.

[Scheduling]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[commands]: /docs/coa/version-733-and-above/coa-commands#-coa-refresh-options
[Rerun a Job]: /docs/api/runs/retry-failed-run
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Creating and Run Jobs

## Before You Begin

You'll need to get an authentication token to run refreshes. Tokens are used for the [Trigger Job to Run][] API or using refresh with the [CLI][]. Review the steps in [Connecting to the API][].

:::info[Deploy Before Refreshing]

You can only refresh if you've deployed your pipeline.

:::

## Step 1: Create a Job

1. Go to **Jobs** in the Build sidebar.
2. Select the **+** sign to create a new Job.
3. You can add to your job by in multiple ways:
   1. Select **Edit** to add [selector queries][]. Selector queries allow you to select a subset of Nodes to be refreshed.
   2. Create subgraphs and then drag them into the Job tab. You can add multiple subgraphs to a Job using this method. For Subgraph naming, editing, and how Subgraphs relate to Jobs and commits, see [Using Subgraphs to Build Your Pipeline][].
4. Take note of your Job ID. In this example it's `jobID: 3`. You'll need the Job ID to run it as part of the refresh.

:::info[Add Subgraphs to Jobs]

You can add a subgraph to a Job by using a selector query or by [drag and drop](/docs/deploy-and-refresh/refresh/managing-jobs).

:::

### Parameters

Parameters inherit from the **Environment** and can be overridden at refresh time. Values overridden at refresh time are omitted from `runTimeParameters` in run results. See [Overwriting at Refresh][]. When you call [Trigger Job to Run][] and include `parameters`, that object is the full runtime parameter map for the Run, not a merge with environment defaults.

## Step 2: Commit Your Job

Jobs need to be committed into Git and deployed to an environment before they can be used. You can read more about making commits in our [Git Integration][] article.

## Step 3: Configure Your Environment

Go to **Build Settings > Environments** and check that the environment you want to refresh is configured. It should have:

- Your data platform credentials
- [Storage Locations and Storage Mappings][]
- [Parameters][] (optional)

## Step 4: Deploy Your Pipeline

Jobs can only be run if they have been deployed first. Review our [Deployment Overview][] to learn different ways to deploy your pipeline.

## Step 5: Run Your Jobs

Use the [Coalesce Scheduler][] (fastest), [API](/docs/deploy-and-refresh/refresh/refreshing-an-environment) or [CLI](/docs/deploy-and-refresh/refresh/refreshing-an-environment) to run a Job.

### Scheduler

The built-in [Coalesce Scheduler][] provides the easiest and fastest way to automate your environment refresh.

### API

Jobs can be triggered with the Start Job endpoint. [Trigger Job to Run][]. By only passing the `environmentID` and leaving the `jobID` out, you can refresh the entire environment. You can also use the `excludeNodesSelector` and `includeNodesSelector` to override the Jobs created. To avoid setting the selectors manually each run, we recommend using Jobs to save and manage nodes.

<Tabs>
  <TabItem value="job" label="Run a Job" default>

```bash
curl --request POST \
     --url https://app.coalescesoftware.io/scheduler/startRun \
     --header 'Authorization: Bearer YOUR-API-TOKEN' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "runDetails": {
    "parallelism": 16,
    "jobID": "4",
    "environmentID": "10"
  },
  "userCredentials": {
  # Use the credentials for your platform. Review the API documentation for more authentication examples.

    "snowflakeAuthType": "Basic",
    "snowflakeRole": "ACCOUNTADMIN",
    "snowflakeWarehouse": "COMPUTE_WH",
    "snowflakeUsername": "SOMEUSER",
    "snowflakePassword": "SOMEPASS"

    "platformKind": "databricks",
    "databricksClientSecret": "******",
    "databricksClientID": "*****",
    "databricksPath": "sql warehouse path",
    "databricksAuthType": "OAuthM2M"
  }
}
'
```

  </TabItem>

  <TabItem value="adhoc-job" label="Run a one-time Job">

You can also the `excludeNodesSelector` and `includeNodesSelector` to run a one-time Job. Leave out the `jobID`.

```bash
curl --request POST \
     --url https://app.coalescesoftware.io/scheduler/startRun \
     --header 'Authorization: Bearer YOUR-API-TOKEN' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "runDetails": {
    "parallelism": 16,
    "includeNodesSelector": "{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION } OR { location: SAMPLE name: ORDERS } OR { location: SAMPLE name: PART } OR { location: SAMPLE name: PARTSUPP } OR { location: SAMPLE name: REGION } OR { location: SAMPLE name: SUPPLIER } OR { location: QA name: STG_PARTSUPP } OR { location: PROD name: STG_PARTSUPP }",
    "environmentID": "10"
  },
  "userCredentials": {
  # Use the credentials for your platform. Review the API documentation for more authentication examples.
  
    "snowflakeAuthType": "Basic",
    "snowflakeRole": "ACCOUNTADMIN",
    "snowflakeWarehouse": "COMPUTE_WH",
    "snowflakeUsername": "SOMEUSER",
    "snowflakePassword": "SOMEPASS"

    "platformKind": "databricks",
    "databricksClientSecret": "******",
    "databricksClientID": "*****",
    "databricksPath": "sql warehouse path",
    "databricksAuthType": "OAuthM2M"
  }
}
```

</TabItem>

<TabItem value="refresh-env" label="Refresh an entire environment">

Leave out the `jobID` and selectors to refresh an entire environment.

```bash
curl --request POST \
     --url https://app.coalescesoftware.io/scheduler/startRun \
     --header 'Authorization: Bearer YOUR-API-TOKEN' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "runDetails": {
    "parallelism": 16,
    "environmentID": "10"
  },
  "userCredentials": {
  # Use the credentials for your platform. Review the API documentation for more authentication examples.
  
    "snowflakeAuthType": "Basic",
    "snowflakeRole": "ACCOUNTADMIN",
    "snowflakeWarehouse": "COMPUTE_WH",
    "snowflakeUsername": "SOMEUSER",
    "snowflakePassword": "SOMEPASS"

    "platformKind": "databricks",
    "databricksClientSecret": "******",
    "databricksClientID": "*****",
    "databricksPath": "sql warehouse path",
    "databricksAuthType": "OAuthM2M"
  }
}
'
```

</TabItem>

</Tabs>

### CLI

Refresh jobs can be triggered using our CLI tool `coa` using `coa refresh`. Learn more in our [CLI Commands][] documentation. By only passing the `environmentID` and leaving the `jobID` out, you can refresh the entire environment. You can also use the `excludeNodesSelector` and `includeNodesSelector` to override the Jobs created. To avoid setting the selectors manually each run, we recommend using Jobs to save and manage nodes.

:::info[CLI Set Up]

Make sure to setup your CLI. Review [CLI set up][].

:::

<Tabs>
  <TabItem value="cli-job" label="Run a Job" default>

  This example assumes you are using a `coa` config file.

  ```bash
    coa refresh --environmentID 1  --jobID 4
  ```

  </TabItem>

  <TabItem value="cli-job-one-time" label="Run a one-time Job">

  You can also the `excludeNodesSelector` and `includeNodesSelector` to run a one-time Job. Leave out the `jobID`.

  ```bash
  coa refresh --environmentID 1 --include '{ location: SAMPLE name: CUSTOMER } OR { location: SAMPLE name: LINEITEM } OR { location: SAMPLE name: NATION } OR { location: SAMPLE name: ORDERS } OR { location: SAMPLE name: PART } OR { location: SAMPLE name: PARTSUPP } OR { location: SAMPLE name: REGION } OR { location: SAMPLE name: SUPPLIER } OR { location: QA name: STG_PARTSUPP } OR { location: PROD name: STG_PARTSUPP }'
  ```

  </TabItem>

  <TabItem value="cli-refresh-env" label="Refresh an entire environment">

  Leave out the `jobID` and selectors to refresh an entire environment.

  ```bash
  coa refresh --environmentID 1 
  ```

  </TabItem>

</Tabs>

## Coalesce App

Only existing, deployed Jobs can be run from the Coalesce App.

Go to the Deploy page, and next to each deploy, click on the menu, and select **Run Job**.

[Connecting to the API]: /docs/api/authentication
[Trigger Job to Run]: /docs/api/runs/startrun
[Overwriting at Refresh]: /docs/deploy-and-refresh/parameters/rtp-refresh
[CLI Commands]: /docs/coa/version-733-and-above/coa-commands
[CLI set up]: /docs/coa
[Git Integration]: /docs/setup-your-project/setup-version-control
[CLI]: /docs/coa
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[Parameters]: /docs/deploy-and-refresh/parameters/
[Deployment Overview]: /docs/deploy-and-refresh/deploy/
[selector queries]: /docs/reference/selector/
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Refreshing Your Pipeline Using the Coalesce Scheduler

The Coalesce Scheduler gives you the ability to schedule data pipeline refreshes directly in the Coalesce app. You no longer need to use a third-party solution simplifying your IT dependency and allowing you to generate data insights faster.

## Before You Begin

- You must have an [Environment Admin][] role.
- The [Environment][] must have credentials added. The Scheduler uses the credentials of the user that created or last modified the Scheduled Job.
- A Job must be **deployed** before it can be scheduled. The Coalesce Scheduler only runs deployed Jobs, not workspace-only changes that are not yet committed and deployed. Learn more about [deploying your pipeline][].

:::warning[Deleting or Changing Credentials]

If you change credentials, including changing authentication type, you'll need to edit each Job to use the new method. If not, the Jobs will fail.

:::

## Subgraph Changes, Commits, and Scheduled Runs

When you adjust a subgraph and that change should affect what a Job refreshes, commit the Job (together with your other Project changes) to version control, then deploy to the environment. Scheduled runs use the **deployed** Job definition. Until you commit and deploy, the Coalesce Scheduler keeps running the last deployed version.

For how subgraphs relate to Jobs and selector queries, see [Using Subgraphs to Build Your Pipeline][]. For the commit and deploy sequence, see [Creating and Run Jobs][].

## How to Schedule a Job

1. In your Coalesce App, go to the **Deploy** page.
2. Next to any Job, click the dropdown and select **View Scheduled Jobs**. Then click **Create Job Schedule**.

    

3. **Configuration**: Fill out the Job Schedule form. If you click on the Job to create a Job schedule, the Project, Environment, and Job Name will be pre-filled.
   1. **Project** - You’ll be able to create Jobs for any Project you have Environment Admin access to.
   2. **Environment** - Select the Environment in the Project.
   3. **Job Name** - Select an existing Job in the Environment.
   4. **Job Schedule Name** - Choose a unique name to make it easy to identify later.
   5. **Cron Expression** - Enter a [Cron][] expression in UTC time for the Job frequency.
   6. **Timezone** - Set the timezone for Job runs.
   7. **Automatic run on save** - When this option is on, the Job runs once as soon as you save the schedule, then continues on the cron schedule. Turn it off if the first run should wait until the next scheduled time instead.
   8. **Retry on Failure** - Toggle to retry the entire Job if it fails. This will retry the entire Job.
      1. **Number of Retries** - Enter the number of times a Job will attempt to run if it fails the first time.
      2. **Time Between Retries** - Enter the number of minutes between each retry.
      3. **Retry Job From** - Beginning will run the entire Job again. Point of Failure will retry the Job from when it failed.

    

4. **Notifications:** Coalesce will send an email notification for successful, retries, and failed Jobs.
   1. **Email on**: Choose if you want emails to be sent for Success only, Errors only (includes retries), or both.
   2. **Email Addresses:** Each email address should be entered on a new line.

    

5. **Parameters** - Enter runtime parameters as JSON for each run this schedule triggers. If the Environment has [default parameters][], they appear in the editor and you can override them for this schedule. Coalesce validates the JSON the same way as in the Deploy Wizard. See [Parameters][] for format and usage in templates.

    

6. Save the Job. You’ll be able to see the Job on the **Job Schedules** page.

:::info[Refresh the Page]

Refresh the page to see the status of a Scheduled Job.

:::

## View Scheduled Jobs

You can view Scheduled Jobs in two ways.

1. Select **View Schedules** next to the Job you want to view the schedules for.

    

2. Select **View Schedules Jobs** next to Deploy to see all Scheduled Jobs for the Environment.

You’ll be able to view a list of Scheduled Jobs on the **Jobs Schedule** page.

## Edit Scheduled Jobs

1. On the **Jobs Schedule** page, in the **Actions** column, select **Edit**.
2. You can edit the **Job Schedule Name**, **Cron Expression**, **Automatic run on save**, and **Parameters**. If you need to edit the Project, Environment, or Job, create a new Job Schedule.

:::warning[Editing Jobs Created By Another User]

Editing Scheduled Jobs created by another user, will use your Coalesce and Data Platform credentials to run the job

:::

## Scheduled Jobs Notifications and Alerts

Notifications can be sent using email. You can configure emails to be sent for Success, Error, or Both.

You can also edit the notifications at any time by editing the Job Schedule.

Notifications are sent for:

- Each successful Job Schedule.
- Each failed Job Schedule.

## Pause or Resume Scheduled Jobs

If you need to stop a schedule without removing it, pause it instead of deleting the Job Schedule.

1. On the **Jobs Schedule** page, in the **Actions** column, select **Pause**.
2. To start runs again, select **Resume** from the same menu.

While a schedule is paused, **Next Execution** shows **Paused.** The Job does not run until you resume.

## Delete Job Schedules

To delete a Job Schedule, go to the **Jobs Schedule** page, in the **Actions** column, select **Delete**. Pausing is a better option when you plan to use the same schedule again soon.

- You can delete a Scheduled Job at any time.

:::warning[Deleting a Job]

If you delete a Job attached to a schedule, then commit the deleted Job change to the deploy branch, then try to deploy, you’ll get a warning message in the Review Plan step. The Jobs Schedules for the deleted Job will be removed at the next scheduled run and won’t be part of the deploy.

:::

## Scheduled Job Status and Results

On the Job Schedule page, you can see:

- **Last Execution** - Last time the Scheduled Job was run displayed in system time.
- **Next Execution** - Next Scheduled Job run time, or **Paused** when the schedule is paused.
- **Run Status -** Most recent status of the Scheduled Job.
  - **Successful**: The Job ran without issue.
  - **Running**: Job is currently running.
  - **Failed**: The Job failed to run.
  - **Test failed**: The Job ran, but the [Node tests][] failed.

To view the results click on the **Run Status** of any scheduled job.

## Scheduler Permissions

- You must be an [Environment Admin][] to create Scheduled Jobs.
- You must be an [Environment Reader][] to view Job Schedules.

## Important Scheduling Information

- You can create multiple schedules for each Job.
- Refreshed All Nodes and Ad-Hoc jobs aren’t part of the Job Scheduler and can only be run using the [CLI or API][].
- Schedules use [Cron][] to run and they are in UTC time.
- By default, a new or edited schedule runs once when you save if **Automatic run on save** is on. Turn that option off so the first run waits for the next cron execution.
- Selecting Run Job from the Jobs menu will run the Job one-time. It will not change the Scheduled Job.
- You can pause a Scheduled Job from the **Actions** column on the **Jobs Schedule** page instead of deleting it or changing the cron expression.
- Scheduled Jobs run in parallel. If more than one Job is scheduled for the same time, they will run at the same time. If there is an ad hoc refresh during a Scheduled Job, they will run at the same time.
- Scheduled Jobs aren’t committed to your Git provider.
- The Coalesce Scheduler only runs **deployed** Jobs. Subgraph or Job changes in the Workspace do not affect scheduled runs until you commit, deploy, and the environment has the updated definition.
- Each Job Schedule can define its own parameter JSON so you can run the same deployed Job on different schedules with different runtime values without creating duplicate Environments.

## Resources

- [Cron Reference][]
- [Parameters][]

[Environment Admin]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Environment]: /docs/get-started/coalesce-fundamentals/environments
[deploying your pipeline]: /docs/deploy-and-refresh/deploy/
[Environment Reader]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Cron]: /docs/reference/cron-reference
[Cron Reference]: /docs/reference/cron-reference
[CLI or API]: /docs/deploy-and-refresh/refresh/#types-of-jobs
[Node tests]: /docs/build-your-pipeline/data-quality-testing/testing-nodes
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs
[Creating and Run Jobs]: /docs/deploy-and-refresh/refresh/refreshing-an-environment
[default parameters]: /docs/deploy-and-refresh/parameters/rtp-default
[Parameters]: /docs/deploy-and-refresh/parameters/

---

## Managing Time Zones, Business Holidays, and Daylight Saving in Coalesce Scheduling

Learn how to handle different timezones, daylight savings, and holidays when scheduling Jobs in Coalesce.

## Manually Adjust

If you're using the Coalesce Scheduler, you can manually change the schedule. For example, Change `0 11 * * *` (6 AM EST) to `0 10 * * *` (6 AM EDT).

## Using Date Dimension Node

Using a Date Dimension Node from our Functional Node Types package, create a [Node level test](/docs/build-your-pipeline/data-quality-testing/testing-nodes) on the first Nodes in your pipeline. It will check for certain conditions and if it's not a business day or the right time, then the test will fail and the pipeline won't run.

Make sure **Continue on Failure** is turned off so it fails and **Run Before**.

```sql
-- Example test to check if current date is a business day
SELECT *
FROM {ref('GENERAL_DEV','DIM_DATE_NODE ') } "DIM_DATE_NODE"
WHERE calendar_date = CURRENT_DATE
AND is_business_day = FALSE
```

:::info[Date Dimension Node Package]

- [Databricks](/docs/marketplace/package/coalesce_databricks_functional-node-types#date-dimension)
- [Snowflake](/docs/marketplace/package/coalesce_snowflake_functional-node-types#date-dimension)

:::

## Using a Third Party Tool

If you manage your scheduling using a tool like Airflow or GitHub actions, you can add a script to check the date and time.

These are just examples and should be adjusted for your use.

### GitHub Actions

- Always schedule jobs in UTC in the GitHub Actions workflow.
- In your Python script:
  - Use `pytz` or `zoneinfo` to convert UTC to local time. There are multiple Python libraries available.
  - Check if today is a business day using a holiday calendar.
  - Run Coalesce CLI only if conditions are met.

<Tabs>
  <TabItem value="github-workflow" label="GitHub Actions" default>

 ```yaml
name: Deploy to Environment

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  COA_VERSION: latest # Pin a specific version in production
  TZ_REGION: "America/New_York" # Local business timezone

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 22.19.0

      # Fetch config secret from GitHub Actions secret repository
      - run: echo '${{ secrets.COA_CONFIG }}' >> COA_CONFIG

      # Install Coalesce CLI if not already present
      - run: npm list | grep "@coalescesoftware/coa@${{ env.COA_VERSION }}" || npm install @coalescesoftware/coa@${{ env.COA_VERSION }}

      # Print version number
      - run: npx coa --version

      # Install Python for holiday/DST checks
      - uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      - run: pip install holidays pytz

      # Run Python script: checks holidays/DST + runs coa commands
      - name: Run Coalesce Deployment with Business-Day Check
        run: python .github/workflows/coalesce_ci.py
        env:
          COA_CONFIG: ${{ secrets.COA_CONFIG }}
          TZ_REGION: ${{ env.TZ_REGION }}

```

  </TabItem>
  <TabItem value="github-action-python-script" label="coalesce_ci.py">

```python

from datetime import datetime

def is_business_day():
    tz_region = os.getenv("TZ_REGION", "UTC")
    tz = pytz.timezone(tz_region)
    today = datetime.now(tz).date()

    us_holidays = holidays.US()
    if today.weekday() >= 5 or today in us_holidays:
        print(f"Skipping run: {today} is weekend/holiday")
        return False
    return True

def run_coalesce():
    coa_config = os.getenv("COA_CONFIG")
    if not coa_config:
        sys.exit("Missing COA_CONFIG environment variable.")

    # Write config to temp file
    with open("coa_config.yaml", "w") as f:
        f.write(coa_config)

    # Plan + Deploy
    subprocess.check_call("npx coa plan --config coa_config.yaml --out ./coa-plan --debug", shell=True)
    subprocess.check_call("npx coa deploy --config coa_config.yaml --plan ./coa-plan --debug", shell=True)

if __name__ == "__main__":
    if is_business_day():
        run_coalesce()
    else:
        sys.exit(0)
```

  </TabItem>
</Tabs>

### Airflow

Here is an example that:

- Always runs at the same local business time regardless of DST.
- Checks that today is both a weekday and not a holiday.
- Only runs Coalesce jobs if conditions are satisfied.

```python
from datetime import datetime, timedelta

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python_operator import PythonOperator

# Local timezone ensures DST consistency
local_tz = pendulum.timezone("America/New_York")
us_holidays = holidays.US()

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 1, 1, tzinfo=local_tz),
    'retries': 1,
    'retry_delay': timedelta(minutes=2),
}

dag = DAG(
    'coalesce_trigger_with_businessday_check',
    default_args=default_args,
    description='Trigger Coalesce only on business days at consistent local time',
    schedule_interval="0 6 * * *",  # 6 AM local time
    catchup=False,
)

def check_business_day(**context):
    today = datetime.now(local_tz).date()
    if today.weekday() >= 5 or today in us_holidays:
        raise ValueError(f"Skipping run: {today} is weekend or holiday")

def trigger_coalesce(**context):
    token = Variable.get('AIRFLOW_VAR_COALESCE_TOKEN')
    sf_key = str(Variable.get('AIRFLOW_VAR_SNOWFLAKE_PRIVATE_KEY'))

    url = "https://app.coalescesoftware.io/scheduler/startRun"
    payload = {
        "runDetails": {
            "parallelism": 16,
            "environmentID": "3",  # replace with your environment ID
            "jobID": "3"           # replace with your job ID
        },
        "userCredentials": {
            "snowflakeAuthType": "KeyPair",
            "snowflakeKeyPairKey": sf_key
        }
    }
    headers = {
        "accept": "application/json",
        "content-type": "application/json",
        "Authorization": f"Bearer {token}"
    }
    resp = requests.post(url, headers=headers, json=payload)
    resp.raise_for_status()

    run_counter = resp.json().get('runCounter')
    if not run_counter:
        raise ValueError("Missing runCounter in response")

    # Push runCounter to XCom for status check
    context['ti'].xcom_push(key='run_counter', value=run_counter)

def check_status(**context):
    token = Variable.get('AIRFLOW_VAR_COALESCE_TOKEN')
    run_counter = context['ti'].xcom_pull(key='run_counter', task_ids='trigger_job')

    url = f"https://app.coalescesoftware.io/scheduler/runStatus?runCounter={run_counter}"
    headers = {"Authorization": f"Bearer {token}", "accept": "application/json"}
    resp = requests.get(url, headers=headers)
    resp.raise_for_status()

    status = resp.json().get('status')
    if status != 'Success':
        raise Exception(f"Coalesce job failed with status: {status}")

# Define tasks
task_check_day = PythonOperator(
    task_id='check_business_day',
    python_callable=check_business_day,
    provide_context=True,
    dag=dag,
)

task_trigger = PythonOperator(
    task_id='trigger_job',
    python_callable=trigger_coalesce,
    provide_context=True,
    dag=dag,
)

task_status = PythonOperator(
    task_id='check_status',
    python_callable=check_status,
    provide_context=True,
    dag=dag,
)

# Set task dependencies
task_check_day >> task_trigger >> task_status
```

---

## Orchestrate Deploys and Refreshes with GitLab and the CLI

GitLab is a popular open-source web-based DevOps platform that provides a comprehensive set of tools for software development, version control, continuous integration/continuous delivery (CI/CD), and project management. It was created by GitLab Inc. and is written in Ruby.

In this tutorial, we go through how to securely set up Coalesce Deployments and Refresh using GitLab.

## Before You Begin

You'll need:

- **GitLab**:  [GitLab][] account with admin privileges to configure pipelines and manage secrets.
- **Project**: A [Project][] with version control configured using your chosen platform.
- **Environment**: At least one [Environment][] configured for deployment.
- **Access Token**: Your Coalesce [access token][] for the CLI configuration file.

### Create and Configure an Environment

1. Create a new [Environment](/docs/setup-your-project/create-your-environments) for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
2. Configure authentication for your Environment based on your data platform:
   - **Snowflake**: Set up [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) with a service account.
   - **Databricks**: Set up [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) with a service principal.
3. Configure Storage Mappings for your Environment.
4. Deploy the Environment to verify configuration.

### Data Platform Permissions

#### Snowflake

- `USAGE` privilege on the database and schema.
- `SELECT`, `INSERT`, `UPDATE`, and `DELETE` privileges on tables involved in transformations.
- `EXECUTE` privilege for procedures or functions referenced by Coalesce.
- Ability to create [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) (recommended for automated deployments).

#### Databricks

- SQL Warehouse configured and accessible.
- Appropriate catalog and schema permissions.
- Ability to create [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) (recommended for automated deployments).

### Configure Your CLI

The Coalesce CLI orchestrates your deployments and refreshes within CI/CD pipelines. You'll create a configuration file with credentials required to authenticate the CLI.

1. Follow the instructions in [CLI Setup](/docs/coa/version-732-and-below/coa-setup) to install the CLI.
2. Create a new text file, for example, `coa_config.txt`, and save it securely. This file contains the authentication credentials your pipeline will use.

Your configuration file should include profiles for each Environment.

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    platformKind=Databricks
    databricksAuthType=OAuthM2M
    databricksClientID=client_id_here
    databricksClientSecret=client_secret_here
    databricksAccountHost=https://accounts.cloud.databricks.com
    databricksWorkspaceHost=https://your-workspace.cloud.databricks.com
    databricksPath=/sql/1.0/warehouses/abc123xyz
    environmentID=456
    ```

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    snowflakeAuthType=Keypair
    snowflakeUsername=service_account_user
    snowflakeRole=TRANSFORMER_ROLE
    snowflakeWarehouse=TRANSFORM_WH
    snowflakeAccount=xy12345.us-east-1
    environmentID=123
    ```

  </TabItem>

</Tabs>

**Configuration Notes:**

- Create separate profiles using `[profile_name]` for different Environments.
- Use service accounts or service principals rather than individual user accounts.
- For Snowflake, include the private key path when using key-pair authentication.
- Store this file securely - it contains sensitive credentials.

## Creating Deployment Pipelines in GitLab

1. Within GitLab, Click **Projects** and navigate to your Project/Coalesce repo where you want to build your CI/CD pipeline.

    

2. Navigate to **Settings > CI/CD**. Expand **Secure Files**.

    

3. Click **Upload File** and upload the secure CLI config file that you saved locally.

    

4. Navigate to **Build > Pipeline Editor**.

    

5. Ensure that the pipeline that you are trying to build “lives” in the branch you are trying to configure an automated deployment for. You can also edit the pipeline config to reflect the trigger branch for the pipeline you want to deploy.
6. Copy and paste the pipeline template below:

        ```yaml
        image: node:20.0.0

        stages:          # List of stages for jobs, and their order of execution
        - download-coa
        - download-secure-file
        - print-version
        - coa-plan
        - coa-deploy

        variables:
        COA_VERSION: latest # Modify this value to reflect your CLI configuration file
        CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt # Modify this value to reflect your CLI configuration file
        COA_PROFILE: default # Modify this value to reflect your CLI configuration file

        cache:
        paths:
            - node_modules/

        download-coa:    # This job downloads the Coalesce CLI.
        stage: download-coa
        script:
            - npm install @coalescesoftware/coa@$COA_VERSION
        only:
            - main

        download-secure-file:       # This job downloads the secure file.
        stage: download-secure-file
        variables:
            SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
        script:
            - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
            - ls -lah node_modules/demo-file-folder/
        only:
            - main

        print-version:   # This job prints the coa cli version.
        stage: print-version
        script:
            - npm coa --version
        only:
            - main

        coa-plan:        # This job creates the deployment plan.
        stage: coa-plan
        script:
            - npx coa plan --config  /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --out /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
        only:
            - main

        coa-deploy:      # This deploys the commit in the branch based on the configuration of the coa_plan.json created in the coa-plan step.
        stage: coa-deploy
        script:
            - npx coa deploy --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --plan /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/coa-plan.json --debug
        only:
            - main
        ```

7. In the above code, the three variables `COA_VERSION`, `CONFIG_FILE_NAME`, and `COA_PROFILE` define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan. They have been given default values of `latest`, `coa_config_gitlab_cicd.txt`, and `default` but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.
8. Once this pipeline is saved and committed, any successful merge or commit to the main branch of this Coalesce will result in the execution of this pipeline, and subsequent deployment into the specified Coalesce environment.
9. This is a simple deployment pipeline, for more advanced configuration options, please see our [CLI documentation][].

## Using GitLab Pipelines to Schedule Refreshes

You can also schedule refreshes with GitLab pipelines using a similar script to the one above. In addition, GitLab has a built-in scheduler that you can use to run the refresh at a cadence of your choosing.

1. Create a new pipeline using the previous section, Creating Deployment Pipelines in GitLab. Except in Step 6, copy the following code to set up your refresh pipeline.

        ```yaml
        image: node:20.0.0

        stages:          # List of stages for jobs, and their order of execution
        - download-coa
        - download-secure-file
        - print-version
        - coa-refresh

        variables:
        COA_VERSION: latest
        CONFIG_FILE_NAME: coa_config_gitlab_cicd.txt
        COA_PROFILE: default
        JOBID:

        cache:
        paths:
            - node_modules/

        download-coa:    # This job downloads the Coalesce CLI.
        stage: download-coa
        script:
            - npm install @coalescesoftware/coa@$COA_VERSION
        only:
            - main

        download-secure-file:       # This job downloads the secure file.
        stage: download-secure-file
        variables:
            SECURE_FILES_DOWNLOAD_PATH: 'node_modules/demo-file-folder'
        script:
            - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
            - ls -lah node_modules/demo-file-folder/

        print-version:   # This job prints the coa cli version.
        stage: print-version
        script:
            - npm coa --version

        coa-refresh:     # This refreshes the specified environment.
        stage: coa-refresh
        script:
            - npx coa refresh --config /builds/coalesce1/GitLab_ci_cd/node_modules/demo-file-folder/$CONFIG_FILE_NAME --profile $COA_PROFILE --jobID $JOBID
        ```

2. In the above code, the three variables `COA_VERSION`, `CONFIG_FILE_NAME`, and `COA_PROFILE` define the CLI version, the name of the configuration file, and the profile as configured in the configuration file that you want to use to generate the deployment plan.

    They have been given default values of `latest`, `coa_config_gitlab_cicd.txt`, and `default` but you may modify these to reflect the name of your configuration file, the version of the CLI that you want to install, and the profile corresponding with the environment that you want to deploy into.

    The optional `jobID` variable specifying the ID of the job that you want to refresh in the environment specified. If the `$JOBID` parameter is removed, all the deployed nodes in the Environment will be refreshed.

3. If you want to refresh multiple jobs sequentially, you will have to add additional stages that copy the `coa-refresh` stage  and specify a different `$JOB_ID`for each one. For additional configuration options, please refer to the [CLI documentation][].
4. Once your pipeline is ready, you can set a schedule by navigating to **Pipeline Schedules > Create a New Pipeline Schedule** and set the desired cadence for your pipeline.

    

    

5. Click **Create pipeline schedule** and verify that the schedule and the pipeline branch is set properly in the next screen.

    

[CLI documentation]: /docs/coa/version-733-and-above/coa-commands
[Project]: /docs/setup-your-project/create-your-project
[Environment]: /docs/setup-your-project/create-your-environments
[GitLab]: /docs/setup-your-project/setup-version-control#gitlab
[access token]: /docs/api/authentication

---

## Orchestrate Deployments with Bitbucket

Bitbucket is a version control tool offered by Atlassian that mimics the functionality of tools like GitHub and GitLab. Coalesce supports using Bitbucket repositories for project version control as well as CI/CD pipelines for automated deployment. In this article, we will go over how to set up a sample Bitbucket pipeline with Coalesce deployments.

## Before You Begin

You'll need:

- **GitLab**:  [Bitbucket][] account with admin privileges to configure pipelines and manage secrets.
- **Project**: A [Project][] with version control configured using your chosen platform.
- **Environment**: At least one [Environment][] configured for deployment.
- **Access Token**: Your Coalesce [access token][] for the CLI configuration file.

### Create and Configure an Environment

1. Create a new [Environment](/docs/setup-your-project/create-your-environments) for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
2. Configure authentication for your Environment based on your data platform:
   - **Snowflake**: Set up [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) with a service account.
   - **Databricks**: Set up [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) with a service principal.
3. Configure Storage Mappings for your Environment.
4. Deploy the Environment to verify configuration.

### Data Platform Permissions

#### Snowflake

- `USAGE` privilege on the database and schema.
- `SELECT`, `INSERT`, `UPDATE`, and `DELETE` privileges on tables involved in transformations.
- `EXECUTE` privilege for procedures or functions referenced by Coalesce.
- Ability to create [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) (recommended for automated deployments).

#### Databricks

- SQL Warehouse configured and accessible.
- Appropriate catalog and schema permissions.
- Ability to create [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) (recommended for automated deployments).

### Configure Your CLI

The Coalesce CLI orchestrates your deployments and refreshes within CI/CD pipelines. You'll create a configuration file with credentials required to authenticate the CLI.

1. Follow the instructions in [CLI Setup](/docs/coa/version-732-and-below/coa-setup) to install the CLI.
2. Create a new text file, for example, `coa_config.txt`, and save it securely. This file contains the authentication credentials your pipeline will use.

Your configuration file should include profiles for each Environment.

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    platformKind=Databricks
    databricksAuthType=OAuthM2M
    databricksClientID=client_id_here
    databricksClientSecret=client_secret_here
    databricksAccountHost=https://accounts.cloud.databricks.com
    databricksWorkspaceHost=https://your-workspace.cloud.databricks.com
    databricksPath=/sql/1.0/warehouses/abc123xyz
    environmentID=456
    ```

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    snowflakeAuthType=Keypair
    snowflakeUsername=service_account_user
    snowflakeRole=TRANSFORMER_ROLE
    snowflakeWarehouse=TRANSFORM_WH
    snowflakeAccount=xy12345.us-east-1
    environmentID=123
    ```

  </TabItem>

</Tabs>

**Configuration Notes:**

- Create separate profiles using `[profile_name]` for different Environments.
- Use service accounts or service principals rather than individual user accounts.
- For Snowflake, include the private key path when using key-pair authentication.
- Store this file securely - it contains sensitive credentials.

## Configure Bitbucket

Bitbucket does not allow for direct file uploads, you'll securely pass this file into the Bitbucket pipelines is through base64 encoding the file locally.

1. Navigate to the folder that you saved your `cli_config.txt` file using the command line and then execute the following command. This will encode the `coa_config` file as a string that you can pass as a secure Bitbucket variable.

    <Tabs>
    <TabItem value="windows" label="Window" default>

    `certutil -encode coa_config.txt encoded_config.base64`

    </TabItem>
    <TabItem value="mac" label="macOS">

    `base64 -i coa_config.txt > encoded_config.base64`
    </TabItem>
    <TabItem value="linux" label="Linux">

    `base64 coa_config.txt > encoded_config.base64`
    </TabItem>
    </Tabs>

2. In your Bitbucket repo, navigate to **Repository Settings > Repository Variables**.

    
    

3. Add the following variables and make sure Secured is checked.

    `SECURE_FILE_BASE64 = <your base64 encoded coa_config.txt file>`

    

## Build Your Pipeline

Now that you have your variables set up, you are ready to start building the CI/CD pipeline. In this section, you'll set up a CI/CD pipeline to trigger upon successful merges, commits, or pull requests in the main branch. If you want to modify this behavior, you can add an additional section under branches in the code in the next step to dictate the pipeline's behavior upon merges into other branches in your project.

- This pipeline runs a set of steps:
  - Downloads the Coalesce CLI
  - Prints the Coalesce CLI version
  - Generates the Coalesce deployment plan
  - Executes the Coalesce deployment to the default profile specified

Within your main branch in your repository, navigate to **Pipelines > Create a New Pipeline** and copy/paste the following code:

```yaml
image: node:22

definitions:
  caches:
    npm: node_modules

pipelines:
  branches:
    main:
      - step:
          name: 'Download Coalesce CLI'
          caches:
            - npm
          script:
            - export COA_VERSION=${COA_VERSION:-latest}
            - npm install @coalescesoftware/coa@$COA_VERSION

      - step:
          name: 'Print Version'
          caches:
            - npm
          script:
            - npx coa --version

      - step:
          name: 'Generate Deployment Plan'
          caches:
            - npm
          script:
            # Decode base64 config file
            - echo "$SECURE_FILE_BASE64" | base64 -d > config.txt

            # Verify config file is not empty
            - |
              if [ ! -s config.txt ]; then
                echo "Error: Config file is empty"
                exit 1
              fi

            # Generate deployment plan
            - npx coa plan --config config.txt --profile default --debug
          artifacts:
            - config.txt
            - coa-plan.json

      - step:
          name: 'Execute Deployment'
          caches:
            - npm
          script:
            - npx coa deploy --config config.txt --plan coa-plan.json --debug
```

[Bitbucket]:https://confluence.atlassian.com/bitbucketserver/using-repository-permissions-776639771.html
[Project]: /docs/setup-your-project/create-your-project
[Environment]: /docs/setup-your-project/create-your-environments
[access token]: /docs/api/authentication

---

## Orchestrate Deploys With GitHub Actions and the CLI

:::warning[GitHub Requires Node 20]

[Node 16](https://github.com/nodejs/Release/#end-of-life-releases) is reached the end of it's life. GitHub now requires Node 20.

Learn more in [GitHub Actions: Transitioning from Node 16 to Node 20](https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/).

:::

## Before You Begin

You'll need:

- **GitHub**:  [GitHub][] account with admin privileges to configure pipelines and manage secrets.
- **Project**: A [Project][] with version control configured using your chosen platform.
- **Environment**: At least one [Environment][] configured for deployment.
- **Access Token**: Your Coalesce [access token][] for the CLI configuration file.

### Create and Configure an Environment

1. Create a new [Environment](/docs/setup-your-project/create-your-environments) for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
2. Configure authentication for your Environment based on your data platform:
   - **Snowflake**: Set up [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) with a service account.
   - **Databricks**: Set up [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) with a service principal.
3. Configure Storage Mappings for your Environment.
4. Deploy the Environment to verify configuration.

### Data Platform Permissions

#### Snowflake

- `USAGE` privilege on the database and schema.
- `SELECT`, `INSERT`, `UPDATE`, and `DELETE` privileges on tables involved in transformations.
- `EXECUTE` privilege for procedures or functions referenced by Coalesce.
- Ability to create [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) (recommended for automated deployments).

#### Databricks

- SQL Warehouse configured and accessible.
- Appropriate catalog and schema permissions.
- Ability to create [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) (recommended for automated deployments).

### Configure Your CLI

The Coalesce CLI orchestrates your deployments and refreshes within CI/CD pipelines. You'll create a configuration file with credentials required to authenticate the CLI.

1. Follow the instructions in [CLI Setup](/docs/coa/version-732-and-below/coa-setup) to install the CLI.
2. Create a new text file, for example, `coa_config.txt`, and save it securely. This file contains the authentication credentials your pipeline will use.

Your configuration file should include profiles for each Environment.

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    platformKind=Databricks
    databricksAuthType=OAuthM2M
    databricksClientID=client_id_here
    databricksClientSecret=client_secret_here
    databricksAccountHost=https://accounts.cloud.databricks.com
    databricksWorkspaceHost=https://your-workspace.cloud.databricks.com
    databricksPath=/sql/1.0/warehouses/abc123xyz
    environmentID=456
    ```

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

    ```txt
    [production]
    token=your_coalesce_token
    domain=https://app.coalesce.io
    snowflakeAuthType=Keypair
    snowflakeUsername=service_account_user
    snowflakeRole=TRANSFORMER_ROLE
    snowflakeWarehouse=TRANSFORM_WH
    snowflakeAccount=xy12345.us-east-1
    environmentID=123
    ```

  </TabItem>

</Tabs>

**Configuration Notes:**

- Create separate profiles using `[profile_name]` for different Environments.
- Use service accounts or service principals rather than individual user accounts.
- For Snowflake, include the private key path when using key-pair authentication.
- Store this file securely - it contains sensitive credentials.

## Upload Actions Secret

To follow best practices, Coalesce recommends that you upload your [COA config file][] as a secure file within the GitHub Actions settings.

1. From within your GitHub project, go to **Settings > Secrets > Actions**

    

2. Click **New repository secret**.
3. Provide a name for it. In this example we will call it `COA_CONFIG`.
4. Find and open your COA `config` file. Default location is `~/.coa`.
5. Copy its contents into the **Value** field.
6. Click **Add Secret** to save.

## Example Deploy on Merge Workflow

1. From within your GitHub project, go to **Actions** and click **New workflow**.
2. Choose **Simple workflow** > **Configure**.
3. You will be presented with a template YAML file. Provide it a name and replace its content with the Deploy Workflow code below.

    ```yaml title="Deploy Workflow"
    name: Deploy to Environment

    on:
    push:
        branches:
        - main

    workflow_dispatch:
    env:
    COA_VERSION: latest # Before utilizing this pipeline in production scenarios, it is advised to pin a specific version. This practice aims to reduce potential disruptions. You can find the released versions on NPM at this link: https://www.npmjs.com/package/@coalescesoftware/coa 
    jobs:
    deploy:
        runs-on: ubuntu-latest
        
        steps:
        - uses: actions/checkout@v3
        - uses: actions/setup-node@v3
        with:
            node-version: 22.19.0

        # Fetch config secret from GitHub Actions secret repository, then write coa config to a temporary file
        - run: echo '${{ secrets.COA_CONFIG }}' >> COA_CONFIG
        
        # Install Coalesce CLI tool, if not installed already
        - run: npm list | grep "@coalescesoftware/coa@${{ env.COA_VERSION }}" || npm install @coalescesoftware/coa@${{ env.COA_VERSION }}

        # Print version number
        - run: npx coa --version

        # Create Deployment Plan
        - run: npx coa plan --config COA_CONFIG --out ./coa-plan --debug

        # Deploy Plan
        - run: npx coa deploy --config COA_CONFIG --plan ./coa-plan --debug
    ```

4. Commit this file to your repository.
5. You'll now see your workflow, and it will automatically run when a commit is merged to the `main` branch. You can also manually run it by clicking **Run workflow**.

## Example Scheduled Refresh Workflow

The following instructions provide an example workflow where an Environment is refreshed hourly using a cron job.

1. From within your GitHub project, go to **Actions** and click **New workflow**.
2. Choose **Simple workflow** > **Configure**.
3. You will be presented with a template YAML file. Provide it a name and replace its content with the Refresh Workflow code below.

    ```yaml title="Refresh Workflow"
    name: Refresh Data

    on:
    # Set a schedule using cron
    schedule:
    - cron: "0 * * * *"
    
    # This option allows the workflow to be triggered manually
    workflow_dispatch:
    
    env:
    COA_VERSION: latest # Versions released on NPM here: https://www.npmjs.com/package/@coalescesoftware/coa
    
    jobs:
    build:
        runs-on: ubuntu-latest
        
        steps:
        - uses: actions/checkout@v3
        - uses: actions/setup-node@v3
        with:
            node-version: 22.19.0
        
        # Fetch coa config secret from GitHub Actions secret repository, then write coa config to a temporary file
        - run: echo '${{ secrets.COA_CONFIG }}' >> COA_CONFIG
        
        # Install Coalesce CLI tool, if not installed already
        - run: npm list | grep "@coalescesoftware/coa@${{ env.COA_VERSION }}" || npm install @coalescesoftware/coa@${{ env.COA_VERSION }}

        # Execute coa CLI, starting coa refresh
        - run: npx coa refresh --config COA_CONFIG
    ```

4. Commit this file to your repository.
5. You'll now see your workflow, and it will automatically run every hour. You can also manually run it by clicking **Run workflow**.

[COA config file]: /docs/coa/version-732-and-below/coa-setup
[Project]: /docs/setup-your-project/create-your-project
[Environment]: /docs/setup-your-project/create-your-environments
[GitHub]: /docs/setup-your-project/setup-version-control#github
[access token]: /docs/api/authentication

---

## Orchestrate Jobs With GitHub Actions and the API

:::warning[GitHub Requires Node 20]

[Node 16](https://github.com/nodejs/Release/#end-of-life-releases) has reached the end of its life. GitHub now requires Node 20.

Learn more in [GitHub Actions: Transitioning from Node 16 to Node 20](https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/).

:::

GitHub Actions serves as a comprehensive platform for continuous integration and continuous delivery (CI/CD), enabling the automation of your build, testing, and deployment processes. It empowers you to design workflows that automatically build and test each pull request submitted to your repository, and seamlessly deploy successfully merged pull requests to the production environment.

In this how-to, we will walk you through best practices for scheduling Coalesce refreshes through the GitHub Actions interface to allow you to automate the triggering of your Coalesce jobs to be on a schedule or upon approved pull request into your main branch in accordance with CI/CD best practices.

## Before You Begin

You'll need:

- **GitHub**:  [GitHub][] account with admin privileges to configure pipelines and manage secrets.
- **Project**: A [Project][] with version control configured using your chosen platform.
- **Environment**: At least one [Environment][] configured for deployment.
- **Access Token**: Your Coalesce [access token][] for the CLI configuration file.

### Create and Configure an Environment

1. Create a new [Environment](/docs/setup-your-project/create-your-environments) for CI/CD deployments. It's recommended to use a dedicated Environment for automated deployments.
2. Configure authentication for your Environment based on your data platform:
   - **Snowflake**: Set up [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) with a service account.
   - **Databricks**: Set up [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) with a service principal.
3. Configure Storage Mappings for your Environment.
4. Deploy the Environment to verify configuration.

### Data Platform Permissions

#### Snowflake

- `USAGE` privilege on the database and schema.
- `SELECT`, `INSERT`, `UPDATE`, and `DELETE` privileges on tables involved in transformations.
- `EXECUTE` privilege for procedures or functions referenced by Coalesce.
- Ability to create [Snowflake Key-Pair Authentication](/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication) (recommended for automated deployments).

#### Databricks

- SQL Warehouse configured and accessible.
- Appropriate catalog and schema permissions.
- Ability to create [OAuth Machine-to-Machine authentication](/docs/setup-your-project/connection-guides/databricks) (recommended for automated deployments).

## Set up GitHub Variables and Environment

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

   1. In GitHub, go to the repo you are using with Coalesce.
   2. Click **Settings > Security > Secrets and variables > Actions**.
   3. You'll create two repository secrets by clicking the secrets tab, called `MY_COALESCE_SECRET` and `DATABRICKS_CLIENT_SECRET`.
       1. `MY_COALESCE_SECRET` - Your API access token from your environment.
       2. `DATABRICKS_CLIENT_SECRET` - Client Secret from Databricks OAuth Machine-to-Machine setup.
  
  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

   1. In GitHub, go to the repo you are using with Coalesce.
   2. Click **Settings > Security > Secrets and variables > Actions**.
   3. You'll create two repository secrets by clicking the secrets tab, called `MY_COALESCE_SECRET`, and `SF_PRIVATE_KEY`.
       1. `MY_COALESCE_SECRET` - Your API access token from your environment.
       2. `SF_PRIVATE_KEY` - Snowflake Private Key from Snowflake Key-Pair Authentication.
   4. Add your private key. The private key **must** contain explicitly typed newline (\\n) characters at the end of each line in the `.pem` file when saved as plain text in the private key secret in GitHub.

   ```txt title="Private key example"
   -----BEGIN PRIVATE KEY-----\n

   MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCv2xVhFpaM6hhf\n

   ce4U5GRfdArGkoqkL2EBRs0zGMn1YYfQ8+zDuN9YkMTNC1pNxQptGn921teGk0wv\n

   cMP+I83P390jqXh56TlQtwn2reRXH7OlLdELttof4VGYb4I6KpdBhDaid8bys2FE\n

   f0r948EXM81Euh9FgmMbc4KzeF1tBDyU0sqAcAJCQXOl95jUR6Wqdp04LXJVoGmI\n

   -----END PRIVATE KEY-----
   ```

  </TabItem>
</Tabs>

## Update Repo With Secrets

1. Create a new folder called `.github` in your main branch.
2. Create a subfolder within .github called `workflows` and file within workflows called `coalesce_test.yml`.
3. Copy and paste the following sample code into `coalesce_test.yml`.
4. Update the "environmentID": "3" and "jobID": "2" to match your Coalesce environment.

In the default state, this workflow is triggered under three circumstances:

- A pull request into the main branch
- A successful merging of an open pull request into the main branch
- Execution on the cron scheduler (commented out). If you want to modify the cron syntax, you can use any POSIX cron scheduler.

The file also includes `workflow_dispatch` so you can trigger the job manually or turn it on and off in the GitHub app.

```yaml
name: Deploy to PROD

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  COA_VERSION: latest

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: 22.19.0

      - run: echo '${{ secrets.COA_CONFIG }}' >> COA_CONFIG

      # Install Coalesce CLI tool, if not installed already
      - run: npm list | grep "@coalescesoftware/coa@${{ env.COA_VERSION }}" || npm install @coalescesoftware/coa@${{ env.COA_VERSION }}

      # Print version number
      - run: npx coa --version

      # Create Deployment Plan
      - run: npx coa plan --config COA_CONFIG --out ./coa-plan --debug

      # Deploy Plan
      - run: npx coa deploy --config COA_CONFIG --plan ./coa-plan --debug
```

## Credentials Required for Start Run API

When you trigger a refresh using the [Start Run][] API (`/scheduler/startRun`), you must include `userCredentials` in the request body. There is no profile-only option; the API requires credentials in each request to authenticate to your data platform.

Store credentials securely in CI/CD secrets (for example, GitHub Actions secrets). Never commit credentials to version control.

### KeyPair (Snowflake)

```json
{
  "runDetails": {
    "environmentID": "<your-environment-id>",
    "jobID": "<your-job-id>"
  },
  "userCredentials": {
    "snowflakeAuthType": "KeyPair",
    "snowflakeUsername": "<username>",
    "snowflakeKeyPairKey": "<private-key-content>",
    "snowflakeKeyPairPass": "<passphrase-if-encrypted>",
    "snowflakeWarehouse": "<warehouse>",
    "snowflakeRole": "<role>"
  }
}
```

### OAuth (Snowflake)

```json
{
  "runDetails": {
    "environmentID": "<your-environment-id>",
    "jobID": "<your-job-id>"
  },
  "userCredentials": {
    "snowflakeAuthType": "OAuth"
  }
}
```

### OAuth Machine-to-Machine (Databricks)

```json
{
  "runDetails": {
    "environmentID": "<your-environment-id>",
    "jobID": "<your-job-id>"
  },
  "userCredentials": {
    "platformKind": "databricks",
    "databricksAuthType": "OAuthM2M",
    "databricksClientID": "<client-id>",
    "databricksClientSecret": "<client-secret>",
    "databricksPath": "<sql-warehouse-path>",
    "databricksAccountHost": "https://accounts.cloud.databricks.com",
    "databricksWorkspaceHost": "https://<workspace>.cloud.databricks.com"
  }
}
```

For full request schemas and authentication options, see [Start Run][].

---

## What's Next?

- [GitHub Actions][]
- [Environments][]
- [Snowflake Key-Pair Authentication][]
- [Cron Syntax][]

---

[Environments]: /docs/get-started/coalesce-fundamentals/environments
[Snowflake Key-Pair Authentication]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[Environment]: /docs/get-started/coalesce-fundamentals/environments
[GitHub Actions]: https://docs.github.com/en/actions
[Cron Syntax]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/crontab.html#tag_20_25_07
[Project]: /docs/setup-your-project/create-your-project
[GitHub]: /docs/setup-your-project/setup-version-control#github
[access token]: /docs/api/authentication
[Start Run]: /docs/api/runs/startrun

---

## Orchestrate Deploys with Azure DevOps

## Before You Begin

- Access to an Azure DevOps account and project. [Sign up for free here][].
- [Git][] must be set up and configured in your Coalesce Org. Azure DevOps Pipelines are flexible and can be configured with all Git providers that Coalesce supports.

## Agent Setup

Azure DevOps has multiple configurations for agents that include self-hosted or Microsoft-hosted (requires an Azure subscription). Coalesce can use an existing agent pool if your organization already has pre-configured agents.

- Details on setting up an Azure DevOps agent can be found in [Microsoft's documentation][].
- Take note of the agent pool name when setting up the agent as you will need this when defining the `pool` in your pipeline definition (`azure-pipelines.yml`). In the examples below, we will use `pool: default` since that is how our agent pool is configured.

## Upload Secure File

To follow best practices, Coalesce recommends that you upload your [COA config file][] as a secure file within the Azure DevOps Pipeline.

1. From within your Azure DevOps Project, select **Pipelines** and subsection **Library** from the left sidebar.
2. Select **\+ Secure File** to upload your COA config. By default this file is in `~/.coa`. Take note of the file name for the next section.

## Create a Deploy Pipeline

1. From within your Azure DevOps Project, select **Pipelines** and subsection **Pipelines** from the left sidebar.

    

2. Select **Create Pipeline** and follow the wizard to connect your Git repository to the Azure DevOps Pipeline.
3. When prompted to **Configure your pipeline**, select **Starter pipeline**
4. When prompted to **Review your pipeline YAML**, replace the `azure-deploy-pipelines.yml` contents with the following code.

```yaml title="azure-deploy-pipeline.yml"
pool: default

variables:
 - name: coalesceCliVersion
   value: latest # Before utilizing this pipeline in production scenarios, it is advised to pin a specific version. This practice aims to reduce potential disruptions. You can find the released versions on NPM at this link: https://www.npmjs.com/package/@coalescesoftware/coa 

trigger:
  branches:
    include:
    - main

steps:

- task: DownloadSecureFile@1
  name: coaConfig
  displayName: 'Download coa config'
  inputs:
    secureFile: 'config'

- task: CmdLine@2
  inputs:
    script: |
      # Install Coalesce CLI tool, if not installed
      npm list -g | grep "@coalescesoftware/coa@$(coalesceCliVersion)" || npm install -g @coalescesoftware/coa@$(coalesceCliVersion);
      
      # Create Deployment Plan
      coa plan  --config $(coaConfig.secureFilePath);

      # Execute Deploy
      coa deploy  --config $(coaConfig.secureFilePath);
```

## Create a Refresh Pipeline

1. From within your Azure DevOps Project, select **Pipelines** and subsection **Pipelines** from the left sidebar.

    

2. Select **Create Pipeline** and follow the wizard to connect your Git repository to the Azure DevOps Pipeline.
3. When prompted to **Configure your pipeline**, select **Starter pipeline**.
4. When prompted to **Review your pipeline YAML**, replace the `azure-refresh-pipelines.yml` contents with the following code.
5. Remember to click **Save**.
6. Now manually test your new pipeline by selecting **Run** to kick off your refresh.

:::info[On the YAML Example Code]

The below is an example for an hourly refresh on the main branch, although it could be modified for a daily deploy, a different branch, or other options.

:::

```yaml title="azure-refresh-pipelines.yml"
pool: default

variables:
 - name: coalesceCliVersion
   value: latest # Before utilizing this pipeline in production scenarios, it is advised to pin a specific version. This practice aims to reduce potential disruptions. You can find the released versions on NPM at this link: https://www.npmjs.com/package/@coalescesoftware/coa 

schedules:
- cron: "0 * * * *"
  displayName: Hourly Refresh
  branches:
    include:
    - main

steps:

- task: DownloadSecureFile@1
  name: coaConfig
  displayName: 'Download coa config'
  inputs:
    secureFile: 'config'

- task: CmdLine@2
  inputs:
    script: |
      # Install Coalesce CLI tool, if not installed
      npm list -g | grep "@coalescesoftware/coa@$(coalesceCliVersion)" || npm install -g @coalescesoftware/coa@$(coalesceCliVersion);
      
      # Start refresh
      coa refresh  --config $(coaConfig.secureFilePath);
```

[Sign up for free here]: https://azure.microsoft.com/en-us/services/devops/
[Git]: /docs/setup-your-project/setup-version-control
[Microsoft's documentation]: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents
[COA config file]: /docs/coa/version-732-and-below/coa-setup

---

## Third-Party DevOps Tools

:::tip[Coalesce Scheduler]

The Coalesce Scheduler is now available. Schedule deployments directly in the app. Read more in [Scheduling Jobs in Coalesce][].

:::

Coalesce integrates seamlessly with popular third-party platforms through our API and CLI.

## Orchestration Options

Tools like Snowflake, Azure Data Factory, and Apache Airflow provide robust orchestration capabilities for managing complex data workflows. These platforms excel at coordinating multiple dependencies and creating scheduling patterns.

## CI/CD Options

Continuous Integration/Continuous Deployment platforms such as GitHub Actions, GitLab, Bitbucket, and Azure DevOps enable you to automate deployments and refreshes as part of your existing development workflows. These tools are ideal for teams following DevOps practices who want to maintain consistent deployment processes.

<DocCardList />

[Scheduling Jobs in Coalesce]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce

---

## Orchestrate Refresh Jobs with Azure Data Factory

Azure Data Factory is a cloud-based data integration service that allows creating and orchestrating data movement and transformation workflows at scale. Data analysts and engineers can visually construct pipelines extracting data from disparate siloed sources, applying data cleansing and enrichment flows leveraging services like Fivetran and publishing to centralized data warehouses and lakehouses.

Built-in triggers coupled with enterprise grade reliability, availability, and scalability has made Azure Data Factory a popular choice for realizing end-to-end orchestration of data pipelines. When working with a column-aware data transformation solution like Coalesce on Azure, Data Factory’s scheduling and automation capabilities help deploy existing data transformation pipelines amongst a feature rich logging and orchestration UI to ensure a trustworthy data ecosystem.

In this article we will explore setting up a basic Coalesce pipeline Refresh Trigger using the Azure Data Factory web UI.

## Set up a Coalesce Environment

In order to execute jobs within Coalesce, we will have to create an Environment where our job will be deployed to, separate from our development Workspace. To do this, follow these steps.

1. Go to **Build > Build Settings > Environments >New Environment**.

    

2. Specify the necessary information in each of the tabs to create your Environment. Review [Environments][] to learn more.
3. Once your environment is set up, navigate to the **Deploy tab** in the top menu.
4. Select **Deploy** next to the Environment that you created,
5. Select the branch from the Workspace or job that you wish to deploy into your new Environment. We recommend deploying the latest commit of your main branch into the production Workspace but you can choose any branch or job that you wish to deploy.
6. Once you’ve deployed into your selected Environment, click **Generate access token** and save it somewhere; you will need it to authenticate the API call made from Azure Data Factory.

## Set up KeyPair Auth in Snowflake

Although the [Coalesce API][] supports Snowflake Username/Password authentication, you should not use it for automated pipeline scheduling in order to conform to security best practices. Instead, we will configure [KeyPair Authentication][] in Snowflake and format the key so it can be used as a secret in ADF. Save the private key generated from this process somewhere safe, as it be need in subsequent sections.

## Configure the Azure Key Vault

In order to store and use the credentials that we need in order to authorize requests from ADF to Coalesce/Snowflake, you need to store the credentials (access tokens, keys, etc.) securely within the Azure Key Vault, and then link the Azure Key Vault to our Data Factory.

Even though you can store secrets less securely at the ADF pipeline level, this is not recommended as compared to the secure flow that is detailed below.

:::info[Azure Administrator]

For the next set of steps, you will have to be an Azure Administrator to have the necessary permissions to create key vaults, secrets, and edit permissions.

:::

1. Create a key vault by logging into the Azure portal and selecting **Key vaults** from the Azure services.

    

2. Create the Key vault by selecting your subscription and proceeding through the menus. In this example, our Key vault is called `coalesceCreds`.

    

3. Go to the new Key vault and go to **Object >Secrets**.

    

4. Select **Generate/Import**.

    

5. Add two secrets:
    - `COA-API-KEY`, which will be the access token that we generated in the section [Set up Coalesce Environment][].
    - `snowflakeKeyPairKey`, which is the Snowflake key that we generated in the section [Set up KeyPair Auth in Snowflake][].

    

    :::warning[Formatting the KeyPairKey]

    The KeyPairKey MUST be inputted in a very specific format, all newlines must be explicitly stated, and all whitespaces removed.

    For example, if your key looks like this:

    

    It must be inputted like this:

    

    :::

6. Go to **Access Control (IAM)** to configure the access policy for the Key Vault, and click **Add Role Assignment**.

    

7. Select the **Key Vault Secrets User** role.

    

8. Click Next, then Select **Managed Identity”** on the next screen and select the name of the Data Factory, in this case, our data factory is called “JJ-Data-Factory-Coalesce”.

    

9. Click **Review + assign**.
10. Navigate to the **Azure Data Factory** and link the key vault to the data factory by going to **Linked Services > New**.

    

11. Create the Linked Service.

## Using Azure Key Vault Secrets in Pipeline Activities

Follow the steps outlined in [Microsoft's Use Azure Key Vault secrets in pipeline activities][] guide to add the secrets to our ADF access policy and import in our secrets and use them in our pipeline.

1. Create a Web Activity type to extract the Access token from the Key vault.
2. Repeat the steps to add a web Activity type that retrieves the Snowflake Key Pair credential. Both activities must have the **Secure output** box checked.

## Create the Coalesce Refresh Activity

You can now begin assembling the Web Activity that will trigger a refresh of the Coalesce environment of our choosing. For this, you'll use the Coalesce API to issue a [`startRun`][] call to trigger the pipeline. See Step 4 for what it should look like at the end.

1. Create a new Web Activity, name it **Coalesce Refresh Trigger**, and drag lines from the other Web Activity modules to it.

    

2. To input the Key Pair Key into the Coalesce Refresh Action Trigger and API body, click on **Body** to bring up the Pipeline Expression Builder for that input. Input the body of the Coalesce [`startRun`][] endpoint that you configured in the API documentation.

    

    ```json title="API Body"
    "runDetails": {"parallelism":16, "environmentID": "2"},  
    "userCredentials": {"snowflakeAuthType": "KeyPair",  
    "snowflakeKeyPairKey":"@{activity( 'Get Snowflake Key Pair  
    Key'). output. value}"}}
    //Remember to update the body text for your use.
    ```

3. To input the Coalesce Access Token, navigate to the **Authorization** header and click the **Value** input to bring up the Pipeline Expression Builder and input the following expression.

    

    ```json title="Authorization"
    Bearer @activity( 'Get Access Token'). output.value}
    ```

4. Your final configuration should match the following image. Enter your apps URL, the method (POST), and the Headers.

    

5. Click **OK** to save the changes.
6. Check the **Secure input** box under **General** to ensure that none of the secure credentials are exposed.

    

## Scheduling Your Coalesce Trigger Pipeline

You're going to schedule your ADF pipeline.

1. Go to the **Edit tab** in Azure Data Factory. Click **Add Trigger > New/Edit**.

    

2. Input the desired cadence and times that you want to trigger the refresh for, or select from any previously selected cadences. See the example below.

    

3. Click **Ok**.

You have now successfully created and scheduled a Coalesce Environment Refresh with Azure Data Factory.

[Environments]: /docs/get-started/coalesce-fundamentals/environments
[Coalesce API]: /docs/api/runs/startrun
[KeyPair Authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[Set up Coalesce Environment]:/docs/deploy-and-refresh/third-party-devops-tools/orchestration/coalesce-refresh-scheduling-with-azure-data-factory#set-up-a-coalesce-environment
[Microsoft's Use Azure Key Vault secrets in pipeline activities]: https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities
[`startRun`]: /docs/api/runs/startrun
[Set up KeyPair Auth in Snowflake]: /docs/deploy-and-refresh/third-party-devops-tools/orchestration/coalesce-refresh-scheduling-with-azure-data-factory#set-up-keypair-auth-in-snowflake

---

## Schedule Coalesce Jobs with Apache Airflow

Coalesce's code-first GUI-driven approach has makes it easier to build, test, and deploy data pipelines than ever before. This considerably improves the data pipeline development workflow when compared to creating directed acyclic graphs (or DAGs) purely with code.

Coalesce relies on third-party services and orchestrators to schedule and execute the Coalesce-built data pipelines, which are organized into [Jobs][] for execution.

[Apache Airflow][] is a common orchestrator used with Coalesce.

Airflow stands out as a widely embraced open-source platform dedicated to crafting, scheduling, overseeing, and archiving workflows. Rooted in Python and backed by a thriving community, it has evolved into an essential tool in the repertoire of data engineers, proficiently handling the orchestration, scheduling, and construction of pipelines.

With respect to orchestrating Coalesce Jobs, Airflow can be used to schedule and execute these Jobs directly, or within the constructs of a larger organizational data pipeline or stack via Airflow DAGs.

This article details the steps for configuring nodes in an Airflow DAG to utilize the [Coalesce API][] to trigger Coalesce Jobs on a schedule.

## Before You Begin

The steps detailed in this article assume the following is already in place:

- You have a basic Airflow web server and scheduler already running.
- You have created the Coalesce Jobs for your data pipelines.
- You have deployed your Coalesce Jobs to your Coalesce Environments you wish to refresh.
- You have [generated your Coalesce Access Token][] as is required to use the Coalesce Command-Line Interface (CLI) or API.
- You have the credentials readily available for a [supported Authentication to Snowflake method for the Coalesce API.][]

## Configure Airflow to Execute Coalesce Jobs on a Schedule

The steps below will detail how to configure Airflow to execute Coalesce Jobs on a Schedule.

### Configure Airflow Variables to Support Authentication to Snowflake

In Airflow, you will need to create variables to store your the Snowflake credentials for your selected Snowflake authentication method.

:::info[KeyPair Authentication]

In this article, we will be leveraging the KeyPair Authentication method to avoid exposing actual username/password credentials in the DAG, and the instructions and screen captures will reflect that authentication method. You will need to modify to support your selected authentication method if it is not KeyPair Authentication.

:::

1. Within Airflow, go to the  **Admin** tab, and click **Variables**.

    

2. Create two variables:

    - `AIRFLOW_VAR_COALESCE_TOKEN`: To hold your [Coalesce Access Token][].
    - `AIRFLOW_VAR_SNOWFLAKE_PRIVATE_KEY`: To hold the private key you generated for Snowflake.

    

### Create and Configure Your Airflow DAG to Run a Coalesce Job

In the place where your DAGs are stored within Airflow:

1. Create a new DAG.
2. Paste in the following code:

    ```python
    from datetime import datetime, timedelta
    from airflow import DAG
    from airflow.operators.bash_operator import BashOperator
    from airflow.models import Variable
    ​
    ​
    default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 01, 01),
    'retries': 1,
    'retry_delay': timedelta(minutes=2),
    }
    ​
    dag = DAG(
    'example_coalesce_trigger_w_status_check_parametrized_token',
    default_args=default_args,
    description='An example Airflow DAG',
    schedule_interval=timedelta(days=1),
    )

    print_hello_command = 'echo "Hello, World"'
    ​
    execute_api_call_command = """
    response=$(curl --location 'https://app.coalescesoftware.io/scheduler/startRun' \
    --header 'accept: application/json' \
    --header 'content-type: application/json' \
    --header 'Authorization: Bearer {{ params.coalesce_token }}' \
    --data-raw '{
    "runDetails": {
    "parallelism": 16,
    "environmentID": "3",
    "jobID": "3" 
    },
    "userCredentials": {
    "snowflakeAuthType": "KeyPair",
    "snowflakeKeyPairKey": "{{ params.sf_private_key }}"
    }
    }')
    ​
    # Extract runCounter from the JSON response using Python and remove the prefix.
    runCounter=$(echo "$response" | python -c "import sys, json; print(json.load(sys.stdin)['runCounter'])" | sed 's/Run Counter: //')
    echo "$runCounter"
    """
    ​
    check_status_command = """
    curl --request GET \
    --url 'https://app.coalescesoftware.io/scheduler/runStatus?runCounter={{ task_instance.xcom_pull(task_ids="execute_api_call") }}' \
    --header 'Authorization: Bearer {{ params.coalesce_token }}' \
    --header 'accept: application/json'
    """
    ​
    # Define tasks
    task_print_hello = BashOperator(
    task_id='print_hello',
    bash_command=print_hello_command,
    dag=dag,
    )
    ​
    task_execute_api_call = BashOperator(
    task_id='execute_api_call',
    params={'coalesce_token': Variable.get('AIRFLOW_VAR_COALESCE_TOKEN'),
    'sf_private_key': str(Variable.get('AIRFLOW_VAR_SNOWFLAKE_PRIVATE_KEY'))},
    bash_command=execute_api_call_command,
    dag=dag,
    )
    ​
    task_check_status = BashOperator(
    task_id='check_status',
    params={'coalesce_token': Variable.get('AIRFLOW_VAR_COALESCE_TOKEN')},
    bash_command=check_status_command,
    dag=dag,
    )
    ​
    # Set task dependencies
    task_print_hello >> task_execute_api_call >> task_check_status
    ```

    :::info[Check Your Environment and Job IDs]

    Ensure that you replace the values for `environmentId` and `jobId` in the code with the relevant ID’s of the Environment and Job in your organization in Coalesce. If you don’t want to specify a job, and instead, run the entire Coalesce pipeline, simply remove the `jobId` as an argument in the payload of the node in the DAG.

    The `environmentId` can be found in the [Deploy Page][] of the Coalesce App. The jobID can be found in a Workspace that contains your Job. See [Find your IDs][] for all the places your ID can be found.

    :::

3. Edit the values for attributes like `start_date`, `retries`, `retry_delay`, `schedule_interval`, etc. to reflect your requirements.
4. Change the name of the DAG to reflect its purpose.

Once configured and the DAG is triggered, manually or on the schedule, you will see the corresponding Job executions and their details in the Activity Feed in the Coalesce App. You are also able to [retrieve run results and details via the API][].

### Running Multiple Coalesce Jobs Using Airflow

You can run or schedule multiple Coalesce Jobs using the Airflow Scheduler for which there are multiple ways of doing so.

1. You can create additional nodes within the same Airflow DAG, each of which calls a different job executing sequentially. You can indicate the order by modifying the task dependencies at the bottom of the DAG code:

    ```text
    refresh_job_1 >> refresh_job_2 >> refresh_job_3 >> ...
    ```

    where each `refresh_job_X` is the name of a task that executes the API call to trigger a refresh of an environment or job.

2. You can create multiple Airflow DAG's that you can schedule independently. Copy the example code into a new DAG within your /dags folder in Airflow, edit the `executeapi_call_command`\_ function with the desired `environmentID` and `jobID` parameters, modify the schedule, and give the DAG a new name.

    ```tex
    dag = DAG(
    'example_coalesce_trigger_w_status_check_parametrized_token',
    default_args=default_args,
    description='An example Airflow DAG',
    schedule_interval=timedelta(days=1),
    )
    ```

:::info[Best Practices]

It is best practice to save the DAG `.py` file with the same name as that given to the DAG within the file.

:::

## Video How To Use Airflow To Schedule Coalesce Jobs

Watch a demo on using Airflow for Coalesce Jobs.

<YouTube id="TpJzIhrgXoQ" />

[Jobs]: /docs/deploy-and-refresh/refresh/
[Apache Airflow]: https://airflow.apache.org/
[Coalesce API]: /docs/api/runs/startrun
[generated your Coalesce Access Token]: /docs/coa/version-732-and-below/coa-setup#get-your-access-token
[supported Authentication to Snowflake method for the Coalesce API.]: /docs/category/snowflake
[Coalesce Access Token]: /docs/coa/version-732-and-below/coa-setup#get-your-access-token
[Deploy Page]: /docs/reference/whats-my-id
[retrieve run results and details via the API]: /docs/api/coalesce/get-runs
[Find your IDs]: /docs/reference/whats-my-id

---

## Scheduling Coalesce Pipelines in Snowflake

One easy way to schedule Coalesce pipelines/jobs without the need to use any third party schedulers is to take advantage of Snowflake stored procedures, Tasks, and functions to make the necessary API calls to the Coalesce endpoints at a custom-defined cadence.

In this article we go over the procedure to set up these functions and Tasks in Snowflake and any necessary authentication required to trigger the Coalesce jobs.

## Before You Begin

1. In order to run these stored procedure setup DDL commands, one will need to be at least a SYSADMIN or higher user role in Snowflake.
2. You will need to setup [Snowflake KeyPair Authentication][] or [Snowflake OAuth][] to authorize into the Snowflake servers through Coalesce via API call. Although it is possible to use username/password authorization, this is less secure than KeyPair or OAuth.
3. If using KeyPair, save your KeyPair Key and decryption password in a safe place as we will need to input it into our Stored Procedures.
4. You'll need to get a [Coalesce API Access Token][].

### Enable Authentication On Your Environment

You can skip this step if you have already enabled [KeyPair][] or [OAuth][] previously.

1. Go to the **Deploy** Tab.
2. Choose the environment you want to deploy, click **Configure Deployment**.
3. If using KeyPair go to **User Credentials** and select Authentication Type as **KeyPair**.
4. If using OAuth, go to **OAuth Settings > Toggle OAuth** on. Enter the **Client ID** an **Client Secret**.
    1. Go to **User Credentials** and select **Snowflake OAuth**.

## Create Network Rule & Token Secret

You will need to add network rules to Snowflake such that it can reach the [Coalesce API][]. In addition, one will need to create a secret holding the Coalesce API Token. Make sure to replace any placeholder `<>` in the below examples with the proper values conforming to your Snowflake roles/Coalesce Tokens.

```sql title="Create a network rule for Snowflake to reach Coalesce API"

-- Create a network rule for Snowflake to reach Coalesce API

CREATE OR REPLACE  NETWORK RULE COALESCE_API_RULE
MODE = EGRESS
TYPE = HOST_PORT
VALUE_LIST = ('APP.COALESCESOFTWARE.IO');

-- Create a Secret containing Token from Coalesce

CREATE OR REPLACE SECRET COALESCE_API_TOKEN 
TYPE = GENERIC_STRING 
SECRET_STRING='<your Coalesce token>';
GRANT USAGE ON SECRET COALESCE_API_TOKEN TO <role>;
```

## Stored Procedure Setup

It is recommended to create these procedures in a separate Database and Schema such that they are not lost with your Data Warehouse nodes. A DB/Schema like `COALESCE.UTILITY` can be a good place to store these. Depending on the authentication scheme that you decide to go with, you will have to follow a separate set of procedures, which are outlined below.

:::info[Find and Replace COMPUTE_WH and SYSADMIN]

The Warehouse and Role used to execute jobs is hard coded in the Procedure / Functions and can be modified depending on your requirements. Search and replace the COMPUTE\_WH and SYSADMIN text in the setup script below.

:::

### KeyPair Auth Procedures

Depending on if the KeyPair key created is encrypted or not, one will have to follow either the [Non-Encrypted Private Key][] or the [Encrypted Private Key][] set of procedures for creating the necessary integrations, secrets and Snowflake functions.

#### Non-Encrypted Private Key

:::warning[Newlines]

Newlines must be encoded as "\\n" within the string.

:::

```sql title="Non-Encrypted Private Key"
-- Create a Secret containing the snowflake private key.
-- Newlines must be encoded as "\n" within the string!.
  
CREATE OR REPLACE SECRET SNOWFLAKE_PRIVATE_KEY
TYPE = GENERIC_STRING
SECRET_STRING = ‘<your snowflake private key>’
GRANT USAGE ON SECRET SNOWFLAKE_PRIVATE_KEY TO <role>;

-- Create Integration to access Coalesce API 

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION COALESCE_API_INTEGRATION
ALLOWED_NETWORK_RULES = (COALESCE_API_RULE)
ALLOWED_AUTHENTICATION_SECRETS = (COALESCE_API_TOKEN, SNOWFLAKE_PRIVATE_KEY)
ENABLED=TRUE;

-- COALESCE_API_RUN_JOB
-- Pass in JobID and EnvironmentID from Coalesce
-- Calls the startRun API to release a job 
-- Returns the runCounter of that job instance
-- eg CALL COALESCE_API_RUN_JOB(5, 3)

CREATE OR REPLACE PROCEDURE COALESCE_API_RUN_JOB(JobID NUMBER, EnvironmentID NUMBER)
RETURNS STRING
language python
runtime_version=3.8
handler = 'run_job'
external_access_integrations=(COALESCE_API_INTEGRATION)
packages = ('snowflake-snowpark-python','requests')
secrets = ('token' = COALESCE_API_TOKEN, 'creds' = SNOWFLAKE_PRIVATE_KEY)
as
$$

def run_job(session, JobID, EnvironmentID):
    token = _snowflake.get_generic_secret_string('token')
    sf_private_key = _snowflake.get_generic_secret_string('creds')
    
    url = "https://app.coalescesoftware.io/scheduler/startRun"
    headers = {
        "accept": "application/json",
        "authorization": "Bearer " + token
    }
    payload = {
    "runDetails": {
        "parallelism": 16,
        "environmentID": str(EnvironmentID),
        "jobID": str(JobID)
        },
    "userCredentials": {
        "snowflakeAuthType": "KeyPair",
   "snowflakeKeyPairKey": sf_private_key,
        "snowflakeWarehouse": "COMPUTE_WH",
        "snowflakeRole": "SYSADMIN"
        }
    }
    
    response = requests.post(url, json=payload, headers=headers)
    if response.status_code == 200:
        if "runCounter" in response.json():
            return response.json()['runCounter']
        else:
            return response.json()
    else:
        return "Error, response: " + str(response.status_code)

$$;
```

#### Encrypted Private Key

:::warning[Must Have Encrypted Private Key]

USE ONLY IF YOUR PRIVATE KEY IS ENCRYPTED

:::

```sql title="Encrypted Private Key"
-- USE ONLY IF YOUR PRIVATE KEY IS ENCRYPTED!
    
CREATE OR REPLACE SECRET SNOWFLAKE_KEY_PAIR_PASS
TYPE = GENERIC_STRING
SECRET_STRING = ‘<your private key decryption password>’
GRANT USAGE ON SECRET SNOWFLAKE_KEY_PAIR_PASS TO <role>;
-- Create Integration to access Coalesce API (w/ ENCRYPTED PRIVATE KEY)

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION COALESCE_API_INTEGRATION
ALLOWED_NETWORK_RULES = (COALESCE_API_RULE)
ALLOWED_AUTHENTICATION_SECRETS = (COALESCE_API_TOKEN, SNOWFLAKE_PRIVATE_KEY, SNOWFLAKE_KEY_PAIR_PASS)
ENABLED=TRUE;

-- COALESCE_API_RUN_JOB
-- Pass in JobID and EnvironmentID from Coalesce
-- Calls the startRun API to release a job 
-- Returns the runCounter of that job instance
-- eg CALL COALESCE_API_RUN_JOB(5, 3)

CREATE OR REPLACE PROCEDURE COALESCE_API_RUN_JOB_ENC(JobID NUMBER, EnvironmentID NUMBER)
RETURNS STRING
language python
runtime_version=3.8
handler = 'run_job'
external_access_integrations=(COALESCE_API_INTEGRATION)
packages = ('snowflake-snowpark-python','requests')
secrets = ('token' = COALESCE_API_TOKEN, 'creds' = SNOWFLAKE_PRIVATE_KEY, 'keyPass' = SNOWFLAKE_KEY_PAIR_PASS)
as
$$

def run_job(session, JobID, EnvironmentID):
    token = _snowflake.get_generic_secret_string('token')
    sf_private_key = _snowflake.get_generic_secret_string('creds')
    keyPairPass = _snowflake.get_generic_secret_string('keyPass')
    
    url = "https://app.coalescesoftware.io/scheduler/startRun"
    headers = {
        "accept": "application/json",
        "authorization": "Bearer " + token
    }
    payload = {
    "runDetails": {
        "parallelism": 16,
        "environmentID": str(EnvironmentID),
        "jobID": str(JobID)
        },
    "userCredentials": {
        "snowflakeAuthType": "KeyPair",
   "snowflakeKeyPairKey": sf_private_key,
  "snowflakeKeyPairKey": keyPairPass,
        "snowflakeWarehouse": "COMPUTE_WH",
        "snowflakeRole": "SYSADMIN"
        }
    }
    
    response = requests.post(url, json=payload, headers=headers)
    if response.status_code == 200:
        if "runCounter" in response.json():
            return response.json()['runCounter']
        else:
            return response.json()
    else:
        return "Error, response: " + str(response.status_code)

$$;
```

## Email Notifications

If you want to setup email alerts on refresh statuses for pipelines, one can create a notification integration and follow the instructions in Snowflake's [Sending Email Notifications][].

```sql

ALTER USER <user> SET EMAIL = '<email address>';

CREATE NOTIFICATION INTEGRATION COALESCE_EMAIL
    TYPE=email
    ENABLED=true
    ALLOWED_RECIPIENTS=('<email address1>', '<email address2>');
```

Once your stored procedures are created and saved, they can be called on a schedule using [Snowflake Tasks][].

### Example: Run a Job on a Schedule

```sql
-- Run a job on a schedule
CREATE TASK TASK_RUN_JOB_PROCESS_DIMS 
SCHEDULE = 'USING CRON 0 8-17 * * 1-5 America/Los_Angeles'
WAREHOUSE = COMPUTE_WH 
AS CALL RUN_JOB_SEND_EMAIL(5, 3, 'my_email@gmail.com');
```

### Example: Run a Job When Data Changes Based on a Stream

```sql
-- Run a job when data change based on a STREAM
CREATE TASK TASK_RUN_JOB_PROCESS_DIMS 
SCHEDULE = '1 minute' 
WHEN SYSTEM$STREAM_HAS_DATA('DEV_STG.STR_ORDERS') 
WAREHOUSE = COMPUTE_WH 
AS CALL COALESCE_API_RUN_JOB_COMPLETE(5 , 3);
```

### Example: Run a Secondary Job on Completion of a Prior Job

```sql
-- Run a secondary job on completion of a prior job
CREATE TASK TASK_RUN_JOB_PROCESS_FACTS 
AFTER TASK_RUN_JOB_PROCESS_DIMS
AS CALL COALESCE_API_RUN_JOB_COMPLETE(5 , 3);

ALTER TASK TASK_RUN_JOB_PROCESS_DIMS RESUME;
ALTER TASK TASK_RUN_JOB_PROCESS_FACTS RESUME;
```

## Video: Schedule Your Coalesce Jobs Using Snowflake

<YouTube id="MoqY-tcPzp4?si=1KPz7mdQEWd0LCVv" />

[Snowflake KeyPair Authentication]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[Snowflake OAuth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-oauth
[KeyPair]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[OAuth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-oauth
[Coalesce API]: /docs/api/runs/startrun
[Non-Encrypted Private Key]: /docs/deploy-and-refresh/third-party-devops-tools/orchestration/scheduling-coalesce-pipelines-in-snowflake#non-encrypted-private-key
[Encrypted Private Key]: /docs/deploy-and-refresh/third-party-devops-tools/orchestration/scheduling-coalesce-pipelines-in-snowflake#encrypted-private-key
[Sending Email Notifications]: https://docs.snowflake.com/en/user-guide/email-stored-procedures
[Snowflake Tasks]: https://docs.snowflake.com/en/user-guide/tasks-intro
[Coalesce API Access Token]: /docs/api/

---

## COA Deployment Error Resolution

This guide helps you quickly find and fix common deployment errors. It consolidates links to existing docs and adds resolution steps for errors that frequently appear in support tickets.

## Quick Reference Table

| Error or Scenario | Where to Find Help |
| :---------------- | :----------------- |
| Permission denied or env-deploy errors | [Coalesce Permission Requirements](#coalesce-permission-requirements) |
| Snowflake "Insufficient privileges" in view validation | [Snowflake View Validation Privileges](#snowflake-view-validation-privileges) |
| Blank column names or plan validation failure | [Blank Column Names][blank-column-names] |
| Storage mapping errors | [Storage Mapping Deployment Failures][storage-mapping-errors] |
| Plan failures (renames, dropped tables, metadata drift) | [Troubleshooting Deploys and Refreshes][troubleshooting-index] |
| Invalid column metadata (no failed stages) | [Invalid Column Metadata][invalid-column-metadata] |
| Primary key or schema change failures | [Primary Key Schema Change Failures][primary-key-schema] |

## Coalesce Permission Requirements

To deploy or refresh through the API, CLI, or Coalesce App, the account or service principal must have the right Coalesce role and data platform access.

### Environment-Level Permissions

- **Environment Admin**: Required to deploy. This role manages Environment settings, reads project documentation, and deploys through the API, CLI, or Coalesce App.
- **Environment Reader**: Can view documentation and certain API functions; cannot deploy.

If you see permission or "env-deploy" related errors when deploying (for example, from CI/CD or the CLI), ensure the account has the **Environment Admin** role on the target Environment. Service accounts used for automated deployments must be added to the Org, Project, and Environment with this role.

For details, see [Adding Users and Setting Permissions][] and [RBAC Roles and Permissions][].

### Data Platform Permissions

Your Environment credentials (Snowflake, Databricks, etc.) must have sufficient privileges to create and modify objects. For Snowflake, see [Setting Up a Snowflake Service Account][]. For CI/CD, see [Data Platform Permissions][] in the GitHub Actions guide.

## Snowflake View Validation Privileges

You may see errors such as:

`SQL access control error: Insufficient privileges to operate on table 'STG_DISCOUNTS_DATA'`

during the **Validating object exists** stage when deploying views. The role may have ownership on both the view and the underlying table, and the same query may run successfully when executed manually in Snowflake.

### Why This Happens

Snowflake evaluates view privileges at different times. During deployment, Coalesce runs validation queries (for example, `SELECT 1 FROM "DB"."SCHEMA"."VIEW_NAME" LIMIT 0`). Snowflake may enforce privileges on underlying tables differently in this context than when you run the query interactively.

### How To Resolve

1. **Grant explicit privileges on underlying objects**: Ensure the deployment role has `SELECT` on all tables and views referenced by the view definition:
   - `GRANT SELECT ON ALL TABLES IN SCHEMA <db>.<schema> TO ROLE <coalesce_role>;`
   - `GRANT SELECT ON ALL VIEWS IN SCHEMA <db>.<schema> TO ROLE <coalesce_role>;`
   - For future objects: `GRANT SELECT ON FUTURE TABLES IN SCHEMA ...` and `GRANT SELECT ON FUTURE VIEWS IN SCHEMA ...`

2. **Verify the role in use**: Confirm the role configured in the Environment credentials is the one with these grants. Check with `SHOW GRANTS TO ROLE <role_name>;` in Snowflake.

3. **Check for ownership vs. privilege**: Ownership alone can be insufficient in some Snowflake configurations. Explicit `SELECT` grants on underlying tables and views are recommended.

For a full setup, see [Setting Up a Snowflake Service Account][]. For general permission issues when adding sources, see [Permission Issues When Adding Sources][permission-issues].

## What's Next?

- [Troubleshooting Deploys and Refreshes][troubleshooting-index] for a full overview of deployment and refresh troubleshooting
- [Storage Mapping Deployment Failures][storage-mapping-errors] for database and schema mapping issues
- [Understanding Presync][presync] for how Coalesce reconciles off-platform changes

---

[Adding Users and Setting Permissions]: /docs/setup-your-project/add-users-set-permissions
[RBAC Roles and Permissions]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Setting Up a Snowflake Service Account]: /docs/setup-your-project/connection-guides/snowflake/snowflake-service-accounts
[Data Platform Permissions]: /docs/deploy-and-refresh/third-party-devops-tools/continuous-integration/schedule-jobs-with-github-actions-and-the-api#data-platform-permissions
[permission-issues]: /docs/reference/error-library/permission-issues
[troubleshooting-index]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes
[storage-mapping-errors]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/storage-mapping-errors
[presync]: /docs/deploy-and-refresh/deploy/understanding-presync
[blank-column-names]: /docs/reference/error-library/blank-column-names-plan-validation-failure
[invalid-column-metadata]: /docs/reference/error-library/invalid-column-metadata-deployment-failure
[primary-key-schema]: /docs/reference/error-library/primary-key-schema-change-failures

---

## Data Type Changes in Snowflake

Learn how to change column data types in Snowflake when those changes are unsupported by `ALTER TABLE … ALTER COLUMN`, and how to keep Coalesce metadata in sync afterward. You may see errors such as `SQL compilation error: cannot change column COLUMN_NAME from type DATE to TIMESTAMP_NTZ(9)` when deployment attempts an unsupported conversion. This workflow preserves column order and is suitable for Type 2 SCD tables and other cases where a simple drop-and-recreate would lose history or break dependencies.

## When To Use This Workflow

Snowflake does not support certain column changes using `ALTER TABLE … ALTER COLUMN`. These include:

- Changing a column data type to a different type (for example, DATE to TIMESTAMP_NTZ)
- Adding or changing a column default (except when the default is a sequence)
- Decreasing the length of a text or string column
- Changing the scale of a number column

When you change a Node, Coalesce attempts to CLONE the table, ALTER the copy, then SWAP the cloned table with the original and delete the original. This keeps the original data safe if an operation fails. The ALTER step fails for unsupported changes because Snowflake rejects the data type conversion. Advanced Deploy Nodes use direct ALTER (no clone or swap) and fail the same way. The recommended approach is to make the change off-platform in Snowflake, then sync Coalesce metadata.

Use this workflow when:

- **Column order matters:** Type 2 SCD tables, dimension tables with strict ordering, or downstream consumers that depend on column position.
- **You must preserve history:** A simple DROP and CREATE would destroy change-tracking history in SCD tables.
- **You need a repeatable, documented process:** This workflow gives you a clear sequence you can follow and adapt.

For simpler cases where you don't need to retain data, you can drop the table in Snowflake, commit your Coalesce changes, and deploy to re-create the table with the new structure. The table will be empty until the next refresh. For cases where column order and history matter, this guide focuses on the copy-table migration pattern that preserves structure and data.

## Prerequisites

Before you start, ensure you have:

- **Backup or Time Travel:** Snowflake Time Travel can restore data if something goes wrong. Confirm your table has sufficient retention, or take a manual backup.
- **Downtime or maintenance window:** The SWAP step is atomic, but data migration can take time for large tables. Plan accordingly.
- **Warehouse permissions:** You need `CREATE TABLE`, `ALTER TABLE`, `INSERT`, and `SELECT` on the table and schema.
- **Coalesce access:** You'll need to commit changes and deploy, and possibly clear Node metadata. Ensure you have the right Workspace and Environment permissions.

:::warning[Advanced Deploy Nodes]
If the table is managed by an Advanced Deploy Node, a failed deployment does not update Coalesce metadata. That can leave the database and Coalesce out of sync and block future deployments until metadata is corrected. This workflow includes steps to recover from that state.
:::

## Phase 1: Create a Copy Table With Modified DDL

Create a new table with the same structure as the original, but with the modified column data types. The copy table will hold the migrated data before you SWAP it with the original. Using Coalesce to generate the DDL ensures it matches what Coalesce expects and preserves column order.

1. In your Coalesce Workspace, update the data type for the target columns in the Node definition.
2. Run a **Validate Create** or **Create** operation to generate the DDL SQL.
3. Copy the generated DDL into Snowflake. Modify the table name to `[ORIGINAL_TABLE_NAME]_COPY` and adjust the database or schema to match your target Environment's storage mapping.
4. Execute the statement to create the copy table. Preserve column order exactly. It must match the original so that the SWAP and any downstream logic remain consistent.

**Example:** Changing `amount` from `VARCHAR` to `NUMBER(18,2)` in `sales_fact`:

```sql
-- DDL generated from Coalesce, table name changed to sales_fact_COPY
CREATE TABLE my_schema.sales_fact_COPY (
  id NUMBER,
  customer_id NUMBER,
  amount NUMBER(18,2),  -- Changed from VARCHAR
  order_date DATE
  -- ... other columns in same order
);
```

:::info[Type 2 SCD Tables]
For Type 2 SCD tables, include all effective-date, expiration, and current-flag columns in the same order. Preserving column order ensures that views, reports, and Coalesce Node definitions that reference column position continue to work.
:::

## Phase 2: Migrate Data Column by Column With Type Conversions

Copy data from the original table into the copy table. For columns whose type changed, use `CAST` or `TRY_CAST` to convert values. Handle nulls and invalid values as needed.

1. Write an `INSERT INTO … SELECT` that maps each column from the original to the copy.
2. For the changed columns, use the appropriate conversion (for example, `TRY_CAST(amount AS NUMBER(18,2))`).
3. Run the insert. For large tables, consider batching or using a warehouse with sufficient resources.

    **Example:**

    ```sql
    INSERT INTO my_schema.sales_fact_COPY (
      id,
      customer_id,
      amount,
      order_date
    )
    SELECT
      id,
      customer_id,
      TRY_CAST(amount AS NUMBER(18,2)),  -- Converts; invalid values become NULL
      order_date
    FROM my_schema.sales_fact;
    ```

4. Validate row counts and spot-check data to ensure the conversion produced the expected results.

:::info[Invalid or Truncated Data]
`TRY_CAST` returns NULL for values that cannot be converted. If you need different behavior (for example, default values or error handling), adjust the SELECT logic accordingly. For string length decreases, ensure no values exceed the new length or handle truncation explicitly.
:::

## Phase 3: SWAP To Replace the Original Table

Use Snowflake’s `ALTER TABLE … SWAP WITH` to atomically replace the original table with the copy. The original table becomes the backup; the copy becomes the production table.

1. Run the SWAP:

    ```sql
    ALTER TABLE my_schema.sales_fact SWAP WITH my_schema.sales_fact_COPY;
    ```

2. After the SWAP, `sales_fact` has the new schema and data. `sales_fact_COPY` now holds the old table. Drop the copy table, or keep it briefly as a backup before dropping.
3. Verify the production table has the correct schema and data.

:::warning[Downstream Dependencies]
The SWAP is atomic and instant. Any views, streams, or tasks that reference the table will now see the new structure. Ensure your type conversions did not introduce nulls or values that break those dependencies.
:::

## Phase 4: Sync Coalesce Metadata

Coalesce stores metadata about your tables from the last successful deployment. After you change the table off-platform, you must sync that metadata so Coalesce matches the actual warehouse state.

### Option A: Commit and Deploy

Use this when:

- You've made straightforward changes in Coalesce (column additions, renames, data type changes)
- The changes were made through the Coalesce UI rather than directly in Snowflake
- You expect Presync to handle the reconciliation automatically
- This is your first attempt at deploying changes
- The table structure in Snowflake should match what's defined in your Coalesce Nodes

Complete these steps:

1. Return to Coalesce. The Node definition should already reflect the new column data types from Phase 1.
2. Commit your changes.
3. Deploy to the Environment. Presync will compare Coalesce metadata with the actual warehouse state and reconcile. If the table structure matches what you deployed, metadata updates and you're done.

### Option B: Enable Column Resync on Affected Nodes

Use this when:

- Coalesce metadata is out of sync with the actual Snowflake table structure
- You see "metadata update" messages in deployment logs but no actual changes occur
- You've made manual changes in Snowflake that Coalesce isn't detecting
- Option A failed and you're getting deployment errors related to column mismatches
- You want to pull in column descriptions or other metadata from Snowflake back into Coalesce

Column resync forces a restore of all metadata for that object and resolves inconsistencies during deployment. For setup details, see [Enable Column Resync on Node Types][].

1. Open the affected Node in your Workspace.
2. Navigate to the Node's configuration or properties.
3. Add or enable the **is column resync enabled** property at the top of the Node configuration.
4. Commit and deploy. Presync will reconcile metadata with the actual table state in Snowflake.

### Option C: Disable and Re-enable Nodes for Deployment

Use this as a last resort when:

- Options A and B have failed to resolve the issue
- There are significant structural differences between Coalesce and Snowflake that can't be reconciled
- You've changed Node types (for example, from table to dynamic table)
- You're dealing with complex metadata corruption issues
- You're willing to re-create the objects completely

:::warning[Data Loss Risk]
This approach drops and re-creates tables in Snowflake, which can result in data loss if not handled properly. Use carefully.
:::

This approach drops the objects in Snowflake and removes them from Coalesce metadata, then re-creates them from the current Node definition.

1. Right-click on the affected Nodes and disable them for deployment.
2. Commit and deploy this change. This drops the objects in Snowflake and removes them from Coalesce metadata.
3. Re-enable the Nodes for deployment.
4. Commit and deploy again. This re-creates the objects based on the current Node definition.

### Option D: Contact Coalesce Support

When the above options don't work, contact Coalesce Support with:

1. Your Environment ID
2. The specific Node IDs that need metadata cleared
3. A description of the sync issue you are experiencing

:::info[What Metadata Clearing Affects]
Clearing metadata only affects Coalesce's understanding of what was last deployed. It doesn't impact your Snowflake objects or Workspace definitions.
:::

**General recommendation:** Start with Option A, escalate to Option B if you encounter metadata sync issues, and only use Option C when the other options fail. Option C should be used carefully as it involves dropping and recreating objects, which can result in data loss if not handled properly.

For details on how Presync works, see [Understanding the Presync Process][].

## Recovery From Failed Deployments

When a deployment fails partway, Coalesce doesn't update Environment metadata. That is intentional: metadata is only updated after a successful deployment. The result can be drift between the database and Coalesce.

### Detecting Metadata Drift

You may notice drift when:

- A deployment fails with errors like "column already exists" or "object not found."
- You made changes off-platform (for example, a data type change) and Coalesce still shows the old schema.
- Advanced Deploy Nodes block future deployments because Coalesce expects a different table state than what exists in Snowflake.

### Clearing Node Metadata So Presync Can Reconcile

To reconcile metadata, use one of the options in [Phase 4: Sync Coalesce Metadata](#phase-4-sync-coalesce-metadata): enable column resync on the affected Nodes, disable and re-enable the Nodes for deployment, or contact Coalesce Support for backend metadata clearing.

After metadata is cleared or resynchronized, run a deployment. Presync compares Coalesce’s view with the actual warehouse state and updates metadata to match. The next deploy will only apply changes that are not already present, avoiding duplicate column adds or other conflicts.

### When To Use Rollback versus Manual Fixes

- **Rollback:** Use [Rollback a Deployment][] when you want to revert data structures to a prior commit. Rollback redeploys the previous commit; it doesn't restore data. Use Snowflake Time Travel to restore data if needed.
- **Manual fixes:** Use the workflow in this guide when you have already changed the table off-platform and need to sync Coalesce. Clearing Node metadata and redeploying lets Presync adopt the current state.

## Alternative: Manual Deletion When Data Retention Is Not Needed

If you don't need to retain the data in the table, you can use a simpler approach:

1. In Coalesce, make the necessary data type changes in the Node mapping and commit.
2. In Snowflake, drop the table. All data in that table will be deleted and lost.
3. Deploy the commit from Coalesce. The deployment will create the table with the expected columns and data types. The table remains empty until the next refresh runs.

:::warning[Data Loss]
Dropping the table permanently deletes all data. Use this approach only when you can afford to lose the data or will repopulate from upstream sources.
:::

## How This Differs From Drop-and-Recreate

A simple DROP and CREATE is faster but has trade-offs:

| Approach | Column order | SCD history | Downtime | Complexity |
| -------- | ------------ | ----------- | -------- | ---------- |
| **Copy + SWAP (this workflow)** | Original order preserved; new columns added using ALTER go to end | Preserved | Minimal (SWAP is atomic) | Higher |
| **DROP + CREATE** | Lost (new table may have different order) | Lost | Full (table missing during re-creation) | Lower |

For Type 2 SCD tables, drop-and-recreate destroys the change-tracking history. For tables with strict column ordering requirements, the copy-table workflow preserves the original column order (new columns added using ALTER statements are appended to the end, regardless of their position in Coalesce).

---

## What's Next?

- [Understanding the Presync Process][]: How Presync reconciles Coalesce with your warehouse
- [Rollback a Deployment][]: How to revert to a prior deployment state
- [Tables Dropped Outside of Coalesce][]: Why off-platform changes can cause deployment failures and metadata mismatches

---

[Understanding the Presync Process]: https://docs.coalesce.io/docs/deploy-and-refresh/deploy/understanding-presync
[Rollback a Deployment]: https://docs.coalesce.io/docs/deploy-and-refresh/deploy/rollback-a-deployment
[Tables Dropped Outside of Coalesce]: https://docs.coalesce.io/docs/faq/build-faq-troubleshooting/tables-dropped-outside-of-coalesce
[Enable Column Resync on Node Types]: https://docs.coalesce.io/docs/build-your-pipeline/sync-node-descriptions#enable-column-resync-on-node-types

---

## Troubleshooting Deploys and Refreshes

You can view the results of a deploy or refresh three ways:

- The Deploy Page
- API
- CLI

## Common Deployment Failure Scenarios

When a deployment fails, the cause often falls into one of these categories. Use the table below to find where to get help.

| Scenario | Where to find help |
| :------- | :----------------- |
| COA deployment errors (permissions, Snowflake view validation, blank columns) | [COA Deployment Error Resolution][coa-deployment-errors] |
| Storage mapping errors (invalid database/schema, orphaned locations, missing mappings) | [Storage Mapping Deployment Failures][storage-mapping-errors] |
| Deployment failed and refresh is blocked | [When deployment fails and refresh is blocked](#when-deployment-fails-and-refresh-is-blocked) |
| Plan failures (unexpected renames, dropped tables, metadata drift) | [Plan failures](#plan-failures) |
| Data type changes in Snowflake (unsupported ALTER COLUMN, copy-table migration) | [Data Type Changes in Snowflake][data-type-changes-snowflake] |
| Deploy plan shows Replace Table, Create Table, or unexpected table re-creation (Snowflake) | [Snowflake Table Re-creation During Deployment][snowflake-table-recreation] |
| Deployment timeouts (`DEADLINE_EXCEEDED`, frozen UI), or runs that never start or stay queued | [Job timeouts and run failures][job-timeouts-run-failures] |
| Snowflake authentication (key pair, auth policies, connection failures) | [Snowflake Key Pair Authentication][key-pair-auth] |

### When Deployment Fails and Refresh Is Blocked

When a deployment fails, refresh is blocked until the deployment is fixed. You have a few options:

- **Fix and retry**: Resolve the underlying error (for example, fix Storage Mappings or permissions), then redeploy. Once deployment succeeds, refresh will run normally.
- **Rollback**: Redeploy the previous successful commit to restore the Environment to a known-good state. See [Rollback a Deployment][].
- **Force refresh (use with caution)**: If you're sure the failed deployment doesn't affect the Nodes you need to refresh, you can use `forceIgnoreWorkspaceStatus` to allow refresh to proceed. See [Managing Refresh Jobs in Failed Deployment Environments][managing-refresh-failed].

:::warning[Use Caution]

Only use `forceIgnoreWorkspaceStatus` when you've confirmed the Nodes in your refresh job are unaffected by the failed deployment. Running refresh in a partially deployed Environment can cause inconsistent or incorrect data.

:::

If the underlying problem is a timeout, frozen UI, or a Job that never starts or stays queued, see [Job timeouts and run failures][job-timeouts-run-failures].

### Plan Failures

Deployment plans can fail when Coalesce detects unexpected changes between your Workspace and the target Environment. Common causes include:

- **Unexpected table renames or dropped tables**: Objects were changed or removed directly in the data platform.
- **Metadata drift**: Column types, names, or structures differ from what Coalesce expects.
- **Storage mapping issues**: Missing or invalid mappings for Storage Locations used by your Nodes.

For details on how Coalesce reconciles these differences, see [Understanding Presync][presync]. For Storage Mapping errors, see [Storage Mapping Deployment Failures][storage-mapping-errors].

## The Deploy Page

On the [Deploy Page][] you can check the status of current or completed runs.

The feed includes:

- Connection details
- Parameters used
- Details for each Node and its steps
  - Query ID
  - Error messages

> Snowflake
>
> Query ID only supported for Snowflake.

1. The run status next to the Environment name.
2. Open each Environment to see the status of deploys and refreshes. They are color coded. You can hover over each one to get a quick overview of the run information.
3. The type of run and the status is color coded.
4. A list of all the runs, the type, and status.

The color will depend on the status:

- **Green**: The run was successful with no errors.
- **Yellow**: The Refresh was successful, but had [column or Node test][] failures.
- **Red**: The Deploy or Refresh failed.

## Individual Run Page

Clicking on any of the Deploy or Refresh statuses will take you to the Individual Run Page.

### Understanding the Individual Run Page

In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.

The heading contains:

- The phase and number of Nodes that failed in that phase. In this example it's one Node in the add phase. `Run failure reason:  RunPhasedDeploy encountered failures: delete phase - none; alter phase - none; add phase - 1`
- The status and number of Nodes. In this example, 11 were successful and 1 failed.

The row contains:

- The failed Node. The failed Node in this example is `V_Nation`.

Double-clicking on the error will open a new window that includes the error message and the SQL for that stage.

The error for this run is `SQL compilation error: Object 'DOCS_TESTING.DEV.STG_NATION1' does not exist or not authorized.`

:::info[Ways to Resolve]

Once the error is resolved, you can [run the Job from the point of failure][], [rollback your deployment][], or [update your pipeline and redeploy][].

:::

### Deploy and Refresh Status

After a deploy or refresh, you'll be able to see which Nodes were run and their status.

| Status     | Description                                                                                            |
| :--------- | :----------------------------------------------------------------------------------------------------- |
| Successful | The Node finished executing all stages successfully.                                                   |
| Waiting    | The Node is waiting to run.                                                                            |
| Queued     | The Node is part of the current run and is waiting execution.                                          |
| Running    | The Node is currently running and stages may have already been completed.                              |
| Failed     | The Node has finished running and encountered an error. Some stages might have completed successfully. |
| Skipped    | A step was skipped. Some stages might have completed successfully.                                     |
| Canceled   | The run was canceled while running. Some stages might have completed successfully.                     |

## API Run Status

Use the [List Run Results][] API to get the results of a deploy or refresh.

In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.

```json
    {
      "nodeID": "97378833-0000-46bc-9582-e68ded3ae016",
      "runState": "error",
      "isRunning": false,
      "name": "STG_ORDERS",
      "queryResults": [
        {
          "endTime": "2024-07-16T15:13:19.357Z",
          "error": {
            "errorString": "SQL compilation error:\nDatabase 'DOCUMENTATION_SNOWFLAKE_TEST' does not exist or not authorized.",
            "errorDetail": "002003"
          },
          "isRunning": false,
          "name": "Delete Table",
          "queryID": "01b5b591-0905-106b-0012-7b030fac7362",
          "sql": "\n    DROP TABLE IF EXISTS \"DOCUMENTATION_SNOWFLAKE_TEST\".\"DEV\".\"STG_ORDERS_COALESCE_INTERNAL_TABLE\"",
          "startTime": "2024-07-16T15:13:10.641Z",
          "status": "Failure",
          "success": false
        }
      ]
    },
...
```

The response is organized by `nodeID`. The response will contain the current status, the Node name, the name of the stage that was executed, and the error message if applicable. Review the full response example for all fields.

You can see `"nodeID": "97378833-0000-46bc-9582-e68ded3ae016"`, failed because `"SQL compilation error:\nDatabase 'DOCUMENTATION_SNOWFLAKE_TEST' does not exist or not authorized."`. The database it was trying to run the stage in does not exist.

:::info[Ways to Resolve]

Once the error is resolved, you can [run the Job from the point of failure][], [rollback your deployment][], or [update your pipeline and redeploy][].

:::

## CLI Run Status

Use [`coa runs`][] to get the deployment status.

When deploying or refreshing using the CLI, add the `--out` flag to return in-depth results as JSON. For example, `coa deploy --out debug.json`.

In this example, you'll review a failed deployment, but this can also be applied to Refresh and Jobs.

```bash title="Printed to the terminal"
....
|2024-07-18T17:00:45.155Z|[RENDERER]()|)|info:Pyodide initialization complete in  1.3305933750000003  seconds
|2024-07-18T17:00:45.159Z|[RENDERER]()|)|info:Pyodide initialization complete in  1.3246505830000004  seconds
|2024-07-18T17:00:45.198Z|[RENDERER]()|)|info:Pyodide initialization complete in  1.363098707999999  seconds
|2024-07-18T17:00:55.135Z|[CLI]|(org:alRPfzuHZouUUN0Zupc9)|(env:2)|(user:Owb7B45m0UNSL4OumkDNpcFifTA3)|run:28)()|)|error:An error occurred during Deployment: {
  error: {
    errorString: 'RunPhasedDeploy encountered failures: \n' +
      '        delete phase - 7; \n' +
      '        alter phase - none; \n' +
      '        add phase - 1\n' +
      '      ',
    errorDetail: 'RunPhasedDeploy encountered failures: \n' +
      '        delete phase - e3e488cc-232f-4e7f-a266-16fe8b4dfd79,e38d8549-bfd0-4044-80ef-108ab297b431,e1e2ea88-de6e-4930-8aaa-dd5ffd1851d9,a7ee70b6-d2a3-4a59-acea-f5a0951b6981,97378833-0000-46bc-9582-e68ded3ae016,3a85b97f-6e6e-4d26-8986-8161f1dd378c,7771b0de-037f-4d9a-a1ad-22400fc0414b; \n' +
      '        alter phase - none; \n' +
      '        add phase - bde6322a-45d0-4964-9606-6a0d0a9867d5\n' +
      '      '
  }
}
```

When a deploy finishes, it will return a status in the command line.

The `errorString` will tell you which nodes failed in each phase.

- The delete phase had 7 Nodes that encountered errors.
- The alter phase had no errors.
- The add phase had 1 Node that encountered errors.

The `errorDetail` will return a list of Node IDs that failed for each phase.

### Using JSON Output

By adding `--out` to return the results as JSON, you can get a detailed error message. Here you can see the error details: `SQL compilation error:\nObject 'DOCUMENTATION_DOCS_TESTING.DEV.STG_LINEITEM' does not exist or not authorized."`

```json title="Using --out"
{
  "runResults": {
    "runStartTime": {
      "seconds": 1721322042,
      "nanoseconds": 300000000
    },
    "runEndTime": {
      "seconds": 1721322054,
      "nanoseconds": 238000000
    },
    "runType": "deploy",
    "runStatus": "failed",
    "runID": 28,
    "runResults": [
      {
        "nodeID": "bde6322a-45d0-4964-9606-6a0d0a9867d5",
        "queryResultSequence": {
          "isRunning": false,
          "name": "STG_LINEITEM",
          "queryResults": [
            {
              "exportedRefs": [],
              "sql": "\nSELECT 1 FROM \"DOCUMENTATION_DOCS_TESTING\".\"DEV\".\"STG_LINEITEM\" LIMIT 0",
              "renderEndTime": {
                "seconds": 1721322044,
                "nanoseconds": 129000000
              },
              "type": "sql",
              "fields": [],
              "startTime": {
                "seconds": 1721322043,
                "nanoseconds": 719000000
              },
              "success": false,
              "error": {
                "errorDetail": "002003",
                "errorString": "SQL compilation error:\nObject 'DOCUMENTATION_DOCS_TESTING.DEV.STG_LINEITEM' does not exist or not authorized."
              },
              "rows": [],
              "stageExecutionStartTime": {
                "seconds": 1721322051,
                "nanoseconds": 791000000
              },
              "invalidExportedRefs": [],
              "warehouse": null,
              "queryID": "01b5c13c-0905-144e-0012-7b030fbc081a",
              "isRunning": false,
              "renderStartTime": {
                "seconds": 1721322043,
                "nanoseconds": 719000000
              },
              "stageExecutionEndTime": {
                "seconds": 1721322053,
                "nanoseconds": 565000000
              },
              "endTime": {
                "seconds": 1721322053,
                "nanoseconds": 565000000
              },
              "name": "Validating Source Exists",
              "status": "Failure"
            }
          ]
        },
        "runState": "error"
      },

 
```

:::info[Ways to Resolve]

Once the error is resolved, you can [run the Job from the point of failure][], [rollback your deployment][], or [update your pipeline and redeploy][].

:::

## Node and Column Test Failures

If a Node or column test fails during a **Refresh**, then the Refresh status will be yellow and the test failure will be in the results. The API will also have a `hasTestFailure:true` added.

### Coalesce App

Failures are also visible on the Individual Run Page, or check the color of a Refresh to see the status.

- **Yellow** - The Refresh was successful, but had [column or Node test][] failures.
- **Red** - The Deploy or Refresh failed.

<figure>
  
  <figcaption>Test failures have a yellow status on the deploy page.</figcaption>
</figure>

<figure>
  
  <figcaption>Failed tests will show on the Individual Run Page</figcaption>
</figure>

### API

The following endpoints will have a `hasTestFailure:true` added:
  
- [List Run Results][]
- [List Run][]
- [Get Run][]
- [Get Job Status][]

In the following example, the field `hasTestFailure:true` is present because a test failed. It failed on the test for `"name": "N_REGIONKEY: Unique"`. You can look at the error to see what the failure was.

```json
  "error": {
      "errorString": "Test failed - records matching fail condition were returned",
      "errorDetail": "Test failed - records matching fail condition were returned"
  },
```

The status will be success, while success will be false. This combination also indicates that while the refresh run finished, there was a failure and in this example, it was the column level test for unique.

```json
  "status": "Success",
  "success": false,
```

```json title="List Run Results With Failed Test" {25,65-68,74-75}
{
    "data": [
        {
            "nodeID": "596874d4-5555-4114-9bdc-b2de778fd98c",
            "runState": "complete",
            "isRunning": false,
            "name": "NATION",
            "queryResults": [
                {
                    "endTime": "2025-03-26T15:44:27.156Z",
                    "isRunning": false,
                    "name": "Validating Source Exists",
                    "queryID": "01bb44d0-0907-dbe8-0012-7b031a5930e6",
                    "sql": "\nSELECT 1 FROM \"SNOWFLAKE_SAMPLE_DATA\".\"TPCH_SF100\".\"NATION\" LIMIT 0",
                    "startTime": "2025-03-26T15:44:26.920Z",
                    "status": "Success",
                    "success": true,
                    "warehouse": "COMPUTE_WH"
                }
            ]
        },
        {
            "nodeID": "99293300-f747-4990-a176-e82d2973694e",
            "runState": "complete",
            "hasTestFailures": true,
            "isRunning": false,
            "name": "STG_NATION",
            "queryResults": [
                {
                    "endTime": "2025-03-26T15:44:28.397Z",
                    "isRunning": false,
                    "name": "Truncate Stage Table",
                    "queryID": "01bb44d0-0907-d01f-0012-7b031a590a5e",
                    "sql": "\n\t\t\tTRUNCATE IF EXISTS \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\"\n\t\n\t\t\n\t\n\t\n\t\t\n\t\n\t\n\t\t\n\t\n\t\t\t\n\t\t\t\t\n",
                    "startTime": "2025-03-26T15:44:27.968Z",
                    "status": "Success",
                    "success": true,
                    "warehouse": "COMPUTE_WH"
                },
                {
                    "endTime": "2025-03-26T15:44:29.597Z",
                    "isRunning": false,
                    "name": "Insert STG_NATION",
                    "queryID": "01bb44d0-0907-dadd-0012-7b031a58fbee",
                    "rowsInserted": 25,
                    "sql": "\n\t\n\t\t\t\tINSERT INTO \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\"\n\t\t\t\t(\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_NATIONKEY\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_NAME\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_REGIONKEY\",\n\t\t\t\t\t\n\t\t\t\t\t\t\"N_COMMENT\"\n\t\t\t\t\t\n\t\t\t\t)\n\t\t\t\n\t\n\t\t\tSELECT\n\t\t\t\n                \"NATION\".\"N_NATIONKEY\" AS \"N_NATIONKEY\", \n\t\t\t\n                \"NATION\".\"N_NAME\" AS \"N_NAME\", \n\t\t\t\n                \"NATION\".\"N_REGIONKEY\" AS \"N_REGIONKEY\", \n\t\t\t\n                \"NATION\".\"N_COMMENT\" AS \"N_COMMENT\"\n\t\t\t\n\t\n\t\t\tFROM \"SNOWFLAKE_SAMPLE_DATA\".\"TPCH_SF100\".\"NATION\" \"NATION\"\n\t\n\t\t\t\n\t\n\t\t\n\t\n\t\n\n\n\n\t\n\n\t\n\t\t\n\t\t\t\n",
                    "startTime": "2025-03-26T15:44:28.956Z",
                    "status": "Success",
                    "success": true,
                    "warehouse": "COMPUTE_WH"
                },
                {
                    "endTime": "2025-03-26T15:44:30.739Z",
                    "isRunning": false,
                    "name": "N_NATIONKEY: Unique",
                    "queryID": "01bb44d0-0907-dc64-0012-7b031a58cc06",
                    "sql": "SELECT 1 WHERE EXISTS (\n\t\t\t\n            SELECT \"N_NATIONKEY\", COUNT(*)\n            FROM \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\" \n            GROUP BY \"N_NATIONKEY\"\n            HAVING COUNT(*) > 1\n\t\t\n\t\n\t\t\n\t\n\t\t\n\t\t\t\n)",
                    "startTime": "2025-03-26T15:44:30.327Z",
                    "status": "Success",
                    "success": true,
                    "warehouse": "COMPUTE_WH"
                },
                {
                    "endTime": "2025-03-26T15:44:31.783Z",
                    "error": {
                        "errorString": "Test failed - records matching fail condition were returned",
                        "errorDetail": "Test failed - records matching fail condition were returned"
                    },
                    "isRunning": false,
                    "name": "N_REGIONKEY: Unique",
                    "queryID": "01bb44d0-0907-dbe8-0012-7b031a5930fa",
                    "sql": "SELECT 1 WHERE EXISTS (\n\t\t\t\n            SELECT \"N_REGIONKEY\", COUNT(*)\n            FROM \"TATIANA_GENERAL\".\"DEV\".\"STG_NATION\" \n            GROUP BY \"N_REGIONKEY\"\n            HAVING COUNT(*) > 1\n\t\t\n\t\n\t\t\n\t)",
                    "startTime": "2025-03-26T15:44:31.330Z",
                    "status": "Success",
                    "success": false,
                    "warehouse": "COMPUTE_WH"
                }
            ]
        }
    ]
}
```

## Retry From Failure

If a Job failed, you can retry the Job from the point of failure. You can use the:

- API - Use the [Rerun a Job][] endpoint.
- Coalesce App - You can select the Job that failed and click **Retry From Failure**.
- Scheduler - You can configure [scheduled Jobs][] to retry from beginning or from the point of failure.

When you retry a Job from failure, in the Coalesce app, you'll be able to see the previous run details.

## Rollback Your Deployment

You can redeploy the commit that was deployed just prior to the deployment you wish to rollback. Learn more in [Rollback a Deployment][].

## Redeploy

You can edit your pipeline, commit the changes, and redeploy to fix any errors.

[Deploy Page]: /docs/deploy-and-refresh/overview-of-the-deploy-interface
[coa-deployment-errors]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/coa-deployment-error-resolution
[data-type-changes-snowflake]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/data-type-changes-snowflake-workflow
[snowflake-table-recreation]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/snowflake-table-recreation-during-deployment
[storage-mapping-errors]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/storage-mapping-errors
[key-pair-auth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[job-timeouts-run-failures]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/job-timeouts-and-run-failures
[managing-refresh-failed]: /docs/faq/deploy-refresh-faq-troubleshooting/managing-refresh-jobs-in-failed-deployment-environments
[presync]: /docs/deploy-and-refresh/deploy/understanding-presync
[List Run Results]: /docs/api/coalesce/get-run-results
[`coa runs`]: /docs/coa/version-733-and-above/coa-commands/#-coa-runs-options-command
[Rollback a Deployment]: /docs/deploy-and-refresh/deploy/rollback-a-deployment
[run the Job from the point of failure]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes#retry-from-failure
[update your pipeline and redeploy]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes
[Rerun a Job]: /docs/api/runs/retry-failed-run
[scheduled Jobs]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[column or Node test]: /docs/build-your-pipeline/data-quality-testing/
[List Run]: /docs/api/coalesce/get-runs
[Get Run]: /docs/api/coalesce/get-run
[Get Job Status]: /docs/api/runs/run-status
[rollback your deployment]: /docs/deploy-and-refresh/deploy/rollback-a-deployment

---

## Job Timeouts and Run Failures

Use this guide when a deploy, refresh, or Job run stops making progress, ends with a timeout error, or never appears to start. See [Troubleshooting Deployments and Refreshes][] for the main troubleshooting hub. For performance tuning when slowness is the main symptom rather than a hard failure, see [Troubleshooting Performance Issues in Coalesce][].

## Before You Begin

Use the following as you work through timeouts and queue issues:

- You can review runs on the [Deploy Page][] and open an Individual Run to see per-Node status, errors, and timestamps.
- For how status values such as **Queued** and **Running** are reported in the app and APIs, see [Troubleshooting Deployments and Refreshes][].
- If the Environment’s last deployment failed and refresh will not start, read [Managing Refresh Jobs in Failed Deployment Environments][] before using override flags.

## Tell Stuck or Timed Out Runs Apart From Slow Progress

A long run is not always a failed run. Use these checks before you assume a timeout.

1. Open the Individual Run for the deploy or refresh and watch whether **Duration** counts change over a few minutes. If numbers move and stages complete, the run is likely slow rather than stuck.
2. Compare against recent runs of similar scope. Large graphs or heavy validation naturally take longer.
3. If nothing advances for an extended period, or the UI stops updating while other pages in the Coalesce App load normally, treat the run as stuck or timed out and follow the guidance in the next section.

For warehouse sizing, query history, incremental design, and staging patterns when runs are slow but still progressing, see [Troubleshooting Performance Issues in Coalesce][]. That article focuses on optimization; this one focuses on timeouts, blocked refresh, and runs that do not start.

## When You See DEADLINE_EXCEEDED or a Frozen UI

You might see `DEADLINE_EXCEEDED` or a deploy or refresh that appears frozen in the Coalesce app. These outcomes often share common causes:

- **Large scope** - Many Nodes or heavy object operations in one run.
- **Data platform latency** - The warehouse or engine is slow to return or execute work.
- **Network issues** - Unstable connectivity between your environment and Coalesce or your data platform.

Timeout behavior depends on operation type, platform, and how your organization uses Coalesce. There is no single timeout value that applies to every engine or tenant.

**What to try:**

1. **Retry the run** - Transient platform or network issues sometimes clear on a second attempt.
2. **Reduce parallelism** - When using the API lower the parallelism in run details so fewer Nodes execute at once, which can ease pressure on the data platform and long-running steps.
3. **Run a smaller batch** - Use Job selectors, subgraphs, or a narrower refresh so each run does less work in one pass. Splitting work across multiple runs is often more reliable than one very large run.

If timeouts persist after these steps, contact [Coalesce Support][] with the information in [What to Send Coalesce Support](#what-to-send-coalesce-support).

## When a Run Never Starts or Stays Queued

If a run does not appear, never leaves **Queued**, or a scheduled time passes with no new run, check the following:

- **Job not deployed** - The Coalesce Scheduler runs **deployed** Jobs only. If you created or changed a Job in the Workspace but did not commit and deploy, scheduled runs still use the previous definition or the Job may not run as you expect. See [Scheduling Jobs in Coalesce][] and [Managing Jobs][].
- **Scheduler configuration** - Confirm the Job Schedule is saved, not paused, and that the cron expression is in UTC. Refresh the Job Schedules page to see recent execution status. Email notifications for failures can be configured on the schedule.
- **Credentials and authentication** - Failed authentication can prevent a run from starting or cause immediate failure. Update credentials on the **Environment** where the Job runs. Scheduled Jobs use the credentials of the user who created or last modified the schedule; if you change authentication type or rotate secrets, you may need to update each affected schedule or integration. For the full pattern and automation options, see [Jobs Fail After Changing Authentication or Credentials][] and the credential notes in [Snowflake Key Pair Authentication][].
- **API and CLI triggers** - Integrations must send valid `userCredentials` or an equivalent credential payload on each Start Run request. A missing or expired secret often surfaces as an error on the first step rather than an endless queue; confirm secrets and Environment alignment with your integration docs.

## When Refresh Is Blocked After a Failed Deployment

When deployment is in a failed state, Coalesce blocks refresh until the deployment is repaired or you use an explicit override. For fix, rollback, cautious force refresh, and the `forceIgnoreWorkspaceStatus` warning, see [When Deployment Fails and Refresh Is Blocked][]. For override procedures and flags, follow [Managing Refresh Jobs in Failed Deployment Environments][].

## What to Send Coalesce Support

Include the following so Support can triage quickly:

- **Environment ID** - The target Environment for the run.
- **Run ID** - The identifier shown for the deploy, refresh, or Job run in the Coalesce App, in API responses, or in CLI output.
- **Short description** - What you were running, such as the deploy, refresh, or Job name; what you observed, for example a frozen UI, `DEADLINE_EXCEEDED`, or a queued run with no start; and approximately when it happened, with timestamp and timezone.

If the run is tied to a specific Job Schedule or API integration, mention that context in one or two sentences.

## Related Monitoring and Notifications

You can monitor runs and build alerting using the built-in Job Scheduler email options and Coalesce APIs. See [Job Notifications and Monitoring][] for an overview of scheduler notifications, run status endpoints, and retries.

## What's Next?

- [Troubleshooting Deployments and Refreshes][]
- [Troubleshooting Performance Issues in Coalesce][]
- [Scheduling Jobs in Coalesce][]
- [Managing Refresh Jobs in Failed Deployment Environments][]

---

[Troubleshooting Deployments and Refreshes]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes
[When Deployment Fails and Refresh Is Blocked]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes#when-deployment-fails-and-refresh-is-blocked
[Deploy Page]: /docs/deploy-and-refresh/overview-of-the-deploy-interface
[Managing Refresh Jobs in Failed Deployment Environments]: /docs/faq/deploy-refresh-faq-troubleshooting/managing-refresh-jobs-in-failed-deployment-environments
[Troubleshooting Performance Issues in Coalesce]: /docs/faq/deploy-refresh-faq-troubleshooting/troubleshooting-slow-environments
[Coalesce Support]: https://coalesce.io/contact
[Scheduling Jobs in Coalesce]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Managing Jobs]: /docs/deploy-and-refresh/refresh/managing-jobs
[Jobs Fail After Changing Authentication or Credentials]: /docs/reference/error-library/jobs-failed-due-to-credentials
[Snowflake Key Pair Authentication]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[Job Notifications and Monitoring]: /docs/deploy-and-refresh/monitoring-jobs

---

## Snowflake Table Re-creation During Deployment

When a Snowflake deploy plan shows **Replace Table**, **Create Table**, or **Drop Table** stages, Coalesce is signaling that an object may be re-created or dropped, not only altered in place. Use this guide to read plan output, decide when re-creation is expected, prevent unintended structural changes, and recover if a deploy already ran.

## Before You Begin

You need:

- Access to deploy an Environment and ability to view deployments.
- Familiarity with the Node Types and Packages in your Project. Standard Base Node Types, Advanced Deploy, and Create or Alter Nodes behave differently.
- For recovery steps, permission to redeploy a prior commit and use Snowflake Time Travel on affected tables.

## How to Read the Deploy Plan

Coalesce builds a deployment plan before it runs DDL in Snowflake. Inspect that plan every time you deploy to production or any Environment where table data, grants, or downstream streams matter.

### Review Plan in the Deploy Wizard

During deployment, the **Review Plan** step lists every Node that will change in the target Environment. For each Node, open **View Details** to see generated SQL and the ordered list of stages Coalesce will execute. For the full wizard walkthrough, including the **Changes** tab for metadata-only diffs on altered Nodes, see [Deploy Using the Coalesce App][].

Treat these stage names as re-creation signals:

- **Create Table** - Coalesce creates a new table, often on first deploy to an Environment
- **Replace Table** - Coalesce runs `CREATE OR REPLACE TABLE` on an existing table
- **Drop Table** - Coalesce removes a table or an internal clone table as part of a multi-step alter or materialization change

These stage names usually mean Coalesce is altering in place while preserving data when Snowflake allows the change:

- **Clone Table**
- **Rename Table | Alter Column | Delete Column | Add Column | Edit table description**
- **Swap Cloned Table**

Advanced Deploy Nodes can show **Metadata Update** stages, for example **Metadata Update | Join**, that record configuration changes in comments without running DDL. That is not re-creation. Confirm the stage list includes no **Replace Table**, **Create Table**, or **Drop Table** steps before you assume the deploy is metadata-only.

### Individual Run Page, API, and CLI

After deploy, you can review the same stage names on the Individual Run Page, in [List Run Results][] API responses, or in CLI output when you add `--out` to a deploy command. Each stage includes the SQL Coalesce executed and any error message.

Phased deploy errors reference **delete**, **alter**, and **add** phases. A failure in the delete phase often involves **Drop Table** stages. Failures in the alter phase often involve clone and swap stages.

### Disable Plan Caching While Troubleshooting

If stages look stale or do not match recent Node edits, regenerate the plan with deployment plan caching turned off. See [Caching Deployment Plan][] for how to disable caching in the **Deploy Wizard** or CLI. A full re-render ensures Coalesce evaluates current Node metadata before you deploy.

## When Coalesce Alters Versus Re-Creates

Coalesce prefers non-destructive changes for column-level edits on standard Snowflake Base Node Types (**Work**, **Persistent Stage**, **Dimension**, and **Fact**) when those Nodes are materialized as tables. Logical or configuration changes to the Node definition often trigger full table re-creation instead.

Use the table below to interpret the most common plan patterns for standard base table Nodes. Package-specific Node Types, such as incremental, deferred merge, dynamic tables, and Create or Alter Nodes, follow their own redeployment rules. Open the redeployment section for your Node Type in the [Coalesce Marketplace][] when your Nodes use a specialized Package.

| Your change | Typical deploy behavior | Common stages | Data in the table |
| :---------- | :---------------------- | :------------ | :---------------- |
| Add, drop, or alter a column with a supported type | Alter with clone and swap | **Clone Table**, column alter stages, **Swap Cloned Table** | Preserved |
| Rename the table | Alter | **Rename Table**, often with clone or swap | Preserved |
| Edit table description | Alter | **Edit table description** | Preserved |
| Change join, transformation, DISTINCT, GROUP BY, or ORDER BY | Recreate | **Replace Table** or **Create Table** | Cleared at deploy; reload on refresh |
| First deploy of the Node to the Environment | Create | **Create Table** | Empty until refresh |
| Switch materialization: table, view, or transient | Drop and create | **Drop Table** then **Create Table** or **Replace Table** | Depends on path; treat as re-creation |
| Change Storage Mapping or Storage Location | Drop at old location, create at new | **Drop Table** at prior location, **Create Table** at new location | Not migrated automatically |
| Unsupported column type change | Alter attempt may fail | Clone/swap alter stages, then error | Use copy-table workflow instead |

Column-level changes on standard base Nodes use a clone, alter, and swap pattern so existing rows stay in place when Snowflake accepts the DDL.

:::info[Environment docs and re-creation]

High-level Environment documentation describes column changes using `ALTER` instead of drop and create. That applies to column-only edits on standard table Nodes. Definition and configuration changes on those same Nodes still use `CREATE OR REPLACE` and show **Replace Table** in the plan. Both behaviors are expected.

:::

### Advanced Deploy Nodes

Advanced Deploy uses the same clone and swap path for column-level changes. Many configuration edits, such as join, transformation, or DISTINCT changes, appear as **Metadata Update** stages with `is` and `was` comments instead of DDL.

Materialization switches, such as table to view or table to transient table, use explicit **Drop Table** and **Create Table** or **Replace Table** stages. See the redeployment section for your Node Type in [Snowflake Base Node Types Advanced Deploy][].

### Create or Alter Package

The [Create or Alter Package][] uses Snowflake `CREATE OR ALTER` for many schema changes as a less destructive alternative on supported Node Types. Materialization type switches can still require drop and create. Review the Package redeployment documentation for your Node Type before you deploy breaking DDL to production.

## Common Scenarios

### Plan Shows Replace Table After a Join or Transform Change

You edited the **Join** or **Transform** tab on a Work, Dimension, or Fact table Node. The plan lists **Replace Table**.

This is expected. Coalesce re-creates the table because the materialized query changed. After deploy succeeds, run a refresh or schedule one to reload data. Downstream Snowflake streams on the table can become stale after `CREATE OR REPLACE TABLE`. Re-create or refresh those streams in Snowflake if your pipeline depends on them.

### Plan Shows Create Table on First Deploy

The Node has never been deployed to this Environment, or Coalesce treats the target location as new.

**Create Table** on first materialization is normal. The table exists after deploy but is empty until a refresh loads rows.

### Migrating an Existing Snowflake Table Into Coalesce

You pointed a Node at a table that already exists and holds data populated outside Coalesce. The plan shows **Create Table** instead of adoption.

[Understanding Presync][] compares Coalesce metadata with Snowflake before deploy. When an object already exists with a matching structure at the expected Storage Location, Presync can adopt it instead of re-creating it. Check:

- Database, schema, and table name match the Node **Storage Location** and naming
- Column names and types match what Coalesce generated for the Node
- The deploy role can see the existing object

A naming or location mismatch often looks like an unexpected **Create Table** even when the table already exists elsewhere in Snowflake.

### Storage Mapping or Location Change

You changed a Node **Storage Location** or updated Environment **Storage Mappings** so objects move to a different database or schema.

Coalesce treats objects at the old location as removed and objects at the new location as new. The plan can include **Drop Table** at the old location and **Create Table** at the new one. Coalesce does not automatically copy data between locations.

If objects at the old location were deleted manually before deploy, the **Drop Table** stage can fail because there is nothing to drop. Restore the old schema or object long enough for Coalesce to complete the drop step, or adjust mappings and metadata with your platform admin before you redeploy. See [Storage Mapping Deployment Failures][] for related errors.

### Package or Node Type Upgrade

Upgrading a Package version or switching Node Types can change which edits trigger metadata stages versus DDL. After an upgrade, regenerate the deploy plan with caching disabled and compare stages to your prior successful deploy.

Open the redeployment documentation for the new Package version. Advanced Deploy in particular separates metadata-only updates from drop-and-re-create paths when materialization type changes.

### Unsupported Data Type Change

You changed a column to a type Snowflake cannot convert with `ALTER COLUMN`. The plan may show alter stages, but deploy fails during the alter step with a Snowflake compilation error.

Do not accept a simple drop-and-re-create when column order or Type 2 SCD history matters. Follow [Data Type Changes in Snowflake][] for the copy-table migration workflow instead.

## Impact of Table Re-Creation

When Coalesce runs `CREATE OR REPLACE TABLE` or drops and re-creates a table, Snowflake replaces the table object. Plan for these side effects before you deploy.

- **Data** - Rows in a re-created table are replaced by the new definition. Data returns after refresh for materialized Nodes, but any data loaded only outside Coalesce is gone unless you recover it separately.
- **Grants and privileges** - Privileges do not always carry over on every DDL path the way they do with explicit `COPY GRANTS`. Review grants after re-creation if downstream roles need continued access.
- **Streams** - Validate and refresh stream status in Snowflake after deploy when streams depend on the table.
- **Tags and comments** - Object tags and column comments managed outside Coalesce can be lost on drop-and-re-create paths. Sync descriptions in Coalesce when you rely on column comments in Snowflake.

Snowflake retains dropped table data for [Time Travel][] according to your account retention settings, even when Coalesce re-creates the object.

## Prevent Unintended Re-Creation

Follow these practices before you deploy structural changes to production Environments.

1. Open **Review Plan** and inspect **View Details** for every Node you changed. Stop the deploy if you see unexpected **Replace Table**, **Create Table**, or **Drop Table** stages.
2. Test the same commit in a lower Environment first. Compare stage names and SQL to what you expect from your Node edits.
3. Disable deployment plan caching when stages do not reflect recent Workspace changes.
4. For production schema evolution where re-creation is risky, evaluate the [Create or Alter Package][] on supported Node Types.
5. Before you change **Storage Mappings**, plan how objects at the old location will be dropped and whether you need to migrate data separately.
6. When you onboard existing warehouse tables, verify names, locations, and column structures so Presync can adopt objects instead of creating new ones. See [Understanding Presync][].

:::warning[Stop the deploy when impact is unclear]

If you cannot explain why a Node shows **Drop Table** or **Replace Table**, do not deploy to production. Resolve the underlying Node, mapping, or Package change first, or deploy to a non-production Environment to validate the plan.

:::

## Recover After Re-Creation

### Roll Back Structures

Redeploy the commit that was deployed immediately before the change you want to undo. [Rollback a Deployment][] restores structures in Snowflake:

- New tables or columns from the rolled-back deploy are dropped
- Tables or columns dropped during that deploy are re-created
- Column changes from that deploy are reverted with `ALTER`

Rollback does not restore row data inside re-created tables. Treat rollback as a structural undo, not a data restore.

### Restore Data With Time Travel

When rollback re-creates a table or column but you need the previous rows, use Snowflake Time Travel after rollback completes. The [Rollback a Deployment][] guide links to Snowflake Time Travel documentation for point-in-time recovery.

For unsupported type changes where you already lost history, follow [Data Type Changes in Snowflake][] instead of relying on a simple re-create path.

### Tables Dropped Outside Coalesce

If a managed table was missing before deploy, Coalesce may re-create it automatically. That behavior is related to Presync and metadata reconciliation, not the same as replacing a table because you changed a join. See [Tables Dropped Outside of Coalesce][] for lifecycle guidance and how missing objects interact with deploy.

## What's Next?

- [Understanding Presync][] - How Coalesce adopts, updates, or re-creates objects before deploy
- [Storage Mapping Deployment Failures][] - Errors when mappings or locations are invalid
- [Data Type Changes in Snowflake][] - Copy-table workflow when `ALTER COLUMN` is unsupported
- [Rollback a Deployment][] - Structural rollback and Time Travel for data recovery
- [Snowflake Base Node Types][] - Alter versus re-create triggers for Work, Persistent Stage, Dimension, and Fact Nodes
- [Snowflake Base Node Types Advanced Deploy][] - Metadata stages and materialization switch behavior
- [Create or Alter Package][] - `CREATE OR ALTER` alternative for supported schema changes
- [Caching Deployment Plan][] - Disable caching when plan stages look wrong
- [Troubleshooting Deploys and Refreshes][] - Hub for deployment and refresh errors

[Caching Deployment Plan]: /docs/deploy-and-refresh/deploy/caching-deployment-plan
[Deploy Using the Coalesce App]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment
[Coalesce Marketplace]: /docs/marketplace
[Create or Alter Package]: /docs/marketplace/package/coalesce_snowflake_create-or-alter-package
[Data Type Changes in Snowflake]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/data-type-changes-snowflake-workflow
[List Run Results]: /docs/api/coalesce/get-run-results
[Rollback a Deployment]: /docs/deploy-and-refresh/deploy/rollback-a-deployment
[Snowflake Base Node Types]: /docs/marketplace/package/coalesce_snowflake_base-node-types
[Snowflake Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy
[Storage Mapping Deployment Failures]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/storage-mapping-errors
[Tables Dropped Outside of Coalesce]: /docs/faq/build-faq-troubleshooting/tables-dropped-outside-of-coalesce
[Time Travel]: https://docs.snowflake.com/en/user-guide/data-time-travel
[Troubleshooting Deploys and Refreshes]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes
[Understanding Presync]: /docs/deploy-and-refresh/deploy/understanding-presync

---

## Storage Mapping Deployment Failures

Storage Mapping deployment failures happen when your Environment can't properly connect to target databases and schemas. This guide covers the "No schema found" error, its causes and fixes, plus other common storage mapping issues.

## The "No Schema Found" Error

You may see this error during deployment:

```text
No schema found for location [STORAGE_LOCATION] using database [DATABASE] and schema [SCHEMA] on [platform]. Please contact support for assistance.
```

This error usually occurs when an object in your Project previously referenced a different schema location, and that schema has since been deleted. During deployment, Coalesce attempts to clean up objects from their old locations. If the previous schema no longer exists, presync cannot find it and deployment fails.

## Causes

### Schema Does Not Exist in the Data Platform

Coalesce expects the schema to exist in the target database. If it was never created or was dropped outside Coalesce, presync cannot find it.

- **Typical cases:** New Storage Location, schema dropped in Snowflake, schema renamed or moved.
- **How to confirm:** In Snowflake, run `SHOW SCHEMAS IN DATABASE <database_name>;` and verify the schema exists.

### Missing or Insufficient Permissions

The service account used by Coalesce may not have access to the schema. Snowflake returns no schema when the role lacks `USAGE` on the database or schema.

- **Typical cases:** New environment, role changes, schema created by another role.
- **How to confirm:** Run the same `SHOW SCHEMAS` query as the Coalesce service account or role.

### Trailing or Leading Whitespace in Storage Mappings

Schema names entered manually (for example, using the override toggle) can include invisible spaces. Coalesce looks for the exact string, so `MY_SCHEMA` with a trailing space does not match `MY_SCHEMA`.

- **Typical cases:** Manual override of database or schema in Storage Mappings, copy-paste from docs or emails.
- **How to confirm:** Inspect the mapping value character by character or re-type the schema name.

### Invalid or Placeholder Values in Storage Mappings

Mappings can use placeholders like `"N/A"`, empty strings, or incorrect database or schema names.

- **Typical cases:** New Environment before mappings are set, mappings copied from a template, typos.
- **How to confirm:** Check **Build Settings** → **Environments** → **Storage Mappings** for each Storage Location.

### Orphaned Storage Locations

The database or schema was dropped in the platform but is still referenced in Environment mappings. Coalesce tries to use a location that no longer exists.

- **Typical cases:** Schema or database cleanup, migration, environment reset.
- **How to confirm:** Compare Storage Mappings to what exists in the data platform.

### Workspace versus Environment Mapping Mismatch

Workspace Storage Mappings are for development only. Deployment uses Environment mappings. If Environment mappings are missing or wrong, deployment fails even when the Workspace looks correct.

- **Typical cases:** First deploy to a new Environment, Environment not configured after Workspace setup.
- **How to confirm:** Check **Deploy** tab → Environment → Storage Mappings for the target Environment.

### Uncommitted Storage Mapping Changes

Storage mapping changes must be committed to version control before deployment. Uncommitted changes are not used at deploy time.

- **Typical cases:** Mapping updated in the UI but not committed, deployment from a different branch or commit.
- **How to confirm:** Open the Git modal and ensure all mapping changes are committed and pushed.

### Internal Metadata or Caching Issues (Rare)

In some cases, updating mappings does not resolve the error. Internal metadata may be out of sync.

- **Typical cases:** Complex migration, schema or location changes, prior failed deployments.
- **How to confirm:** After ruling out the above causes, this is usually identified with support.

## Fixes

### Create the Schema in the Data Platform

If the schema is missing:

1. In Snowflake (or your platform), create the schema:

   ```sql
   CREATE SCHEMA IF NOT EXISTS <database_name>.<schema_name>;
   ```

2. Grant the Coalesce service account `USAGE` on the database and schema.
3. Retry deployment.

### Grant Permissions to the Service Account

1. Identify the role used by the Coalesce connection for that Environment.
2. Grant:

   ```sql
   GRANT USAGE ON DATABASE <database_name> TO ROLE <coalesce_role>;
   GRANT USAGE ON SCHEMA <database_name>.<schema_name> TO ROLE <coalesce_role>;
   ```

3. Add any other privileges needed for the objects Coalesce will create (for example, `CREATE TABLE`, `SELECT` on sources).
4. Retry deployment.

### Remove Whitespace from Storage Mappings

1. Go to **Build Settings** → **Environments** and select the target Environment.
2. Open **Storage Mappings**.
3. For the affected Storage Location, re-enter the database and schema names (do not copy-paste).
4. Ensure there are no leading or trailing spaces.
5. Save and commit the changes.
6. Retry deployment.

### Correct Invalid or Placeholder Values

1. Go to **Build Settings** → **Environments** → **Storage Mappings**.
2. For each Storage Location, set valid database and schema values that exist in the platform.
3. Remove or fix any mappings using `"N/A"`, empty strings, or incorrect names.
4. Save and commit.
5. Retry deployment.

### Update or Remove Orphaned Storage Locations

1. In the platform, confirm which databases and schemas exist.
2. In **Build Settings** → **Environments** → **Storage Mappings**, compare each mapping to the platform.
3. For mappings that point to dropped locations:
   - Update to the new database or schema if the objects were moved.
   - Remove the mapping if the location was intentionally deleted.
4. Save and commit.
5. Retry deployment.

### Configure Environment Storage Mappings

1. Go to the **Deploy** tab.
2. Select the target Environment.
3. Open **Storage Mappings** for that Environment.
4. Map every Storage Location used by your Nodes to the correct database and schema for that Environment.
5. Save and commit.
6. Retry deployment.

### Commit Storage Mapping Changes

1. Open the Git modal (version control).
2. Confirm all Storage Mapping changes are staged and committed.
3. Push to the remote if your deployment uses the remote branch.
4. Retry deployment.

### Contact Support (When Self-Service Fixes Fail)

If you have:

- Verified the schema exists and the service account has permissions
- Confirmed Storage Mappings are correct and committed
- Removed whitespace and invalid values
- Checked for orphaned locations

and the error persists, contact [support@coalesce.io](mailto:support@coalesce.io) with:

- Full error message
- Deployment ID
- Storage Location name and database or schema from the error
- Steps you've already tried

## Other Storage Mapping Issues

The following issues can also cause deployment failures. They are covered in the Causes and Fixes sections above, or are distinct scenarios.

### Overlapping Database and Schema Mappings

When your development Workspace and deployment Environment point to the same database or schema, you'll see conflicts and unexpected behavior.

- **How to fix:** Use different databases and schemas for each Environment. For example, use `DEV_DB` for development, `QA_DB` for testing, and `PROD_DB` for production.

### Location Added in Failed Deployment

When a new Node adds a Storage Location and deployment fails before that change is applied, the location is never created in the target Environment. Later deployment plans may fail because Coalesce expects the mapping to exist, but it was never added.

- **How to fix:** Add the missing Storage Location mapping to the Environment in **Build Settings** → **Environments** → **Storage Mappings** with valid database and schema values. Redeploy to apply the change.

### Missing Storage Location Definitions

This happens when Storage Locations exist in your Workspace but aren't defined in the target Environment mappings.

- **How to fix:** Make sure all Storage Locations used in your Nodes are defined in the target Environment's Storage Mappings with valid database and schema references.

### Deployment Plan Generation Failures

The deployment process validates Storage Mappings at multiple stages. A failure at any stage stops the entire deployment.

- **How to fix:** Review deployment plan errors carefully. Fix any Node configuration issues, circular dependencies, or invalid references before trying again.

### Clearing Metadata

In some cases, clearing the Environment metadata can help resolve persistent schema reference issues. If you suspect stale metadata, contact [support@coalesce.io](mailto:support@coalesce.io) for guidance on clearing it.

## Prevention

To reduce storage mapping failures:

- Create databases and schemas before adding Storage Locations or deploying.
- Use consistent naming and avoid manual override when possible.
- Grant the Coalesce service account the required privileges on databases and schemas.
- Commit Storage Mapping changes before deploying.
- Prefer managing objects through Coalesce instead of dropping or renaming them directly in the platform.

## Best Practices

Follow these practices when configuring Storage Mappings:

- **Separate Environment Mappings:** Each Environment (DEV, QA, PROD) should map to completely different databases and schemas to avoid conflicts.
- **Commit Storage Changes:** Always commit Storage Mapping changes to version control before attempting deployment.
- **Verify Database and Schema Existence:** Make sure target databases and schemas exist with proper permissions before deployment.
- **Use Consistent Storage Location Names:** Storage Location names should be the same across all Environments. Only the database and schema mappings should differ.

## What's Next?

- [Understanding the Presync Process][] for how presync validates databases and schemas
- [Storage Locations and Storage Mappings][] for concepts and configuration
- [Support Information][] if you need help from Coalesce

---

[Understanding the Presync Process]: https://docs.coalesce.io/docs/deploy-and-refresh/deploy/understanding-presync
[Storage Locations and Storage Mappings]: https://docs.coalesce.io/docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[Support Information]: https://docs.coalesce.io/docs/get-started/support-information

---

## Can I Implement a Drop-Down Selector?

Currently there isn't a way to offer a multi-select dropdown, but you can use the [Node Config Options][] tabular to get something similar.

To get this, use the following code.

<Tabs>
  <TabItem value="dropdownSelector" label="dropdownSelector" default>

    ```yaml
      - displayName: MultiSelect Options
    attributeName: options_multiselect
    type: tabular
    columns:
      - type: dropdownSelector
        displayName: Option1
        attributeName: Option1
        options:
          - Create or Replace
          - Create If Not Exists
          - Create At Existing Stream
        isRequired: false

      - type: dropdownSelector
        displayName: Option2
        attributeName: Option2
        options:
          - Create or Replace
          - Create If Not Exists
          - Create At Existing Stream
        isRequired: false

      - type: dropdownSelector
        displayName: Option3
        attributeName: Option3
        options:
          - Create or Replace
          - Create If Not Exists
          - Create At Existing Stream
        isRequired: false
        
    isRequired: false
    ```

  </TabItem>
  <TabItem value="stagenode" label="Using a copy of the Stage Node">

        ```yaml   
        capitalized: Copy of Stage
        short: STG
        plural: Stages
        tagColor: '#2EB67D'

        config:
        - groupName: Options
        items:
        - type: materializationSelector
            default: table
            options:
            - table
            - view
            isRequired: true

        - type: multisourceToggle
            enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}" 

        - type: overrideSQLToggle
            enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"
            
        - displayName: Multi Source Strategy
            attributeName: insertStrategy
            type: dropdownSelector
            default: INSERT
            options:
            - "INSERT"
            - "UNION"
            - "UNION ALL"
            isRequired: true
            enableIf: "{% if node.isMultisource %} true {% else %} false {% endif %}"

        - displayName: Truncate Before
            attributeName: truncateBefore
            type: toggleButton
            default: true

        - displayName: Enable Tests
            attributeName: testsEnabled
            type: toggleButton
            default: true
            
        - displayName: Pre-SQL
            attributeName: preSQL
            type: textBox
            syntax: sql
            isRequired: false

        - displayName: Post-SQL
            attributeName: postSQL
            type: textBox
            syntax: sql
            isRequired: false

        - displayName: MultiSelect Options
            attributeName: options_multiselect
            type: tabular
            columns:
            - type: dropdownSelector
                displayName: Option1
                attributeName: Option1
                options:
                - Create or Replace
                - Create If Not Exists
                - Create At Existing Stream
                isRequired: false

            - type: dropdownSelector
                displayName: Option2
                attributeName: Option2
                options:
                - Create or Replace
                - Create If Not Exists
                - Create At Existing Stream
                isRequired: false

            - type: dropdownSelector
                displayName: Option3
                attributeName: Option3
                options:
                - Create or Replace
                - Create If Not Exists
                - Create At Existing Stream
                isRequired: false
                
            isRequired: false
        ```
  </TabItem>

</Tabs>

[Node Config Options]: /docs/build-your-pipeline/user-defined-nodes/node-config-options/

---

## Why Don’t Data Types and Nullability Match Between Coalesce and Snowflake?

## Mismatched Data Types

**Problem:**

Field lengths defined in Coalesce may not match those in Snowflake. For example, Coalesce might show `VARCHAR(8)` while Snowflake shows `VARCHAR(167890)`. Changing the lengths in Coalesce and recreating the node does not fix the mismatch.

**Explanation:**

Mismatches between data types occur because Snowflake [doesn’t require explicit data types for Dynamic Tables][]. The initial version of the Dynamic Table package didn’t apply data types defined in the mapping grid to the resulting SQL. This caused discrepancies between what’s defined in Coalesce and what Snowflake actually uses.

## Nullability Settings Not Applied

**Problem:**

Nullable field settings defined in Coalesce don’t always match the behavior in Snowflake. For example, a field marked as `FALSE` in Coalesce may appear as `NULLABLE` in Snowflake.

**Explanation:**

Dynamic Tables don’t support explicit nullability declarations in their syntax. Instead, nullability is inferred from the upstream source tables. This means any nullability settings defined in the Coalesce mapping grid won’t be reflected in the final table in Snowflake.

To control nullability, make sure upstream nodes define the desired null behavior.

## To Fix

As of [Dynamic Tables][] version 1.1.10 of our Dynamic Tables package, data types are now included in the `CREATE OR REPLACE` statements. Any new Dynamic Table nodes created with this version or later will automatically apply the data types defined in the mapping grid.

If you’re still seeing mismatches, confirm that your project is using version 1.1.10 or higher.

[Dynamic Tables]: /docs/marketplace/package/coalesce_snowflake_dynamic-tables/
[doesn’t require explicit data types for Dynamic Tables]: https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table#optional-parameters

---

## Why Are There Charecters Being Added To My Node Name?

If characters are being added to your Node name, such as a dot (.) or a dash (-), check the following areas:

* Check for blank spaces, dot, or other characters in your Workspace parameters.
* If using a custom Node, there isn't anything that renames the Node.

---

## Quoted Identifiers in Coalesce

When you create or reference objects in Coalesce, the platform wraps identifiers to match your connected data platform. Snowflake Projects use **double quotes**. Databricks and BigQuery Projects use **backticks**. Understanding these differences ensures your SQL runs correctly in Transforms, the Join tab, and Pre-SQL or Post-SQL.

## Quoted Identifiers in Snowflake

Snowflake treats identifiers differently depending on how they are quoted and cased:

* Creating an object in **uppercase** with double quotes allows case-insensitive references later:

    ```sql
    CREATE TABLE "EDW"."DIM"."DIM_CUSTOMER";
    ```

You can then reference it as `DIM_CUSTOMER`, `dim_customer`, or `DiM_cuSToMeR`.

* Creating an object in **lowercase or mixed-case** with double quotes requires exact matching:

  ```sql
  CREATE TABLE "EDW"."DIM"."dim_customer";
  CREATE TABLE "EDW"."DIM"."Dim_Customer";
  ```

  These must be referenced as `"dim_customer"` or `"Dim_Customer"`.

### Current Limitations

Coalesce always applies double quotes to preserve case sensitivity. There is currently **no way to disable this behavior**. If you use lowercase or mixed-case names, they will override Snowflake’s case-insensitive defaults.

### Snowflake Parameter Solution

You can make quoted identifiers case-insensitive with:

```sql
ALTER SESSION SET QUOTED_IDENTIFIERS_IGNORE_CASE = TRUE;
```

At the workspace level in Coalesce, you can configure:

```json
{
  "QUOTED_IDENTIFIERS_IGNORE_CASE": true,
  "QUERY_TAG": "Coalesce"
}
```

## Quoted Identifiers in Databricks

Databricks uses backticks for identifiers. While Databricks itself can support double-quoted identifiers with ANSI settings, **Coalesce does not support generating double-quoted identifiers for Databricks projects**. Instead, Coalesce generates SQL with backticks to match Databricks standards.

## Quoted Identifiers in BigQuery

BigQuery Projects use backticks for identifiers, the same quoting style as Databricks Projects in Coalesce.

While BigQuery SQL can use double-quoted identifiers in some settings, **Coalesce generates backtick-quoted identifiers for BigQuery Projects**. Use backticks when you add column or table names by hand in Transforms or the Join tab.

Example generated pattern:

```sql
`my_project`.`my_dataset`.`stg_customer`.`c_custkey`
```

If you copy Snowflake examples that use double quotes, replace those identifiers with backticks before you run the Node on BigQuery.

## Formatting Data Transformations

When writing transformations, use the syntax for your Project platform:

<Tabs groupId="quoted-identifiers-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REPLACE({{SRC}}, 'Supplier#', '')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
ROUND(`forecast_hourly_metric`.`temperature`, 1)
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REPLACE(`supplier`.`s_name`, 'Supplier#', '')
```

  </TabItem>
</Tabs>

### Platform-Specific Code Generation

When you create a Databricks or BigQuery Project in Coalesce, generated SQL uses backticks for identifiers. [Helper tokens][] such as `{{SRC}}` and [ref functions][] resolve using that Project's quoting style.

[Helper tokens]: /docs/reference/helper-tokens
[ref functions]: /docs/reference/ref-functions

---

## Handling Duplicate Records in SCD Type 2 Dimension Nodes

## Issue

When using SCD Type 2 dimension nodes, you may encounter errors when business keys aren't unique enough for a new record to be inserted, instead resulting in an error.

## Solution

The error occurs when multiple rows in the source data match the same business key in the target table. This prevents the merge operation from determining which record should replace the existing one in the target.

* **Re-evaluate and adjust business or merge keys**:
  * Consider using composite keys. For example `FIRST_NAME` + `LAST_NAME` instead of just `LAST_NAME`.
  * Use the ID column as the business key.
* **Set a session variable in the Pre-SQL**:
  * `alter session set ERROR_ON_NONDETERMINISTIC_MERGE = true;`
  * This may result in data loss since it will pick records at random to include or exclude in the merges.

---

## How Do I Create A Node From An Existing Table?

1. First, add the table in your data platform as a [Storage Location][].
2. The create a [Storage Mappings][] for the data you added.
3. Add it as a [data source][]to the DAG.

[Storage Location]: /docs/get-started/coalesce-fundamentals/environments/
[Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments/
[data source]: /docs/setup-your-project/add-a-data-source

---

## Multiple Workspace Connections

<!-- vale Coalesce.FirstPerson = NO -->

## Can I Have Multiple Connections Saved for a Workspace?

Each Workspace can only have one connection saved to a Snowflake account. Connection information is stored at the branch level, so each Git branch supports one Snowflake account connection.

However, within that single connection, you can access unlimited databases and schemas through Storage Locations and Storage Mappings. This means you don't need multiple connections to work with data across different databases.

Each developer must authenticate individually to Workspaces. If multiple users share a Workspace, each person needs to reconnect using their own credentials.

---

## What's Next?

* [Create a Workspace](/docs/setup-your-project/create-a-workspace)
* [Workspaces](/docs/get-started/coalesce-fundamentals/workspaces)

---

---

## Tables Dropped Outside of Coalesce

You should always manage the lifecycle of database objects, like tables and views, from the Coalesce App. Coalesce needs to have full control over the objects it creates and manages in your data platform.

When you drop a table directly in your data platform, you can cause deployment failures and metadata mismatches.

## What Happens When You Drop a Table?

If you drop a table in your data platform that is managed by a Node in Coalesce, you may see the following issues.

### Deployment Failures

During a deployment, Coalesce may try to access a managed object that it expects to exist. If you dropped that object outside of Coalesce, the deployment will fail because it can't find it.

### Automatic Table Re-Creation

When Coalesce runs a deployment, it compares its metadata with the actual state of your data platform environment. If it detects that a managed table is missing, it will try to re-create it based on the Node definition. For how Coalesce alters versus re-creates tables during normal deploys, and when **Replace Table** or **Create Table** stages are expected, see [Snowflake Table Re-creation During Deployment][].

### Nodes Remain in the Coalesce UI

Even if you drop the underlying table in the data platform, the Node will still appear in your Workspace graph and search results. This can cause confusion because the UI shows a Node that doesn't point to a real table.

### Metadata Mismatches

Dropping tables outside of Coalesce creates a mismatch between the Coalesce metadata and your data platform environment. This can lead to unexpected errors and behavior during builds and deployments.

## Best Practice: Deleting Nodes in Coalesce

The correct way to drop a table is to delete the corresponding Node from your graph in Coalesce.

- Open your **Workspace**.
- Find the **Node** that represents the table you want to drop.
- Right-click the **Node** and select **Delete Node**.
- Commit and deploy this change.

This process ensures that both the table in the data platform and the metadata in Coalesce are removed correctly and stay in sync.

[Snowflake Table Re-creation During Deployment]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/snowflake-table-recreation-during-deployment

---

## Managing Refresh Jobs in Failed a Deployment Environments

## Issue

- **Problem**: Unable to run refresh jobs when the target environment deployment is in a failed status.
- **Context**: Not all jobs require deployment and you need to run jobs independent of deployment status.

## Solution

:::warning[Fixing the Deploy is Recommended]

We recommend fixing the deployment, instead of ignoring the failed state. Only use the provided solution if you are sure this is the best path. Use caution when using the  `forceIgnoreWorkspaceStatus`flag and confirm that the nodes in the job you want to refresh are unaffected by the failed deploy.

:::

Use the `forceIgnoreWorkspaceStatus` flag. This flag will allow refresh jobs to proceed even when the environment is in a failed state, targeting unaffected parts of your Directed Acyclic Graph (DAG).

### API

Review [Start Run][].

```json JSON
curl --request POST \
     --url https://app.coalescesoftware.io/scheduler/startRun \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "runDetails": {
    "parallelism": 16,
    "forceIgnoreWorkspaceStatus": true
  },
  "userCredentials": {
    "snowflakeAuthType": "Basic"
  }
}
'
```

### CLI

Review [COA refresh options][].

```bash
coa refresh --forceIgnoreEnvironmentStatus true
```

[COA refresh options]: /docs/coa/version-733-and-above/coa-commands#-coa-refresh-options
[Start Run]: /docs/api/runs/startrun

---

## Troubleshooting Performance Issues in Coalesce

## Check Your Local Environment

Start by ruling out issues with your local setup. These are the most common sources of slowness and are the quickest to fix.

* **Use a Supported Browser:** Confirm that you are using **Google Chrome**. Other browsers are not officially supported and may cause unexpected behavior.
* **Check Your Network Connection:** As a browser-based application, Coalesce requires a strong and stable network connection to perform optimally. Check if other websites are loading quickly or run a [network speed test][].
* **Review Security Software:** Corporate security software can sometimes block components of the Coalesce service, leading to performance degradation. Work with your security or IT team to ensure that Coalesce is allowed and not being blocked.
* **Free Up System Resources:** If your computer's memory (RAM) is nearly full, you may see an impact on Coalesce's performance. Close unnecessary applications and browser tabs to free up resources.

## Pipeline Design and Optimization

An inefficient pipeline design can significantly impact run times, regardless of the underlying data platform. Review your graph for common performance issues.

* **Review your data pipeline** Operations like `GROUP BY`, `DISTINCT`, and `ORDER BY` are computationally expensive and can slow down your jobs. Use these only when necessary.
* **Use persistent staging tables.** Instead of creating complex, nested views, use persistent staging tables to break up long-running operations and improve parallel execution.
* **Break down complex pipelines.** Large, monolithic pipelines can be difficult to manage and optimize. Break complex pipelines into more manageable, separable chunks.
* **Use incremental loading.** For large datasets, use proper incremental loading strategies to process only new or changed data, rather than performing full table scans with every run.

## Snowflake-Specific Optimizations

If you are using Snowflake, these optimizations may help.

* **Check warehouse sizing.** Make sure your Snowflake warehouse is the right size for your jobs. If you see jobs waiting in line or taking too long, try using a larger warehouse.
* **Check query details.** Use Snowflake's query history to look for performance bottlenecks like large table scans or disk spilling, which indicates the warehouse is too small for the operation.
* **Consider clustering keys or search optimization.** For very large tables, implementing a clustering key or using Snowflake's search optimization service can dramatically improve query performance.
* **Check how many jobs are running at once.** If multiple jobs are running at the same time, you may see them waiting in line. Check your warehouse setup to make sure it can handle all the jobs running at once.

If you are still experiencing slowness after checking these items, we encourage you to [open a request][] with Coalesce Support for further investigation.

[network speed test]: https://www.speedtest.net/
[open a request]: mailto:support@coalesce.io

---

## Troubleshooting Browser Login Issues and Blank Screens

If you’re unable to access Coalesce after logging in, or you see a blank or spinning screen, the issue is often related to network restrictions or blocked domains. These problems commonly occur in secured corporate environments where firewalls, VPNs, or SSL inspection tools interfere with required Coalesce services.

This guide explains how to identify and resolve browser-based login issues and blank screens.

## Network and Firewall Configuration

The most common cause of blank or frozen screens is network filtering that blocks required Coalesce domains. Work with your IT or network team to allowlist the following domains for outbound HTTPS traffic:

- `https://storage.coalescesoftware.io/`
- `https://app.coalescesoftware.io`
- `https://*.app.coalescesoftware.io`
- `https://firestore.googleapis.com`
- `https://firebasestorage.googleapis.com/`
- `https://identitytoolkit.googleapis.com/`
- `https://securetoken.googleapis.com/`

:::tip[Allowlist Base Domains, Not Version-Specific Paths]
Allowlist the base domains (for example, `storage.coalescesoftware.io`), not version-specific URLs. Versioned paths change with each release and will break when Coalesce updates.
:::

When these domains are blocked, essential assets such as configuration panels, authentication components, and Workspace data cannot load.

If you use EU, APAC, or AWS regions, see [Network Requirements][] for the full list of regional domain variants.

<ButtonLink to="/docs/setup-your-project/setup-requirements/network-requirements">
  Review all Network Requirements
</ButtonLink>

## Browser Troubleshooting Steps

Before making network changes, try these browser troubleshooting steps:

1. Test in a private or incognito window to rule out browser extensions or cached data.  
2. Clear cache and cookies for both Coalesce and any identity providers used for login.  
3. Try a different browser. Coalesce officially supports Google Chrome.  
4. Verify your login URL format. Use your organization’s subdomain instead of the generic login page:  
   `https://yourorg.app.coalescesoftware.io`

## Common Network-Related Issues

Network security tools such as corporate VPNs, SSL inspection, and services like ZScaler can interfere with Coalesce authentication and UI loading.  
If you encounter persistent loading screens or CORS (Cross-Origin Resource Sharing) errors, confirm that your organization’s security tools aren’t filtering or modifying Coalesce traffic. CORS errors often indicate that a proxy or firewall is blocking or modifying responses, rather than a Coalesce configuration issue.

## SSO-Specific Issues

If your organization uses Single sign-on (SSO), confirm that both configuration and network access are correct.

Check that:

- All SSO fields are configured correctly, including Authority, Subdomain, Authorization Server, and OIDC Client ID.  
- Network policies allow outbound access to Coalesce IP addresses for your region.  
- Redirect URIs in your identity provider configuration match exactly with those in Coalesce.

## When to Contact Your IT Team

If the issue continues after following these steps:

- Review corporate firewall and proxy logs for blocked Coalesce domains.  
- Confirm that all domains listed in the Network and Firewall Configuration section are allowlisted.  
- Check for SSL inspection or content filtering on the listed domains.  
- If your firewall flags `storage.coalescesoftware.io` as malware, this is typically a false positive. A VirusTotal scan of the CDN content can verify that it's safe.

---

[Network Requirements]: https://docs.coalesce.io/docs/setup-your-project/setup-requirements/network-requirements

---

## Coalesce EMEA Terms and Conditions

<!-- markdownlint-disable -->
<!-- vale off -->

These Coalesce Terms and Conditions together with the order to which they are attached (the “**_Order_**” and collectively this “**_Agreement_**”) constitute an agreement between Coalesce Automation, Inc., a California Corporation (“**_Coalesce_**,” “**_we_**” or “**_us_**”), and you (“**_Customer_**” or “**_you_**”), dated as of the date set forth on the Order (the “**_Effective Date_**”). Coalesce and Customer are sometimes referred to collectively as the “**_Parties_**” and individually as a “**_Party_**.”

Coalesce and Customer agree as follows:

### 1\. Definitions

Capitalized words used but not defined in this Agreement have the meanings in **Appendix B**.

### 2\. Customer’s Rights in the Services

2.1 

**Rights to Use the Services (“_the Subscription_”)**. Coalesce hereby grants to Customer a limited, nonexclusive, nontransferable, non-sublicensable, revocable right during the Subscription Period to (a) create an Account, (b) access, use, implement and operate, as applicable, the Services and the Coalesce Platform, and (c) invite and enable up to the number of Customer employees or other designees (“**_Authorized Users_**”) permitted by Customer’s Usage Plan on the Order to create an Account and access and use the Services and the Coalesce Platform. Customer is responsible for all Authorized Users’ compliance with this Agreement.

2.2

**Restrictions; Limitations**. Customer may not use the Services or the Coalesce Platform in any manner or for any purpose other than as expressly permitted by this Agreement. Without limitation of the foregoing, the rights granted under this Section 2 do not include or authorize: (a) modifying, disassembling, decompiling, reverse engineering or otherwise making any derivative use of the Services or the Coalesce Platform or using or accessing the Services or the Coalesce Platform to build a competitive product or service; (b) using any data mining, robots or similar data gathering or extraction methods except as provided by the Services or the Coalesce Platform; or (c) using or exploiting (whether commercially or non-commercially) the Services or the Coalesce Platform other than for their intended use. The rights granted under this Section 2 are conditioned on Customer’s and its Authorized Users’ continued compliance with this Agreement.

2.3

**Changes to Coalesce Platform**. Coalesce may change the features, functionality or other aspects of the Coalesce Platform or the Services from time to time and without notice to the Customer, provided that such changes do not materially reduce the functionality of the Services.

2.4

**Suspension of Coalesce Platform**. Coalesce may, in its sole discretion, immediately suspend access to or use of the Services by Customer or any Authorized User if Customer or any Authorized User violates a material restriction or obligation in this Agreement, or if in Coalesce’s reasonable judgment, the Services or any component thereof is about to suffer a significant threat to security or functionality. Coalesce may provide advance notice to Customer of any such suspension based on the nature of the circumstances giving rise to the suspension. Coalesce will use reasonable efforts to re-establish the affected Service after Coalesce determines that the situation giving rise to the suspension has been cured. Coalesce may terminate access to the Services if any of the foregoing causes of suspension are not cured within 30 days after Coalesce’s initial notice thereof. Any suspension or termination by Coalesce under this Section 2.4 will not excuse Customer from its obligation to make payment(s) under this Agreement. Any suspension under this Section shall remain in effect until the applicable breach, if curable, is cured.

### 3\. Eligibility; Registration; Support; Professional Services, Data Practices; Coalesce Responsibilities

3.1

**Eligibility**. Customer represents and warrants that it and all Authorized Users are not: (a) residents of any country subject to a United States embargo or other similar United States export restrictions, including Iran, Cuba, North Korea, the Region of Crimea, Sudan or Syria; (b) on the United States Treasury Department’s list of Specifically Designated Nationals; (c) on the United States Department of Commerce’s Denied Persons List or Entity List; or (d) on any other United States export control list.

3.2

**Registration**. Customer and each Authorized User will need to register for an Account through the Services. Each Account may only be used by one person. Customer will ensure that Customer and each Authorized User that is invited to register for an Account: (a) provide accurate, current and complete information when creating an Account; (b) maintain and promptly update all Account information; (c) do not share passwords with others and restrict access to the Account and their computer or mobile device; (d) promptly notify Coalesce if Customer or any Authorized User discovers or otherwise suspects any security breaches related to such user’s Account; and (e) accept responsibility for all unauthorized access and activities that occur under such Authorized User’s Account. Each Account login password should be chosen carefully and not contain any personal or other information that may be easily guessed by anyone else.

3.3

**Violations**. Customer is liable for all activity that occurs under Authorized Users’ Accounts, Authorized User’s compliance with this Agreement, and any of use, misuse or unauthorized use (including by third parties) of Accounts, and Coalesce reserves the right to terminate the account of any Authorized User for any such unauthorized use. The acts or omissions of any Authorized User or third party under an Authorized User’s Account are considered the Authorized User’s acts or omissions, as applicable. Customer will immediately notify Coalesce of any such violations and take immediate action to remedy such violations.

3.4

**Support**. Coalesce will provide Customer direct contact information for a designated representative (“**_Customer Success Manager_**”) to provide first line support and communication regarding the Services. During the Term, Coalesce will also provide reasonable levels of support to Customer and Authorized Users relating to the Services during normal business hours (9:00 to 17:00 Central European Time, Monday through Friday, excluding holidays) in accordance with the Service Level Addendum attached hereto as **Appendix C**.

3.5

**Professional Services**. Coalesce will provide Customer with the Professional Services, if any, identified on the Order.  Customer agrees to pay any Professional Services Fees identified on the Order in accordance with Section 4.2.

3.6 

**Coalesce Responsibilities**

(a) Coalesce Personnel. Coalesce is responsible for the performance of its employees and contractors under this Agreement. Coalesce may use non-employee contractors for the purpose of providing the Services.

(b) Security; Third Party Service Providers. Customer acknowledges that Customer Data is processed on a distributed network owned and maintained by trusted, industry-standard Third Party cloud providers, that are responsible for securing the network. Coalesce will not make any substantive changes to the Third Party Service provider’s applicable services that would be reasonably anticipated to result in any material loss of security, functionality or performance of the Services. Coalesce will implement and maintain during the Term appropriate technical and organizational measures designed to protect Customer Data from unauthorized access, use and disclosure, including the following: (1) use of Third Party Service providers to provide secure hosting services; (2) physical controls designed to secure its facilities used to process Customer Data from unauthorized access; (3) user authentication and access controls that are designed to ensure that access to Customer Data is limited to individuals with a need to know for purposes of performing services under the Agreement; (4) screening procedures for employees who may have access to Customer Data; (5) procedures designed to ensure that persons authorized to process personal data have committed in writing to maintain the confidentiality of personal data or are under an appropriate statutory obligation of confidentiality; (6) measures for logging and monitoring of the details of all processing of Customer Data; (7) measures designed to ensure that all Customer Data is compartmentalized or otherwise logically distinct from other information of Coalesce; and (8) controls designed to protect the Coalesce networks, devices and systems from malware and unauthorized software.

### 4\. Fees, Payments and Taxes

4.1 

**Fees**. Customer will pay the fees described in the Order (collectively, the *“Fees”*) for the subscription identified on the Order together with any Professional Services Fees identified on the Order.  Fees for additional Authorized Users will be prorated through the end of the current Service Term and will be invoiced to you in the month after the additional Authorized Users are granted access to the Software.

4.2

**Invoiced Payment**

(a) Invoicing schedule for Fees. Coalesce will issue an invoice for the total amount of the Fees for the Initial Subscription Period within 30 days of the commencement of the Initial Subscription Period. Coalesce will issue an invoice for the total amount of the Fees for any Renewal Subscription Period within 30 days of the commencement of the Renewal Subscription Period.  For any Fees for Professional Services or any other fees due, Coalesce will issue invoices after the end of each calendar month during the delivery of the Professional Services.

(b) Payment terms. Customer will pay invoiced Fees without offset or deduction at the address or account for Coalesce set forth on the applicable invoice within 30 days of Customer’s receipt of the corresponding invoice.

(c) Claims. If Customer believes that Coalesce has invoiced Customer incorrectly, Customer must contact Coalesce no later than 30 days after the date of the invoice in which the claimed error or problem appeared in order to receive an adjustment or credit, if any. Inquiries should be directed to [ar@coalesce.io](mailto:ar@coalesce.io).

4.3

**Pricing and Availability**. Prices are in the currency shown on the Order.  Coalesce reserves the right to change the Fees or applicable charges and to institute new charges and Fees at the end of each Subscription Period, upon 60 days’ prior notice to Customer (which may be sent by email).

4.4

**Taxes**. All Fees and other amounts payable by Customer under this Agreement are exclusive of taxes and similar assessments. Customer is responsible for all sales, use, and excise taxes, and any other similar taxes, duties, and charges of any kind imposed by any national, federal, state, or local governmental or regulatory authority on any amounts payable by Customer hereunder, other than any taxes imposed on Coalesce’s income.

### 5\. Term and Termination

5.1

**Term**. The term of this Agreement will commence on the Effective Date and will continue for so long as a Subscription Period is in effect unless and until terminated pursuant to Section 2.4, 5.3, 5.4 or 6.2 of this Agreement (the “**_Term_**”).

5.2

**Renewal**. This Agreement will automatically renew for a period of 12 months upon the conclusion of the original term unless a written cancellation notice is sent to and received by ar@coalesce.io at least 45 days prior to the expiration of the original term. Coalesce may increase Fees upon renewal, and will communicate any increase in Fees at least 60 days prior to the expiration of the original term.

5.3

**Cancellation Policy**. Customer may cancel its Subscription upon at least 45 days’ prior to the end of the then-current Subscription Period by notifying Coalesce by email. Customer will be responsible for all charges (including any applicable taxes and other charges) incurred with respect to Fees processed prior to the cancellation of Customer’s Subscription. Without limiting the foregoing, except as otherwise set forth in Section 5 with respect to Customer’s termination for Coalesce’s material breach of this Agreement, Customer will not receive a refund for any partial Subscription Period or any renewal that occurs during the 30 day notice period. 

5.4

**Termination for Material Breach**. Either Party may terminate this Agreement, effective on written notice to the other Party, if the other Party materially breaches this Agreement, and such breach: (a) is incapable of cure; or (b) being capable of cure, remains uncured 30 days after the non-breaching Party provides the breaching Party with written notice of such breach.

5.5

**Effect of Termination**. In the event of any termination of the Term:

(a) all of Customer’s and each Authorized User’s rights (including the license granted in Section 2.1) under this Agreement will immediately terminate and Customer and all Authorized Users will immediately cease any access or use of the Services;

(b) Customer will be responsible for all charges (including any applicable taxes and other charges) incurred with respect to Fees processed prior to the termination of the Term;

(c) if Coalesce terminates the Term for material breach by Customer under Section 5.4, then Customer will remain responsible for the remaining balance of any Fees and Customer must pay within 30 days of termination all such amounts, as well as all sums remaining unpaid for other Orders under the Agreement plus related taxes and expenses; and

(d) Sections 1, 2.2, 3.3, 4, 5.3, 6 through 11 of this Agreement, together with any other provisions that by their nature are intended to survive, will continue to apply in accordance with their terms.

If Customer terminates the Term for material breach by Coalesce under Section 5.2, then Coalesce shall refund to Customer within 30 days of termination any unused pre-paid Fees on a pro rata basis for the remaining Term following the month in which the termination is effective.  Except for the foregoing, Customer will not receive a refund for any partial Subscription Period.

### 6\. Indemnification

6.1

**By Customer**. Customer will defend, indemnify, and hold harmless the Coalesce Parties from and against all claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys’ fees) arising out of or relating to any third party, Contact or Authorized User claim concerning: (a) Customer’s or Authorized Users’ unauthorized use of the Services including use of the Coalesce Platform other than as permitted under this Agreement; (b) any Customer Data or other data or content related to Customer, Authorized Users or Contacts which Customer provides, uploads, or inputs into the Services; or (c) the combination of the Customer Data with other applications, content or processes. If Coalesce is required to respond to a compulsory legal order or process described above, Customer will also reimburse Coalesce for reasonable attorneys’ fees, as well as the time and materials spent by Coalesce’s employees and contractors responding to the compulsory legal order or process at Coalesce’s then-current hourly rates.

6.2

**By Coalesce**. Coalesce will defend, indemnify, and hold harmless Customer from and against all claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys’ fees) arising out of or relating to any third party claim that alleges that: (a) the Coalesce Platform or Services infringe, misappropriate, or violate any third party intellectual property right or (b) Coalesce has suffered a security breach as a result of its failure to meet its security obligations under this Agreement and such breach resulted in a loss of sensitive, confidential, or personally identifiable Customer Data. Further, should the Coalesce Platform or the Services become, or in Coalesce’s opinion are likely to become, the subject of a claim of infringement or misappropriation, Coalesce will, at its election and expense, (1) obtain for Customer the right to continue using the Coalesce Platform or the Services, pursuant to the terms and conditions of this Agreement, or (2) replace or modify the Coalesce Platform or the Services to become non-infringing or non-misappropriating but functionally equivalent. If neither (1) nor (2) is commercially feasible, Coalesce may terminate this Agreement. Coalesce’s indemnity obligations under this Section 6.2 do not apply to any claim resulting from: (w) Customer, any Authorized User or Contact’s acts or omissions; (x) use not in accordance with this Agreement; (y) modifications, damage, misuse or other action of Customer or any third party; or (z) any failure of Customer to comply with this Agreement.

6.3

**Procedure**. For any claims under this Section 6, the indemnifying party will (a) give the indemnified party prompt written notice of the claim; (b) provide such assistance in connection with the defense and settlement of the claim as the indemnified party may reasonably request; (c) obtain the indemnified party’s written consent prior to (1) selecting and retaining counsel to defend against any claim under this Section 6 and (2) agreeing to any settlement; and (d) comply with any settlement or court order made in connection with the claim (e.g., related to the future use of any infringing Coalesce product or service).

### 7\. Ownership

7.1

**The Services and the Coalesce Platform**. As between Coalesce and Customer, Coalesce owns all right, title, and interest in and to the Services and the Coalesce Platform, together with all Intellectual Property Rights therein or thereto. Except as otherwise specified in Section 2.1 of this Agreement, Customer does not obtain any rights under this Agreement from Coalesce to the Services or the Coalesce Platform, including any related Intellectual Property Rights.

7.2

**Feedback**. Customer and Authorized Users may voluntarily provide Coalesce with Feedback and may make Authorized Users available to Coalesce on a reasonable basis for this purpose. Customer will not, and will ensure Authorized Users do not, provide any such Feedback to any third party without Coalesce’s prior written consent in each instance. Any Feedback Customer or Authorized Users provide to Coalesce will become the sole property of Coalesce. Coalesce will own, and Customer and Authorized Users hereby assign, all exclusive rights, including all Intellectual Property Rights in and to Feedback, excluding, however, any Customer Data included in such Feedback (if any), and Coalesce will be entitled to the unrestricted use and dissemination of Feedback for any purpose without acknowledgment or compensation to Customer or any Authorized Users.

7.3

**Trademarks**. Each Party owns all right, title and interest in and to such Party’s Trademarks and any goodwill arising out of the use of such Trademarks will remain with and belong to such Party and its licensors. Neither Party’s Trademarks may be copied, imitated or used without the prior written consent of the other Party or the applicable trademark holder; provided, that Customer hereby grants to Coalesce a limited right to use Customer’s Trademarks solely as necessary to provide the Services to Customer hereunder. Additionally, Coalesce may, without Customer’s consent, include Customer’s name and other indicia in its lists of current or former customers in promotional and marketing materials.

7.4

**Customer Data and Personal Data**

(a) In connection with the supply of the Services and the performance of this Agreement, Coalesce will process Customer Data on behalf of Customer.  Except as set out in sub-Section (c), Customer will own all Customer Data and hereby instructs and authorizes Coalesce to use Customer Data to provide the Services to Customer and its Contacts.

(b) Both Parties agree to comply with their obligations pursuant to GDPR and any other applicable data protection legislation in relation to their processing of Personal Data provided by the other Party.

(c) For Personal Data that is provided in order for Coalesce to supply the Services and perform its obligations under this Agreement (such as contact details for Authorised Users and contract manager contact details), Coalesce shall be the Data Controller.

(d) For Personal Data that is provided as part of the Customer Data and input by the Customer or Authorised Users to the Coalesce Platform,  Customer will own all such Personal Data and Coalesce will process the Personal Data solely for the purpose of providing the Services to Customer (“**_the Purpose_**”). In respect of this Personal Data, Coalesce is the data processor and will comply with the [Data Processing Addendum](https://coalesce.io/data-processing-addendum/).

(e) Where necessary for the Purpose, Coalesce has the right to disclose Customer Data to its service providers and to authorize such service providers to process the Customer Data in accordance with the terms of this Agreement solely in connection with the Purpose.

(f) With regard to the processing of Personal Data outside of the European Economic Area, the Parties will make all arrangements and take all measures required and necessary under the GDPR.

(g) Customer will comply with the requirements of GDPR with regard to the use of Personal Data by the Customer in the receipt and use of the Services and will provide all legally required notices and collect all legally required consents to ensure that Customer and Coalesce may collect, use, disclose and otherwise process Customer Data, including any Personal Data, in accordance with the terms of this Agreement.

(h) Coalesce may use data related to Customer’s use of the Services in an aggregated statistical and anonymous manner (“**_Aggregated Customer Usage Data_**”) to monitor the performance of the Services, improve the Services, and to develop new product and service offerings, in each case without accessing or analyzing the content of Customer Data. Coalesce may disclose Aggregated Customer Usage Data to third parties.

(i) Without limiting the foregoing, Coalesce will not sell such Customer Data.

### 8\. Limited Warranties and Remedies

8.1

**Warranties**

Coalesce represents and warrants that: (a) the Coalesce Platform will perform in all material respects with the applicable Documentation when operated in accordance with the applicable Documentation; (b) it will provide the Services and any Professional Services with commercially reasonable care and skill and in material compliance with applicable laws; (c) Coalesce is the exclusive owner of the Coalesce Platform and Services, or otherwise has the right to provide access to the same to Customer, and that neither the Coalesce Platform  and Services nor Customer’s access to or use of the same infringes, violates, or misappropriates the patent, copyright, trademark, trade secret, or other Intellectual Property Rights of any third party; (d) there exists no agreement or restriction that would interfere with or prevent Coalesce from entering into this Agreement or rendering Services or providing the Services or Professional Services described herein; and (e) the Coalesce Platform is regularly scanned for viruses, worms, Trojan horses or similar software, hardware, system, or combinations thereof with the potential to corrupt, interfere, or otherwise affect access to the Services.

8.2

**Remedy**

Customer must give Coalesce notice of a material defect or nonconformance within 30 days from when Customer becomes aware of the such defect or nonconformance. Coalesce’s sole obligation with respect to a breach of the warranties in Section 8.1 will be to use commercially reasonable efforts to correct any nonconformance of the Services.

8.3

**Exclusion of Warranties**

The warranty set out in Section 8.1 is the only warranty given by Coalesce or any Coalesce Party in relation to the supply of the Coalesce Platform, the Services and the Professional Services.  All other warranties, express or implied by law or a course of dealing, including any warranties that the Services are fit for Customer’s purpose or will be available without interruption, are excluded.

### 9\. Limitations of Liability; Insurance

9.1

**Force Majeure**

Neither Party will be liable for, or be considered to be in breach of or default under this Agreement on account of, any delay or failure to perform as required by this Agreement as a result of any cause or condition beyond such Party’s reasonable control (including any act or failure to act by the other Party). This paragraph will not apply to any payment obligation of either Party.

9.2

**No Limitation**

Nothing in this Agreement shall limit or exclude either party’s liability for:

(a) death or personal injury caused by its negligence, or the negligence of its employees, agents or subcontractors;

(b) fraud or fraudulent misrepresentation; or

(c) or any other liability which cannot be limited or excluded by applicable law.

9.3

**Limited Claims**

Subject to Section 9.2, Coalesce shall not be liable to Customer, any Affiliate of Customer or its officers, employees or contractors or to any Authorised User, whether in contract, tort (including negligence), for breach of statutory duty, or otherwise, arising under or in connection with the Agreement for:

(a) loss of profits;  
(b) loss of sales or business;  
(c) loss of agreements or contracts;  
(d) loss of anticipated savings;  
(e) loss of use or corruption of software, data or information;  
(f) loss of damage to goodwill; or  
(g) indirect, consequential or special loss.

9.4

**Limited Liability**

Subject to Section 9.2, and except in relation to its indemnity obligations set forth in Section 6.2, Coalesce’s and all Coalesce Parties’ total liability to Customer, whether in contract, tort (including negligence), breach of statutory duty or otherwise, arising under or in connection with this Agreement shall be limited to the greater the Fees paid under the Agreement by Customer in the twelve months prior to the event giving rise to the claim.

9.5

**Insurance**

Each Party will obtain and maintain such insurance policies as may be required by applicable law and as follows: (1) Commercial General Liability written on an occurrence form, including personal and advertising injury with limits of $1,000,000 per occurrence and $2,000,000 general aggregate; and (2) Cyber Liability Errors and Omissions with limits of $1,000,000 per occurrence and in the aggregate.

### 10\. Confidential Information

10.1

Each Party reserves any and all right, title and interest (including any Intellectual Property Rights) that it may have in or to any Confidential Information that it may disclose to the other Party under this Agreement. The Recipient will protect Confidential Information of the Discloser against any unauthorized use or disclosure to the same extent that the Recipient protects its own Confidential Information of a similar nature against unauthorized use or disclosure, but in no event will use less than a reasonable standard of care to protect such Confidential Information.

10.2

The Recipient will use any Confidential Information of the Discloser solely for the purposes for which it is provided by the Discloser and may disclose Confidential Information to its directors, officers, employees, agents, advisors, Affiliate, subcontractors and representatives (collectively, its “Representatives”) that have a need-to-know in connection with the performance of this Agreement, provided that such Party advises its Representatives of the confidential nature of the information.  Each Party shall be liable for the unauthorized disclosure of Confidential Information by its Representatives.

10.3

This Section 10 will not be interpreted or construed to prohibit any use or disclosure of information: (a) that was known to Recipient prior to receiving the same from the Discloser in connection with this Agreement; (b) that is independently developed by the Recipient; (c) that is acquired by the Recipient from another source without restriction as to use or disclosure; (d) that is necessary or appropriate in connection with the Recipient’s performance of its obligations or exercise of its rights under this Agreement; (e) that is required by applicable law (e.g., pursuant to applicable securities laws or legal process), provided that the Recipient uses reasonable efforts to give the Discloser reasonable advance notice thereof (e.g., so as to afford the Discloser an opportunity to intervene and seek an order or other appropriate relief for the protection of its Confidential Information from any unauthorized use or disclosure); or (f) that is made with the written consent of the Discloser. In the event of any breach or threatened breach by the Recipient of its obligations under this paragraph, the Discloser will be entitled to injunctive and other equitable relief to enforce such obligations.

### 11\. Miscellaneous

11.1

**Independent Contractors** 

Each Party is an independent contractor and not a partner or agent of the other. This Agreement will not be interpreted or construed as creating or evidencing any partnership or agency between the Parties or as imposing any partnership or agency obligations or liability upon either Party. Further, neither Party is authorized to, and will not, enter into or incur any agreement, contract, commitment, obligation or liability in the name of or otherwise on behalf of the other Party.

11.2

**Reference Program**

Customer may voluntarily consult with Coalesce and work in good faith to agree on quotes and statements about Customer’s experience with the Coalesce Services. If Customer or an Authorized User volunteers such quotes or statements, Coalesce may, at its option, use such quotes and statements in connection with its sales and marketing activities.

11.3

**No Third Party Beneficiaries**

This Agreement does not create any third party beneficiary rights in any individual or entity that is not a Party to this Agreement.

11.4

**Assignment**

Neither Party may assign this Agreement or any right, interest or benefit under this Agreement without prior written consent of the other Party; provided that either party may assign this Agreement or any right, interest or benefit under this Agreement without such prior written consent to an entity that acquires all or substantially all of the business or assets of such party to which this Agreement pertains, whether by merger, reorganization, acquisition, sale or otherwise. Any attempted assignment in violation of the foregoing will be void. Subject to the foregoing, this Agreement will be fully binding upon, inure to the benefit of and be enforceable by any permitted assignee.

11.5

**Nonwaiver**

The failure of either Party to insist upon or enforce performance by the other Party of any provision of this Agreement, or to exercise any right or remedy under this Agreement or otherwise by law, will not be construed as a waiver or relinquishment of such Party’s right to assert or rely upon the provision, right, or remedy in that or any other instance; rather the provision, right or remedy will be and remain in full force and effect.

11.6

**Dispute Resolution Procedures**

If any dispute arises in connection with this Agreement, the Parties agree to enter into mediation in good faith to settle such a dispute and will do so in accordance with the Centre for Effective Dispute Resolution (CEDR) Model Mediation Procedure.  Unless otherwise agreed between the Parties within 14 working days of notice of the dispute, the mediator will be nominated by CEDR.   If mediation fails, each Party shall be entitled to pursue a remedy in the courts and shall also be entitled, at any time to seek court intervention in the case of any disputes related to Intellectual Property or for  any claim for injunctive relief.

11.7

**Severability**

If any provision of this Agreement is deemed unlawful, void or for any reason unenforceable, then that provision will be deemed severable from this Agreement and will not affect the validity and enforceability of any remaining provisions.

11.8

**Applicable Law**

This Agreement will be interpreted, construed and enforced in all respects in accordance with the laws of England and Wales and the Customer hereby consents to the jurisdiction and venue of the courts of England with respect to any claim arising under or by reason of this Agreement.

11.9

**Entire Agreement**

This Agreement, together with any agreements, Order, or other policy or guideline referenced in this Agreement, constitutes the complete and exclusive statement of all mutual understandings between the Parties with respect to the subject matter hereof, superseding all prior or contemporaneous proposals, communications and understandings, oral or written.

### Appendix B - Definitions

“**_Account_**” means a single user electronic account permitting Customer or Authorized Users to access and use the Services.  
“**_Affiliate_**” means any entity, either directly or indirectly controlling, controlled by or under common control with a Party.  
“**_Coalesce Parties_**” means Coalesce and its Affiliates, independent contractors and service providers, and each of their respective members, directors, officers, employees and agents.  
“**_Coalesce Platform_**” means the software as a service provided by Coalesce under this Agreement including the application which allows Customer to transform data and transfer data from a source repository to a target repository, together with any improvements, updates, bug fixes or upgrades thereto.  
“**_Confidential Information_**” means any information that is proprietary or confidential to the Discloser or that the Discloser is obligated to keep confidential (e.g., pursuant to a contractual or other obligation owing to a third party). Confidential Information may be of a technical, business or other nature (including, but not limited to, information which relates to the Discloser’s technology, research, development, products, services, pricing of products and services, employees, contractors, marketing plans, finances, contracts, legal affairs, or business affairs).  
“**_Contact_**” means any actual or prospective clients of Customer who interact with any Services deployed by Customer or Coalesce on Customer’s behalf.  
“**_Customer Data_**” means any data or information (a) provided by Customer or any Authorized User to Coalesce collected through the Services, input to the Coalesce Platform or accessed by the Coalesce Platform or otherwise or (b) collected through the interaction of Contacts with any Services. For example, Customer Data includes certain Personal Data such as log-in information for Authorized Users.  
“**_Data Protection Law_**” means any and all privacy, security and data protection laws and regulations that apply to the Processing of Personal Data by Coalesce under the Agreement, which may include the GDPR, as applicable.  
“**_Discloser_**” means a Party that discloses any of its Confidential Information to the other Party.  
“**_Documentation_**” means the online documentation relating to the Services furnished or made available by Coalesce to Customer.  
 “**_Feedback_**” means information and feedback (including questions, comments, suggestions, or the like) regarding the performance, features, functionality and overall Customer experience using the Services.  
“**_GDPR_**” means both  
 (i) the Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data; and  
 (ii) UK GDPR being the UK retained version of GDPR as defined in section 3(10) of the Data Protection Act 2018 supplemented by section 205(4).  
 “**_Personal Data_**”, “**_Controller_**”, “**_Processor_**”,  “**_Data Subject_**”, and “**_Processing_**” shall have the meanings ascribed to them in GDPR.  
“**_Initial Subscription Period_**” means the period identified on the Order as the Initial Subscription Period.  
“**_Intellectual Property Rights_**” means any patent, copyright, trademark, service mark, trade name, trade secret, know-how, moral right or other intellectual property right under the laws of any jurisdiction, whether registered, unregistered, statutory, common law or otherwise (including any rights to sue, recover damages or obtain relief for any past infringement, and any rights under any application, assignment, license, legal opinion or search).  
“**_Professional Services_**” means any services requested by the Customer and provided by Coalesce such as support to assist Customer with implementation, training or follow up knowledge sessions.  
“**_Recipient_**” means a Party that receives any Confidential Information of the other Party.  
“**_Renewal Subscription Period_**” means the period identified on the Order as the Renewal Period.  
“**_Services_**” means (a) Coalesce, and any successor or related web site designated by Coalesce, (b) the Coalesce Platform, (c) the software program designed by Coalesce that replies to Contacts’ queries autonomously, (d) any improvements or modifications in or to the foregoing, and (e) other materials or information developed, discovered, authored or reduced to practice in the performance of the Services.  
“**_Subscription Period_**” means the total subscription period being the Initial Subscription Period and any Renewal Subscription Period.  
 “**_Third Party Services_**” means software or services acquired or licensed by Coalesce from a third party that is included in the Services or otherwise made available to Customer or its Authorized Users.  
“**_Trademarks_**” means any trademarks, service marks, service or trade names, logos, and other designations of a Party and its Affiliates.

---

## Troubleshooting Common SSO Errors

This guide explains common single sign-on (SSO) issues in Coalesce and how to fix them. It covers configuration errors, network restrictions, OAuth integration problems, and user management issues.

## Before You Begin

Before troubleshooting, ensure you have:

- Administrative access to your identity provider.
- Coalesce [admin privileges][].
- Network configuration details from your IT team.

## Common SSO Configuration Issues

### SSO Button Not Appearing

**What You'll See:**

- No SSO button visible on login page.
- Users see only standard login form.

**Why This Happens:**

- Incomplete or incorrect SSO configuration.
- Using wrong login URL.
- Missing required configuration fields.

**Solutions:**

- **Correct the Subdomain Field**

   ```txt
   Incorrect: https://yourorg.app.coalescesoftware.io/
   Correct: yourorg
   ```

- **Use the Correct SSO URL**
  - Access format: `https://mySubdomain.<app_domain>`
  - Example: `https://hello.app.coalescesoftware.io/login`
  - Test in incognito or private browser tab
- **Check that these fields are properly configured**
  - Authority
  - Subdomain
  - Authorization Server
  - OIDC Client ID
  - Server-Side Authorization
- **Clear Browser Cache**
  - Use incognito or private browsing mode
  - Clear browser cache and cookies
  - Ensure correct regional URL (US vs APAC environments)

### SSO Button Appears Greyed Out

**What You'll See:**

- SSO button is visible but disabled or grayed out.

**Solution:**

Access Coalesce through your organization's subdomain URL rather than the generic login page:

```txt
Use: https://yourorg.app.coalescesoftware.io
Not: https://app.coalescesoftware.io
```

### Missing Identity Provider Settings

**Common Issues:**

- Missing [Microsoft Graph API permissions][] in Azure app registrations.
- Incorrect authorization server URLs or redirect URIs.
- Extra spaces or characters in configuration fields.

**Solutions:**

- **Ensure these permissions are granted**
  - Profile
  - Email
  - OpenID
- **Check Redirect URIs**
  - Ensure exact match for example: `https://yoursubdomain.app.coalescesoftware.io/login/callback`, `https://yoursubdomain.app.eu.coalescesoftware.io`
- **Validate Configuration Fields**
  - Remove extra spaces or characters
  - Confirm admin consent is granted for API permissions

## Network and Authentication Problems

### Snowflake Network Policies Blocking OAuth Redirects

**What You'll See:**

- SSO login fails during Snowflake OAuth process.
- [Network policy errors][] in Snowflake logs.

You can either:

- **Associate Network Policy with OAuth Security Integration**

    ```sql
    ALTER SECURITY INTEGRATION <oauth_integration> 
    SET NETWORK_POLICY = <oauth_network_policy>;
    ```

- **Create Integration-Specific Network Policy**

    ```sql
    CREATE SECURITY INTEGRATION oauth_kp_int
    TYPE = oauth
    ENABLED = true
    OAUTH_CLIENT = custom
    OAUTH_CLIENT_TYPE = 'CONFIDENTIAL'
    OAUTH_REDIRECT_URI = 'https://example.com'
    NETWORK_POLICY = mypolicy;
    ```

**Additional Steps:**

- Add Coalesce IP addresses to network policy (region-specific).
- Check if policy is applied at user, account, or integration level.
- Recreate security integration if policies were applied after creation.

### Network Security Tools Interfering

**What You'll See:**

- SSO flow interrupted or fails
- Network security tool blocks OAuth requests

**Solutions:**

- Add Coalesce domains and IP addresses to security tool's allowlist.
- Configure tool to allow OAuth redirect flows for Coalesce domains.
- Test using incognito/private browsing mode.
- Work with your IT team to whitelist OAuth endpoints and callback URLs.
  
### Private Link Configuration Issues

**What You'll See:**

- Authentication fails when using private link.
- SSL/TLS certificate errors.

**Solutions:**

- Verify correct private link URL in SSO configuration.
- Update OAuth security integration redirect URIs for private link endpoints.
- Ensure private link DNS resolution works correctly.
- Check SSO provider configuration matches private link subdomain.
- Test with public endpoint first, then migrate to private link.

## OAuth Token and Integration Issues

### Expired OAuth Refresh Tokens

**What You'll See:**

- Users must re-authenticate frequently.
- "Token expired" errors.

**Solutions:**

- **Check Token Validity Settings**

   ```sql
   -- Check current setting in Snowflake (default is 90 days)
   SHOW PARAMETERS LIKE 'OAUTH_REFRESH_TOKEN_VALIDITY';
   ```

- **Implement Retry Mechanisms**
  - Add retry logic in job schedulers
  - Consider switching to key pair authentication for environments
- **Re-authenticate Users**
  - Disconnect and reconnect OAuth credentials in Coalesce
  - Guide users through fresh authentication flow

### Invalid OAuth Security Integrations

**What You'll See:**

- OAuth integration errors in Snowflake.
- Authentication failures using the integration.

**Solutions:**

- **Verify Integration Configuration**

   ```sql
   DESCRIBE SECURITY INTEGRATION <integration_name>;
   ```

- **Recreate Integration**
  - Create new OAuth security integration with correct parameters.
  - Verify client ID and secret match between Snowflake and Coalesce.
  - Ensure redirect URI matches Coalesce callback URL exactly.
  - Confirm integration is enabled.

## User Management Problems

### Multiple Accounts for Same Person

**What You'll See:**

- User has both SSO and non-SSO accounts.
- Permission conflicts between accounts.

**Recommended Approach:**

- Disable non-SSO account for daily operations.
- Transfer project memberships and permissions to SSO account.
- Document account usage purposes.

### Users Created Outside SSO Protocols

**What You'll See:**

- Users bypass SSO during account creation.
- Inconsistent user attributes.

**Solutions:**

- Verify SSO configuration before creating users.
- Ensure users access correct SSO-enabled subdomain.
- Check identity provider configuration for user attributes.
- Delete incorrectly created users and redirect through proper SSO flow.

## Multi-Factor Authentication and SSO

Sign-in problems that mention MFA can come from Coalesce, your identity provider, or your warehouse. Separating those paths saves time.

### Coalesce MFA Versus Identity Provider MFA

Coalesce MFA is configured under **User Settings > Account and Security** and applies only when someone logs in with a username and password in the Coalesce App. Full steps are in [Multi-factor Authentication][].

If you use SSO, your identity provider usually prompts you for MFA, not the Coalesce MFA toggle. If MFA prompts fail when you use SSO, review policies and enrollment in Okta, Microsoft Entra ID, or your other provider with your IdP administrator.

If Snowflake requires MFA for warehouse access, use OAuth with Coalesce as described in [Multi-factor Authentication][].

### Access Tokens After MFA Changes

When MFA is turned on or off for an account, existing access tokens stop working. Issue a new token using [Deploy Page][].

## SCIM and Catalog Provisioning

The SCIM guides under [User and Team Provisioning][] sync Catalog users and teams. They don't create Coalesce Transform logins ahead of time.

Coalesce Transform still relies on just-in-time creation when someone signs in with SSO for the first time, as described in [Single Sign-On][]. If every other SSO check passes but Catalog is missing people or teams, use [SCIM Provisioning Troubleshooting][] together with [SCIM Setup for Microsoft Entra ID][] or [SCIM Setup for Okta][].

## General Troubleshooting Steps

### Initial Diagnostics

- **Browser Testing**
  - Clear browser cache and cookies for both Coalesce and identity provider.
  - Test in incognito/private browser window.
  - Try different browsers.
- **Network Verification**
  - Check with your IT team about network policies blocking OAuth flows.
  - Verify the account has correct permissions in the identity provider and Coalesce.
  - Test from different network locations if possible.
- **Configuration Validation**
  - Verify all configuration fields are correctly entered.
  - Check for extra spaces or special characters.
  - Confirm redirect URIs match exactly.

### Best Practices

- **Account Management**
  - Maintain at least one non-SSO admin account as backup.
  - Use service accounts for API integrations.
  - Document account purposes and usage.
- **Regional Considerations**
  - Use correct regional IP addresses based on Coalesce instance location.
- **Testing Protocol**
  - Always test SSO configuration in staging environment first.
  - Use incognito/private browsing for testing.
  - Test with multiple user accounts and roles.

## When To Contact Support

- Configuration appears correct but SSO still fails.
- Deleted SSO users need manual re-enablement.
- Network policy changes don't resolve OAuth issues.
- Multiple troubleshooting attempts haven't resolved the problem.
- MFA works in the IdP but Coalesce access tokens broke after an MFA toggle.
- Catalog SCIM sync fails after you verified URLs and assignments with your provider.

**Information to Include:**

- Specific error messages
- Organization details and subdomain
- Identity provider type and configuration
- Network environment details
- Steps already attempted

:::info[Contact support]

Contact [Coalesce support][].

:::

## Quick Reference Checklist

**Before Opening Support Ticket:**

- [ ] Verified subdomain configuration is correct.
- [ ] Tested with incognito/private browser.
- [ ] Checked all [required SSO configuration][] fields.
- [ ] Confirmed [network policies][] allow Coalesce IP addresses.
- [ ] Validated identity provider permissions and settings.
- [ ] Confirmed whether the issue is Transform SSO login, Coalesce password MFA, IdP MFA, or Catalog SCIM.
- [ ] Tested with correct regional URL.
- [ ] Attempted browser cache clearing.
- [ ] Verified redirect URIs match exactly.

---

## What's Next?

- [Single Sign-On][]
- [Multi-factor Authentication][]
- [User and Team Provisioning][]
- [SCIM Provisioning Troubleshooting][]
- [Network Requirements][]
- [Role-Based Access Control][]
- [Manage Users][]
- If you can access the SSO button but see a blank or frozen screen after login, see [Troubleshooting Browser Login Issues][].

---

[required SSO configuration]: /docs/organization-and-accounts/organization-management/single-sign-on
[network policies]: /docs/setup-your-project/setup-requirements/network-requirements
[admin privileges]: /docs/organization-and-accounts/organization-management/role-based-access-control
[Microsoft Graph API permissions]: /docs/organization-and-accounts/organization-management/single-sign-on/microsoft-entra-id#microsoft-entra-id-permissions
[Network policy errors]: https://docs.snowflake.com/en/user-guide/network-policies#network-policy-precedence
[Single Sign-On]: /docs/organization-and-accounts/organization-management/single-sign-on
[Network Requirements]: /docs/setup-your-project/setup-requirements/network-requirements
[Coalesce support]: /docs/get-started/support-information
[Role-Based Access Control]: /docs/organization-and-accounts/organization-management/role-based-access-control
[Manage Users]: /docs/organization-and-accounts/organization-management/users
[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Multi-factor Authentication]: /docs/organization-and-accounts/account-management/multi-factor-authentication
[Deploy Page]: /docs/api/authentication
[User and Team Provisioning]: /docs/catalog/administration/user-and-team-provisioning
[SCIM Provisioning Troubleshooting]: /docs/catalog/administration/user-and-team-provisioning/scim-provisioning-troubleshooting
[SCIM Setup for Okta]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-okta
[SCIM Setup for Microsoft Entra ID]: /docs/catalog/administration/user-and-team-provisioning/scim-setup-for-microsoft-entra-id

---

## Exporting Data

You can export some of the data in Coalesce using the following methods.

## Individual Run Screen

Right click on a row in the table to see the export options. This will export the entire run.

## Data Preview Results

In the data preview tab, after a Node has run, it will export the 100 rows returned.

## Copy Unload Node

You can use the [Copy Unload Node][] to external locations such as AWS and others.

[Copy Unload Node]: /docs/marketplace/package/coalesce_snowflake_external-data-package#copy-unload-node-configuration

---

## How To Find Your Snowflake Account Information



---

## Troubleshooting Frequent Authentication Requests

## Issue

The Coalesce App is asking you to re-authenticate multiple times a day.

## Possible Solutions

- Use OAuth authentication.
- Enable third-party cookies in your browser.
- Clear cache for Coalesce and Snowflake.
- If using local password storage, save your password in your browser.
- Change authentication type from Browser to Cloud Storage.

---

## What's Next?

- For other login-related symptoms such as blank screens or spinning loaders, see [Troubleshooting Browser Login Issues][].

---

[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues

---

## General FAQ

<!-- vale Coalesce.FirstPerson = NO -->

## Does Coalesce Have a Dark Mode?

We currently don't offer a dark mode.

## Can I Run Multiple SQL Statements in Pre-SQL and Post-SQL?

They need to be separated by a stage name or have a begin and end.

```sql
--Execute two stages
SELECT * FROM 

{{stage('SOME STAGE INFO')}}

INSERT INTO 
```

```sql
-- This will make it one stage
BEGIN
SELECT * FROM 
INSERT INTO 
END
```

## Can I Filter By Schema Name?

You can't filter by schema, but you can filter by Storage Location. Review [Selector Queries][] for more.

## My Nodes or Columns Have a Prefix or Suffix

- Check that there are no [transformations][] on the column.
- Check to see if the source data includes the prefix or suffix.
- Try updating a single Node or Column at a time.

## Can I Stop a Running Node?

No, once Run has been clicked, it will run until it's finished. If you have started a Job, you can cancel a current Job using the Coalesce App and API.

## Can I Add Other Files to My Coalesce Git Repo?

No, it’s not recommended. Coalesce assumes a specific project and directory structure. Adding other files that are unrelated to Coalesce can lead to:

- **Sync issues** during Project updates or Workspace actions
- **Git errors** when pushing or merging changes
- **Parsing errors** that prevent Coalesce from reading your files correctly

To avoid these problems, only include files required by your Coalesce Project.

[Selector Queries]: /docs/reference/selector/
[transformations]: /docs/build-your-pipeline/transforms/

---

## Managing Case Sensitivity in Snowflake and Coalesce

## Issue

- **Environment Setup:** Snowflake environment configured to ignore case sensitivity.
- **Problem:** Coalesce adds quotation marks around column names if they are not in uppercase, which enforces case sensitivity in Snowflake.

## Solution

1. Add the `QUOTED_IDENTIFIERS_IGNORE_CASE` to your [deployment parameters][]. Review the [Snowflake Parameters documentation][].

    

2. To each node, add `ALTER SESSION SET QUOTED_IDENTIFIERS_IGNORE_CASE = {{parameters.Quoted_identifier_ignore_case}}`in the Pre-SQL for each node that needs it.

    

[deployment parameters]: /docs/deploy-and-refresh/parameters/rtp-deploy
[Snowflake Parameters documentation]: https://docs.snowflake.com/en/sql-reference/parameters#quoted-identifiers-ignore-case

---

## What Data Types Does Coalesce Support?

Coalesce supports all data types in our data platforms. You can expand data types by using our [Packages][] or creating a [custom Node][].

[Packages]: /docs/marketplace
[custom Node]:/docs/build-your-pipeline/user-defined-nodes

---

## Trial Accounts

<!--vale Coalesce.FirstPerson = NO -->

## How Do I Remove the Trial Banner?

If you're a signed customer and still see the **Trial** banner, [contact Coalesce Support](mailto:support@coalesce.io) to have it removed.

## What’s My Account ID or Org ID?

Your account ID is also referred to as your **Org ID**.

To find it:

1. Go to **User Settings**.
2. Open **Support Information**.
3. Your Org ID is listed under **Organization**.

## How Do I Delete My Trial Account?

You’ll need to [contact Coalesce Support](mailto:support@coalesce.io).

## How Do I Check How Much Time Is Left in My Trial?

Check your **Coalesce trial welcome email**. It includes the start and end dates of your trial.

## Will I Receive Emails During My Trial?

Yes. Coalesce sends marketing emails throughout the trial period to help guide you through the product.

---

## Pipeline Documentation

Documentation is an essential component of building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. Coalesce automatically produces and updates documentation as developers work.

## Accessing Generated Documentation

Generated documentation for a **Workspace or Environment** can be accessed in three different ways.

### Workspace and Environments

Both **Workspaces** and **Environments** - From the top of the user interface, through the Docs link.

### Workspaces

Only **Workspaces** - From the **Projects Dashboard**, through the Docs icon.

### Environments

Only **Environments** - From the **Deploy Dashboard** click on the ellipses and then **View Docs**.

## Interface Overview

The new documentation interface consists of three sections.

### 1. Search

Users can search for Node names, physical types, databases, schemas, and storage locations. Node names can be searched for using any portion of the Node's name, while all other objects are searched using a prefix search only. For example, if the storage location is `PROD_RAW`, searching for `RAW` will not yield any results. On the other hand, if a Node is named `STG_CUSTOMER`, searching for either `STG` or `CUST` will be effective.

### 2. Filters

The **Filters** dropdowns allow the user to limit the Nodes currently visible in the **Results List**. For example, if one wanted to see only Nodes of type Stage, then they would select **Node type** > Stage, and the **Results List** will be updated in real time.

### 3. Results List

This section will have the results, which come from a combination of search terms and filters used.

## Sharing Results

Search terms and filters automatically adjust the URL, which makes results easy to share, as other users can use the same URL to access the same results.

## Exporting Documentation

The Coalesce documentation can't be exported. The [Command Line Interface][] and [API][] can be used to programmatically gather information about your Coalesce organization, such as a list of your Environments with their details, Node contents, etc.

## Documentation Walkthrough

You’ll learn more about Coalesces' documentation feature and how to navigate each page.

### Nodes Landing Page

The landing page helps you filter and focus on the specific Nodes you wish to view. In this example, the filter for Dynamic Table Work was added and returns three Nodes of that type. Clicking on a Node name will take you to the documentation for that Node.

### Node Information

This section provides navigation and column documentation for the Node. Expand each section by clicking the arrows, or use the breadcrumbs to navigate.

The header shows:

- **Node name** - `DT_WRK_DT_CUSTOMER_NATION_REGION_DT_ORDERS_LINEITEM`
- **Node Type** -  `Dynamic Table Work`.
- **Database** - `TATIANA_DOCS_DYNAMIC_TABLES`
- **Table** - `DEV`
- **Storage Location** - `WORK`

Information is organized into the following sections.

#### Overview

General information about a Node.

#### Columns

Shows all the columns in a Node.

#### Column View

Click **View** next to a column to display its information, including the column name and its parent.

The column lineage can be navigated by clicking on the ellipses and clicking **Go to column documentation**. It will then take you to the documentation for that column in the preceding Node. You can continue to do that for each column in the lineage. Return to the Node Overview by clicking the Node name.

#### Lineage

This shows the Node Lineage. Use the controls to move around the Node.

#### Create Scripts

Any scripts that ran as part of the Create function. If there is more than one, it will be another tab.

#### Run Scripts

Any scripts that ran as part of the Run function. If there is more than one, it will be another tab.

#### Configuration

Any configuration options available for the Node. This can vary depending on the Node.

[Command Line Interface]: /docs/coa/
[API]: /docs/api/

---

## Coalesce Best Practices

This page summarizes best practices for working with Coalesce Transform. For initial setup, see the [Transform Onboarding Guide][]. For step-by-step configuration, see [Setup and Configuration][].

## Version Control

### Git Commits

- Make frequent commits with **meaningful commit descriptions**.
- Keep each commit focused on a **single unit of work**.
- Communicate with your team before committing to ensure the Workspace is in a **commit-ready state**.
- Designate a single developer to commit to the **main branch** to avoid conflicts.
- Avoid making breaking commits directly to the main branch.

:::info[Having Trouble Making Commits]

If you cannot commit your changes in a Workspace to Git, contact Coalesce support. Overwriting live metadata with a previous commit can cause you to lose recent uncommitted development.

:::

### Git Branches

- Follow a **branching strategy** (for example, feature branches for development).
- Use branches to keep in-progress work separate from the main branch.
- Deploy to target Environments only from the **main branch**.
- Only one person should work on a given branch at a time.

### Git Pitfalls to Avoid

- Never develop directly in your main development Workspace.
- Never merge into a Workspace that has uncommitted changes.
- Only open one instance of the Git modal per Workspace at a time.
- Do not change Git repository settings while there are uncommitted changes in any Workspace.

See [Git Integration][] and [Version Control Best Practices][] for details.

## Workspaces

### Workspace Strategies

Choose a strategy that fits your team size and workflow:

- **One Workspace per branch:** Create a new Workspace for each feature branch; delete when merged. Keeps work isolated.
- **One Workspace per user:** Each developer has their own Workspace and manages branches from it. Good for smaller teams.
- **One Workspace per feature:** Separate Workspaces for distinct features. Reduces conflicts.

### Workspace Guidelines

- Workspaces should map to **different schemas**, except for source Nodes.
- When feature development in a Workspace is complete, merge into the main branch.
- Check out the main branch in the Main Workspace when preparing for deployment.
- Deploy to higher Environments (Production, QA, Testing) only from the main branch in the Main Workspace.

See [Workspaces][] for full details.

## Environment Management

- Each Environment should map to a **different database and schema**.
- You cannot have a Workspace and Environment share the same schemas unless the Workspace is used read-only.
- Create Environments for DEV, QA, and Production matching your Storage Mappings.
- As a feature flows from development to Test, QA, and Production, Storage Mappings should point to the correct locations.
- Each Environment needs its own [authentication][] to your data platform.

See [Environments][] and [Create Your Environments][] for setup.

## Deployment

- Commit work to the branch intended for deployment before deploying.
- Configure target databases and schemas in your data platform before deployment.
- Configure [Parameters][] for environment-specific values.
- Test in lower Environments (DEV, QA) before deploying to Production.
- Decide whether to deploy using the [CLI][] or [Coalesce App][] based on your workflow.

See [Deployment Overview][] for details.

## Refresh and Jobs

- Ensure each Environment has [Storage Mappings][] and [authentication][] configured before refreshing.
- Create [Jobs][] for the Nodes you want to run.
- Deploy with your Jobs before refreshing (Jobs run only on deployed Nodes).
- Configure [Parameters][] for runtime values.
- Use the [Coalesce Scheduler][], [CLI][], [API][], or external tools for scheduling.

See [Refresh Your Pipeline][] and [Scheduling Jobs][] for details.

## Platform-Specific Notes

### Snowflake

- Set source tables to **READ** (data read by Coalesce).
- Set targets to **READ/WRITE** (where Coalesce writes transformed data).

### Databricks

- Your Workspace must use [Unity Catalog][].

## Pipeline Design

Coalesce recommends using pipelines instead of Common Table Expressions (CTEs) for complex data processing in an analytical environment. For a comparison and rationale, see [CTEs vs Pipelines for Complex Data Processing][].

CTEs are supported when needed. See the Coalesce documentation for how to use them.

## What's Next?

- [Transform Onboarding Guide][] for initial setup through deployment and team rollout
- [DataOps and Git Best Practices][] for deeper version control guidance

---

[Transform Onboarding Guide]: /docs/get-started/transform-onboarding-guide
[Setup and Configuration]: /docs/setup-your-project
[Git Integration]: /docs/git-integration/
[Version Control Best Practices]: /docs/git-integration/version-control-best-practices
[Workspaces]: /docs/get-started/coalesce-fundamentals/workspaces
[Environments]: /docs/get-started/coalesce-fundamentals/environments
[Create Your Environments]: /docs/setup-your-project/create-your-environments
[authentication]: /docs/category/connection-guides
[Parameters]: /docs/deploy-and-refresh/parameters/
[CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli/
[Coalesce App]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment/
[Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[Jobs]: /docs/deploy-and-refresh/refresh/
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[API]: /docs/api/runs/startrun/
[Refresh Your Pipeline]: /docs/deploy-and-refresh/refresh/
[Scheduling Jobs]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Deployment Overview]: /docs/deploy-and-refresh/deploy/
[Unity Catalog]: https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspaces
[CTEs vs Pipelines for Complex Data Processing]: https://coalesce.io/product-technology/ctes-vs-pipelines-for-complex-data-processing-why-coalesce-offers-the-better-approach/
[DataOps and Git Best Practices]: /docs/guides/dataops-and-git-best-practices

---

## When To Use Copy Objects, Duplicate, or Create New Project

|                                 | What Happens                                                                                                                                                       | Objects Copied                                                                                    | Objects Not Copied                                | When To Use                                                                                                                                         |
| :------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------ | :------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Copy Objects from Workspace** | Makes a copy of objects in a workspace to another, existing workspace. Information. The objects don't have to be committed. Uses the same Git repo as the Project. | Node, Packages, Jobs, Subgraphs, Macros                                                           | Connection Settings and Storage Location and Storage Mappings | When you need to move information between workspaces, but don't need the Storage Location or Storage mappings.                                                 |
| **Duplicate Workspace**         | Duplicates an existing Workspace into a new Workspace. Uses the same Git repo as the Project.                                                                      | Anything that is part of the commit and branch. This includes Nodes, Storage Location, Storage Mappings, and the connection account URL.| Connection Settings. You'll need to re-authenticate.| When you want to create a workspace quickly that is based on a certain commit and has the same Storage Locations and Storage Mappings as the original workspace. |
| **Create Workspace**            | Creates a new workspace. Uses the same Git repo as the Project.                                                                                                    | Storage Locations are copied over. You can add more or edit them after the Workspace is created.    | Connection Settings and Storage Mappings | When you want to start a new Workspace and don't need the Storage Mappings or connection settings.                                        |
| **Create Project**              | Creates a brand new project. You can either use an existing Git URL or add a brand new Git URL.                                                                    | None. Even if using an existing Git URL, no information is copied.                                | N/A                                               | Want to start fresh or working on a new business initiative.                                                                                      |

## Copy From Workspace

1. Select **Copy Objects from Workspace** for the workspace you want to copy.

    

2. Select the objects you want to copy. You'll get alert if the object already exists based on the ID. You can decide to copy over an object if it already exists, the information will be overwritten.

## Duplicate Settings

1. Click **Duplicate Workspace** next the workspace you want to duplicate.

    

2. Follow the set up wizard to complete duplicating the Workspace.

## Create a New Workspace

1. Click **Create Workspace**.

    

2. Follow the set up wizard to complete creating a new Workspace.

## Create a New Project

1. Make sure you are on the Projects page. If you are on the [Build page][] , select the **back arrow**.
2. Click the **plus sign** (+) next to Projects.
3. Fill out the Project Details.
4. Enter your Git repo URL.
    1. Coalesce supports [many providers][] .
5. Then select a Git account to use. It should be able to view and make requests to the Git repo in the previous step.
6. If you don't a Git repo configured, then select **Add New Account**.
    1. Enter an account nickname. This will displayed in the interface.
    2. Enter the Git username and [token][].
    3. Enter the **Author Name**, which identifies the committer.
    4. Enter the **Author Email**, which identifies the committer email.
    5. Click **Add**.
    6. Select the Git account you just created in the drop down, then click **Test Account**.
7. Once successful, click **Finish**.
8. Now that you've added a Project, you need to add a [**Workspace**][] .

[Build page]: /docs/build-your-pipeline/the-build-interface/
[many providers]: /docs/setup-your-project/setup-version-control
[token]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
[**Workspace**]: /docs/get-started/coalesce-fundamentals/workspaces

---

## Directed Acyclic Graph (DAG)

A Directed Acyclic Graph, or DAG, is a concept from mathematics and computer science that describes a specific type of graph.

- **Directed**: This means that the connections between the points in the graph (known as nodes) have a direction. In a directed graph, each connection (edge) points from one node to another, indicating a one-way relationship.
- **Acyclic**: This term means that there are no cycles in the graph. A cycle occurs when you can start at one node and follow a path of edges that eventually loops back to the starting node. In a DAG, loops aren't allowed.
- **Graph**: In this context, a graph is a collection of nodes (which can represent various objects like tasks, events, or states) and edges (which represent the relationships or connections between these nodes).

## Key Characteristics of a DAG

- **Directionality**: Each edge has a direction, showing the relationship from one node to another.
- **No Cycles**: Once you have left a node, you can't go back to it again. This prevents the chance of circling back to the node through the edges.
- **Topological Ordering**: Since DAGs have directed edges and no cycles, you can list the nodes in a linear order. This order respects the direction of the edges, meaning for every directed edge from node A to node B, A appears before B in order.

### Applications of DAGs

DAGs are incredibly useful in scenarios where you need to represent tasks that must be done in a specific order.

Here are a few areas where DAGs are commonly used:

- **Project Scheduling**: In project management, tasks that depend on the completion of other tasks can be represented using a DAG. This helps in planning the order of operations.
- **Data Processing**: Many data processing workflows involve steps that depend on the outputs of previous steps. DAGs can help in mapping out these dependencies clearly.
- **Blockchain Technology**: Some newer cryptocurrencies use DAGs instead of traditional blockchain structures to record transactions. This can offer improvements in scalability and speed.

### Example of a DAG

When working in Coalesce, you'll use a DAG to build your pipeline. In the graph you have nodes and the connections between them. Each connector is one way, so the first nodes will execute, followed by the next nodes. `CUSTOMER > STG_CUSTOMER > DIM_CUSTOMER`.

You can change some of the node execution through [Ref Functions][].

---

## What's Next?

- [**Graphs and Subgraphs**][] - Learn how to view parts of your node graph in isolation and organize your work into logical sections.
- [**Nodes and Node Types**][] - Discover the different types of nodes available in Coalesce and how they function as building blocks in your data pipeline.
- [**Organizing Your DAG**][] - Explore best practices for structuring your directed acyclic graph for better maintainability and performance.
- [Using Subgraphs to Build Your Pipeline][] - Organize the Build view with Subgraphs and connect them to Jobs.
- [**The Build Interface**][] - See how to visualize and work with your DAG in the Coalesce application.

---

[Ref Functions]: /docs/reference/ref-functions
[**Graphs and Subgraphs**]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs
[**Nodes and Node Types**]: /docs/get-started/coalesce-fundamentals/nodes
[**Organizing Your DAG**]: /docs/build-your-pipeline/organizing-your-dag
[**The Build Interface**]: /docs/build-your-pipeline/the-build-interface/
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Environments

Environments are used for deploying your data pipeline to your data platform. They manage efficient deployment strategies and allow you to configure both your Workspace and Environment. This lets you complete your development in a Workspace and then deploy to any Environment you want, such as production or QA, with each having different Storage Locations, Storage Mappings, Parameters, and other options.

## Key Concepts

- **Workspace**: Where you develop and test your data pipeline.
- **Environment**: Where you deploy or refresh your production data pipeline.
- **Development vs Deployment**: No development happens directly in an Environment - all changes come through defined Git branches and commits.

## Why Environments Are Important

Environments serve several purposes in the data pipeline lifecycle:

- **Separation of Development and Production**: Environments let you to keep development work isolated from production systems, preventing accidental impacts on live data.
- **Deployment Control**: They enable controlled deployment of changes through a structured process, ensuring that only tested and approved changes reach production.
- **Environment-Specific Configuration**: Each environment can have different configurations (credentials, parameters, storage locations) appropriate for its purpose.
- **Organized Testing Progression**: Environments are a structured was of testing progression from development to QA to production.

## Common Environment Types

- **Development**: For initial testing after development in Workspaces,
- **QA/Test**: For quality assurance testing before production.
- **Staging**: For final verification in a production-like setting.
- **Production**: The live environment where transformations run on actual business data.

## Important Notes

- Objects are created in a steady state through Git branches and commits.
- Data loading occurs on demand, with schedules/triggers, or an automated process such as CI/CD pipeline.
- Column changes use `ALTER` instead of `DROP/CREATE`.
- Deleted objects are automatically cleaned up (DROPPED) in your data platform.
- No development happens in Environments, work is done in Workspaces. Each Workspace can have multiple Environments.

## Environment Settings Overview

To edit or create an Environment, go to **Build Settings > Environments**.

1. **Settings** - Includes general account information such as the Environment name and the connection URL. Take note of the Environment ID. You'll use this when making API requests to see information about the deployment. You can also delete your Environment here.
2. **User Credentials** - Configure how you connect to your Data Platform. For Snowflake, key pair authentication is recommended. See [Setting Up a Snowflake Service Account for Coalesce][].
3. **Storage Mappings** - To deploy or refresh your environment, you need to have your Storage Mappings setup. Storage Mappings are represent the physical locations in your Data Platform. This is where your data pipeline will deploy. Learn more about Storage Mappings.
4. **Parameters** - Parameters are defined as a JSON blob set by the user that can be accessed in the metadata during template rendering. Learn more about [Parameters][].
5. **OAuth Settings** - If you're using OAuth to connect to your data platform, you'll need to configure OAuth on your data platform.

---

## What's Next?

- [**Deployment Overview**][] - Learn how to deploy your data transformations to the environments you've configured.
- [**Refresh Your Pipeline**][]- Understand how to execute transformations and update data in your deployed environments.
- [**Parameters**][] - Discover how to make your pipeline flexible across different environments using parameters.
- [**Step 9: Create Your Environments**][] - Go through the steps to creating a new environment.

---

[**Deployment Overview**]: /docs/deploy-and-refresh/
[**Refresh Your Pipeline**]: /docs/deploy-and-refresh/refresh/
[**Parameters**]: /docs/deploy-and-refresh/parameters/
[Parameters]: /docs/deploy-and-refresh/parameters/
[**Step 9: Create Your Environments**]: /docs/setup-your-project/create-your-environments
[Setting Up a Snowflake Service Account for Coalesce]: /docs/setup-your-project/connection-guides/snowflake/snowflake-service-accounts

---

## Graphs and Subgraphs

## Graphs

Your Node Graph or DAG (directed acyclic graph) is all the Nodes in your data pipeline. The Node graph will show each Node and their relationship to each other

## Subgraphs

Subgraphs are a way to view parts of your node graph in isolation. This allows you to break down work in progress into logically separate parts. They are defined by the user manually. Adding a Node to a subgraph creates a reference to the original Node in the Subgraph. A node can live in multiple Subgraphs.

---

## What's Next?

- [**The Build Interface**][] - Learn how to use the interface where you'll spend most of your time creating nodes and transforming data.
- [**Organizing Your DAG**][] - Learn best practices on using Graphs and Subgraphs.
- [Using Subgraphs to Build Your Pipeline][] - Name Subgraphs, edit membership, choose when to add another Subgraph, and connect Subgraphs to Jobs.

---

[**Organizing Your DAG**]: /docs/build-your-pipeline/organizing-your-dag
[**The Build Interface**]: /docs/build-your-pipeline/the-build-interface/
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Coalesce Fundamentals

These guides introduce fundamental concepts you need to understand in order to use Coalesce effectively.

<DocCardList />

---

## Nodes and Node Types

A Node in Coalesce is the visual representation of database objects (tables and views) that form the building blocks of your data pipeline. Nodes are classified by Node Type.

Each Node is comprised of Node Properties and Options that can be configured according to your requirements, as well as elements where the contents of the Node are defined. For example, the Mapping View where column and transformation specifications are defined and the Join Editor where relationships between sourcing Nodes are specified.

Nodes are classified into Node Types. Each Node Type has a specification or definition, a create template, and a run template which will drive the options available to Nodes of this type as well as how the SQL is generated for and how each Node of that type behaves during deployment (DDL; tied to the create template) and refresh (DML; tied to the run template).

## Node Types

The **Node Types** section shows all the currently available node types that can be used.

:::info[Git and Nodes]

All node types and nodes can be checked in to Git and deployed to the environment.

:::

Each node type is defined by a node specification YAML file, a **Create Template** and a **Run Template**.

The node specification consists of the following:

- Attributes of the node type (such as name prefix, graph color)
- UI elements that are used to configure the attributes of each individual instance of that node type

The **Create Template** for a node is executed during **Deploy**, while the **Run Template** is executed during **Run/Refresh**.

Coalesce comes with default Node Types which are not editable. However, they can be duplicated and edited, or new ones can be made from scratch.

To access Node Types, in your **Workspace** go to **Build Settings > Node Types**.

|Field|Description|
|---|---|
|Status|Green indicates all aspects of that node type are valid  Red  indicates there is invalid info within the node type that needs to be revised.|
|Name|This is the type of node. Coalesce provides the Dimension, Fact, Stage, View, and Persistent Stage node types, but a user can also create a new type.|
|Default Storage Location|Default storage location that will be chosen automatically for new nodes of this type. Each workspace will have a Global Default for all nodes, which can be overridden here.|
|Actions|All node types can be duplicated for further customization. The name of this duplicate will default to `Copy of <Node Type>` but can be modified in the Node Type Editor. Additionally, custom node types can be edited. Note: Default node types and node types in use in the application or node type editor cannot be deleted.|
|Enabled/Disabled|This feature allows a user to add any Enabled node types in the [Node Graph][].|

## Default Node Types

Node types control the actual creation and data manipulation of all database objects within your data pipeline. Node types are defined by Jinja/SQL templates and a YAML **Node definition** which has numerous options for customization in the **Node Type Editor**.

### Source Nodes

Source nodes represent the tables, external tables, or views where raw data is queried to be processed by subsequent nodes. They are the source tables that already exist in your data platform. They are not manually editable other than to change their location or add a description.

### Stage Nodes

Stage Nodes are intermediate Nodes in your pipeline where you prepare data by applying business logic. They're designed to hold the current batch of data, making them ideal for temporary data staging.

#### Default Behavior

By default, Stage Nodes truncate data before every run and reload all data from the source. This "truncate and reload" strategy ensures your stage table is always a complete, fresh copy of the source data.

When the **Truncate Before** option is enabled (the default setting):

- The stage table is emptied before each run.
- All data from the source is reloaded.
- The table reflects an exact copy of the source.

#### Alternative Behavior

You can toggle off **Truncate Before** in the Stage Node configuration. This appends new records instead of replacing all data. However, this approach can lead to duplicate records if the same data is processed multiple times.

#### Incremental Processing

If you need to process only new or changed records, consider these approaches:

- Use [incremental load Nodes][] instead of basic Stage Nodes.
- Implement high watermark logic using a timestamp column.
- Use Persistent Stage Nodes that support merge operations.

The stage layer is typically where incremental load logic is applied in data pipelines. However, basic Stage Nodes default to full refresh for simplicity and data consistency.

#### Materialization Options

Stage Nodes can be materialized as either a table or a view. Tables are the default option because they make troubleshooting easier.

Divide complex business logic by splitting it into multiple Stage Nodes.

### Persistent Stage Nodes

Persistent Stage Nodes are similar to Stage Nodes with a few key differences:

- They don't truncate data by default.
- Data persists across multiple runs.
- They support tracking change history of columns based on a business key.

:::info

Use Persistent Stage nodes when you want to keep the raw data for extended periods rather than just for the current batch load.

:::

### Fact Nodes

Fact nodes represent Coalesce's implementation of a Kimball Fact Table. A fact table consists of the measurements, metrics, or facts of a business process and is typically located at the center of a star schema or schema surrounded by dimension tables.

A fact table can be configured to use a business key to MERGE data at the appropriate grain of the data. Alternatively, if no business key is selected, the Fact Node will use an INSERT statement to populate the data.

You can read more about [Fact Tables][].

### Dimension Nodes

Dimension nodes are generally descriptive in nature and describe Facts (see Fact Nodes above). For example time and location. They can also be used to store history.

A business key at the appropriate grain of the data is required for Dimension Nodes.

#### Supported Dimension Nodes

Coalesce currently supports Type 1 and Type 2 slowly changing dimensions.

- To create a Type 1 dimension (current state of the data) - **do not** choose a change tracking column.  
- To create a Type 2 dimension (current and historical data) - **do choose** a change tracking column.

You can read more about Dimension Tables [Core Concepts][].

### View Nodes

View nodes are used for creating a generic SQL VIEW. Many other node types can also be materialized as SQL VIEWs. The purpose of having this node type is for improved readability when looking at a DAG. Learn more in [Understanding View Nodes][].

They are disabled by default but can be toggled in your **Build Settings** > **Node Types**.

## User Defined Nodes

User Defined Nodes or UDNs are nodes that can be configured for your needs. Learn more in [User Defined Nodes][].

## Package Nodes

After installing a package, you'll see the package listed under Node Types. Each package can contain multiple nodes and the type will always be the package name. Learn more about [Packages][].

---

## What's Next

- [**User Defined Nodes**][] - Learn how to create custom node types.
- [**Building Your Pipeline**][] - Get started building your data transformation pipeline.
- [**Marketplace**][] - View all Coalesce Nodes available for your pipeline.

---

[Node Graph]: /docs/build-your-pipeline/the-build-interface#dag-or-graph-view
[Fact Tables]: https://www.kimballgroup.com/2008/11/fact-tables/
[Core Concepts]: https://www.kimballgroup.com/category/dimension-table-core-concepts/
[User Defined Nodes]: /docs/build-your-pipeline/user-defined-nodes/
[**User Defined Nodes**]: /docs/build-your-pipeline/user-defined-nodes/
[Packages]: /docs/marketplace/manage-packages
[**Building Your Pipeline**]: /docs/category/build-your-pipeline
[**Marketplace**]: /docs/marketplace
[incremental load Nodes]: /docs/marketplace
[Understanding View Nodes]: /docs/get-started/coalesce-fundamentals/view-nodes

---

## Projects

In Coalesce, your work is organized into Projects. It's similar to a folder on your computer that helps organize your work. Projects can be organized by purpose, department, or business area.

In the following example of a Coalesce Organization that has 5 **Projects**, Data Foundations, Compliance, Campaign Analysis, Financial Analysis, and Revenue Operations.

Each **Project** in Coalesce is tied to a single Git repository, which allows for easy version control and collaboration. Within a project, you can create one or multiple [**Workspaces**][], each with its own set of code and configurations.

Each Project and Workspace has its own set of deployable [**Environments**][], which can be used to test and deploy code changes to production, development, and QA.

:::info[Git and Projects]

It's a good idea to use one Git repository per project.

:::

## Manage Your Projects

For an in-depth guide on creating Projects, take a look at [Step 2: Create Your Project][].

### Create a Project

1. Go to the Project page. If you are on the [Build page](/docs/build-your-pipeline/the-build-interface/) , click the **back arrow**.
2. Click the **plus sign**(+) next to Projects.
3. Enter the Project name and description. Click Next.
4. Enter you version control repository URL. You can Skip and Create to start using the project, but you won't be able to deploy.
5. Then select an account to use. It should be able to view and make requests to the Git repo in the previous step.
6. If you don't a repo configured, then click **Add New Account**.
7. Once successful, click **Finish**.

### Copying Project Objects

As part of **Projects**, there is copy functionality than can make copying **Nodes**, **Macros**, **Jobs**, **Subgraphs**, and **Storage Locations** (excluding mappings) from one **Workspace** to another.  **Jobs** and **Subgraphs** have their definition copied over, but the **Nodes** themselves will not be copied over. You can access it by clicking on the ellipsis next to an individual Workspace and follow the interface prompts from there. Learn more in [When To Use Copy Objects, Duplicate, or Create New Project][].

### Deleting a Project

To delete an existing **Project**, you can click the ellipsis next to the relevant one and select **Delete Project**. A confirmation window will appear, and once you confirm, the **Project** will be deleted.

---

## What's Next?

- [**Step 2: Create Your Project**][] - Go in depth on creating a new project.

---

[When To Use Copy Objects, Duplicate, or Create New Project]: /docs/get-started/coalesce-fundamentals/copying-workspace-and-project-data
[**Workspaces**]: /docs/get-started/coalesce-fundamentals/workspaces
[**Environments**]: //docs/get-started/coalesce-fundamentals/environmentss
[**Step 2: Create Your Project**]: /docs/setup-your-project/create-your-project
[Step 2: Create Your Project]: /docs/setup-your-project/create-your-project

---

## Storage Locations and Storage Mappings

Storage Locations and Storage Mappings are closely related but serve different purposes in managing and structuring data within Coalesce.

<BestPractices title="Storage Location and Mapping Best Practices">
- A good rule is to create target schemas for DEV, QA, and Production. Then map them in Coalesce.
- All development Workspaces should be ideally mapped to different schemas (except source nodes). You can't have a Workspace AND Environment on the same set of schemas, unless the Workspace is used as read-only (which cannot be enforced).
</BestPractices>

## Storage Location

A **Storage Location** in Coalesce represents a logical destination for the database objects you use as sources and create within the platform. It acts as a container or grouping for these objects, such as views and tables, allowing you to organize and manage them effectively. Each Coalesce node, which represents a specific database object, is mapped to a single Storage Location. However, a single Storage Location can be mapped to multiple Nodes, providing flexibility in organizing your data.

:::info[Storage Location Mappings]

Storage location mapping definitions are committed in Git for Environments, not for Workspaces.

:::

## Storage Mapping

**Storage Mapping** is a physical destination within your database that is defined for a Storage Location in a given Workspace or Environment. It represents the actual database and schema where the Storage Location resides. It allows you to define the exact location where your database objects will be stored within your Data Platform.

By using Storage Locations and Storage Mappings in Coalesce, you can achieve granular control over the organization and storage of your database objects. You can map different Storage Locations to different physical locations, enabling you to tailor the storage of your data based on specific requirements.

For example, in a development workspace, you can map each of your Storage Locations to the same physical location, effectively placing all database objects in a single schema used as a scratch working area.

Meanwhile in production, you can map each Storage Location to a different physical location for each storage location, allowing for the use of roles on specific schemas to control access to objects.

## Example of Storage Location and Storage Mappings

In this example, one of the Storage Locations created is `SILVER` and it's mapped to the database and schema `coalesce_aws_dbx.tatiana_product_performance_analytics`.

## Storage Location Versus Storage Mappings

|     |     |     |
| --- | --- | --- |
| **Storage Location** | A logical destination for the objects you use as sources for and create within Coalesce. For example, views and tables in the form of Nodes. Each Coalesce Node is mapped to a single Storage Location, but a single Storage Location may be mapped to many Nodes. | Logical, and does not reconcile to a specific physical database or schema. Physical mappings are defined in Storage Mappings. Typically scoped or organized by a the database and schema structure in your account, but is environment independent.For example: `SOURCE_SALESFORCE` and `DWH_SALES` vs. `UAT_SOURCE_SALESFORCE` and `UAT_DWH_SALES` |
| **Storage Mapping** | A physical destination, database and schema that is defined for a Storage Location for a given Workspace or Environment. | Physical, environment specific mapping of a logical Storage Location at the database and schema level  Reconciles to a specific physical database or schema. For example, the Storage Location `SOURCE_SALESFORCE` in your UAT Environment has a Storage Mapping of database `UAT_SOURCE` and schema `SALESFORCE`. |

## Add a Storage Location

1. Launch your Workspace.
2. Go to the **Build Settings**, <BuildSettings width="22" height="22"/>.
3. Click on **Storage Locations**.
4. You can create as many Storage Locations as you need. It's a good idea to have a source data storage location and a transformation storage location. A best practice is to separate your Nodes into different storage locations that represent different groupings of datasets and to name them appropriately.

## Create or Edit Storage Mappings

When adding sources in the build interface, only physical locations that are mapped to a Storage Location for that environment or workspace are available to be added. This means that you will need to define a Storage Location for source tables and map that to a physical location for that environment or workspace. Make sure you have configured your Storage Locations and [connected to your data platform][]. Then you can map the physical locations in your Data Platform to the Storage Locations you created in Coalesce.

1. Go to **Build Settings**, <BuildSettings width="22" height="22"/>.
2. Click **Workspace Settings** <CogSVG/>, and go to **Storage Mappings**.
3. Any Storage Locations will be listed here. You can create as may mappings and locations as needed.
4. Now you can map your database to the Storage Location. Choose the database location you want to map to your Storage Location.

## Override Mapping Values

To input database and schema objects manually, instead of choosing them from the dropdown menu, activate the **Override Mapping Values** option. This step may be necessary when mapping to a database or schema for which you lack access permissions.

## Changing Your Database or Schema or Edit Storage Mappings

If you need to update the information from your data provider, here are a few tips.

* Always create a new database or schema.
* After creating the new schema, change the Storage Mappings to match the new schema and deploy the changes.
* The deploy process will recognize the Storage Mappings change and update the Nodes from the old schema to the new one.

:::danger[Existing Database and Schema]

Do not change existing deployed database or schema information. This will cause errors in Coalesce. Always create a new database and then update the Storage Mappings.

:::

[connected to your data platform]: /docs/category/connection-guides

---

## Identity and Surrogate Keys in Coalesce

A **surrogate key** is a system-generated identifier that uniquely represents a row in a table. Unlike business keys such as customer ID or order number, surrogate keys have no business meaning.

Surrogate keys are commonly used to:

- Join Fact and Dimension Nodes.
- Support slowly changing dimensions (SCDs).
- Guarantee uniqueness for each row.
- Improve performance when joining tables.

## Surrogate Keys in Coalesce

- Coalesce automatically generates surrogate keys for supported Node Types.
- Surrogate key behavior is defined at table creation time and is handled through Node Types or templates.

## Identity Columns in Coalesce

Coalesce implements surrogate keys using **identity columns**. An identity column is a numeric column whose values are automatically generated by the data platform when rows are inserted.

Identity columns in Coalesce:

- Are defined in the `CREATE TABLE` statement.
- Generate sequential numeric values.
- Are excluded from insert and merge statements.
- Cannot be populated using transforms or SQL expressions.

Because identity generation occurs at the database level, identity columns must be defined when the table is created.

## Snowflake Identity Implementation

When using Snowflake, Coalesce generates surrogate keys using `NUMBER IDENTITY` columns in the table DDL.

Example pattern generated by Coalesce:

```sql
SURROGATE_KEY NUMBER IDENTITY
```

In Snowflake:

- `IDENTITY` is a column attribute applied to a numeric type.
- Values are generated automatically during inserts.
- The identity column doesn't appear in `INFORMATION_SCHEMA.SEQUENCES` because it's embedded in the table definition.

## Databricks Identity Implementation

When using Databricks, Coalesce generates surrogate keys using the following syntax:

```sql
BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1)
```

In Databricks:

- Identity values are always generated by the engine.
- The column must be defined in the table DDL.
- The surrogate key column is excluded from insert logic.

## BigQuery Identity Implementation

When using BigQuery, Coalesce generates surrogate keys using `GENERATED BY DEFAULT AS IDENTITY` on an integer column in the table DDL.

Example pattern generated by Coalesce:

```sql
surrogate_key INT64 GENERATED BY DEFAULT AS IDENTITY
```

In BigQuery:

- Identity values are generated by the engine on insert.
- The column must be defined in the table DDL.
- The surrogate key column is excluded from insert logic.

:::info[Creating Identity Columns]

- Use a Node Type with an existing identity column.
- Create a custom Node using `isSurrogateKey: true` in the Node Template.

:::

## Node Types That Use Surrogate Keys

Different Coalesce Node Types use surrogate keys in different ways.

### Dimension Nodes

Dimension Nodes include surrogate keys by default. These surrogate keys are required to support SCD Type 2 behavior and to maintain consistent joins with downstream Fact Nodes.

### Fact Nodes

Fact Nodes can include surrogate keys when required. In these cases, the surrogate key typically acts as the table's primary key, while foreign keys reference Dimension surrogate keys.

### Persistent Stage Nodes

Persistent Stage Nodes can include surrogate keys when they're designed to store data long-term. This ensures row-level uniqueness even when business keys overlap.

### Data Vault Components

Data Vault hubs, satellites, and links often use surrogate keys to ensure uniqueness and maintain relationships between entities.

## How Surrogate Keys Are Defined

Surrogate keys are identified in Node Definitions using metadata:

```yaml
isSurrogateKey: true
```

This metadata tells the Node template to:

- Generate the appropriate identity syntax in the `CREATE TABLE` statement.
- Exclude the surrogate key column from insert and merge operations.

Surrogate key behavior must be handled by the Node Type or template. It cannot be added to Nodes that don't already have it as part of the Node Template.

## Alternative Surrogate Key Strategies

While identity columns are the default implementation, Coalesce also supports alternative approaches through templates.

### Sequences

Sequences can be used instead of identity columns when you need explicit control, such as during migrations or when coordinating key ranges across environments. Unlike identity columns, sequences appear in `INFORMATION_SCHEMA.SEQUENCES`.

### Hash-Based Keys

Hash-based surrogate keys can be generated by hashing one or more business keys. This approach produces repeatable values that are consistent across environments.

Coalesce supports MD5, SHA1, and SHA256 hash algorithms.

## Incremental Load Behavior

During incremental processing:

- New records receive new surrogate keys.
- Existing records retain their surrogate keys.
- For SCD Type 2 dimensions, changed records create new versions with new surrogate keys, while keeping the same business key.
- Fact Nodes maintain referential integrity by joining on surrogate keys.

## Migration Considerations

When loading legacy data:

- Existing surrogate keys can be inserted directly.
- New records continue incrementing from the highest existing value.
- Identity columns cannot be reseeded after creation.

If reseeding is required, templates can be customized to:

- Use a starting value parameter.
- Replace identity columns with sequences seeded above the legacy maximum.

---

## Understanding Deploy, Refresh, and Jobs in Coalesce

This guide goes over the Deploy, Refresh, and Jobs concepts in Coalesce.

## Deploy and Create

When you deploy your pipeline, you're executing the Data Definition Language (DDL) operations that establish the physical database structures needed for your transformation logic.

**During a deployment:**

- Creates or modifies database tables and columns.
- Executes Data Definition Language (DDL) statements.
- Performs ALTER, CREATE, and DELETE operations on your database objects.

### When to Deploy

- You've created a new data pipeline.
- You've made structural changes to your existing pipeline.
- You need to add new tables or columns to support additional data.

## Refresh and Run

A refresh operation executes the transformations defined in your pipeline, taking data from source tables and applying your business logic.

**During a refresh:**

- Runs defined data transformations
- Executes Data Manipulation Language (DML) statements
- Performs MERGE, INSERT, UPDATE, and TRUNCATE operations

### When to Refresh

- New data has arrived in your source systems.
- You want to update existing data with the latest transformations.

## Jobs

Jobs are how you organize and manage your refresh operations. Rather than refreshing your entire pipeline every time, jobs allow you to refresh specific parts of your pipeline based on your needs.

A job in Coalesce is a defined subset of nodes (transformation steps) that can be run together. Jobs are created using selector queries that identify which parts of your pipeline to include or exclude.

### Types of Jobs

Coalesce offers three main ways to refresh your data:

- **Predefined Jobs**: Created and saved in the Coalesce app with specific IDs, these can be scheduled and run repeatedly.
- **Ad-Hoc Jobs**: One-time jobs defined using include/exclude selectors, useful for testing or specific data processing needs.
- **Full Pipeline Refresh**: Refreshes all nodes in your pipeline, typically used for initial loads or complete refreshes.

:::info[Deploy Before Refresh]

You must deploy before you can refresh—you need containers before you can fill them with data.

:::

## Ways to Deploy and Refresh in Coalesce

- **Coalesce App**: Use the web interface for manual deploys and refreshes.
- **Coalesce Scheduler**: Schedule jobs directly within the Coalesce app.
- **Command Line Interface (CLI)**: Automate deploys and refreshes with command-line tools.
- **API**: Integrate with other systems using Coalesce's REST API.
- **Third-Party Tools**: Connect with orchestration tools like Apache Airflow, Azure Data Factory, or GitHub Actions.

## Best Practices

- **Create targeted jobs**: Break your pipeline into logical jobs that can be run independently.
- **Schedule wisely**: Consider data dependencies and processing windows when scheduling jobs.
- **Use parameters**: Leverage parameters to make your pipeline flexible across environments.

---

## What's Next?

- [**Create an Environment**][] - Step by step instructions on setting up your Environment for deployments.
- [**Coalesce Scheduler**][] - Learn how to scheduler Jobs right in Coalesce.
- [**Deployment**][] - Go over deployment and requirements before deploying in Coalesce.
- [**Refresh**][] - Learn how to create Jobs and refresh your Environment.

---

[**Create an Environment**]: /docs/setup-your-project/create-your-environments
[**Coalesce Scheduler**]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[**Deployment**]: /docs/deploy-and-refresh/deploy/
[**Refresh**]: /docs/deploy-and-refresh/refresh/refreshing-an-environment

---

## Understanding View Nodes in Coalesce

View Nodes in Coalesce create generic SQL views in your data warehouse. While many Node types can be materialized as views, the View Node type exists specifically to improve DAG readability by making views visually distinct from other Node types.

## When View Nodes Are Created

Views are created and modified exclusively during deployment operations, not during refresh jobs.

### Initial Deployment

When you deploy a View Node for the first time, Coalesce executes a "Create View" stage that runs a `CREATE OR REPLACE` statement in your target environment.

### Redeployment Triggers

Views are recreated during deployment when you make changes to:

- View definition or SQL logic
- Table descriptions
- Secure options (converting to or from secure views)
- View names
- Column definitions

## No Execution During Refresh**

Views are excluded from refresh jobs entirely. They have no run stages, and no stages are rendered during a refresh operation. This exclusion happens because views are configured using a materialization type of "view."

### Why Views Don't Refresh

Views don't store data—they compute results dynamically at query time. When other Nodes reference a view during a refresh job, the database engine runs the view's SQL to generate results on demand, automatically providing the latest data from underlying tables without needing to refresh the view itself.

## Dynamic Data Updates

Views automatically reflect current data from their source tables every time they're queried. You never need to manually refresh a view. The database engine executes the view's SQL dynamically, ensuring query results always reflect the most recent state of the underlying data.

## View Node Configuration Options

View Nodes offer several configuration options to customize their behavior:

- **Distinct:** Toggle for `SELECT DISTINCT` functionality
- **Multi-source:** Toggle for combining multiple sources
- **Override Create SQL:** Toggle for custom SQL
- **Multi Source Strategy:** Choose between `UNION` versus `UNION ALL` when combining sources

### Override Create SQL Feature

The Override Create SQL option allows you to write completely custom SQL for the view definition.

## View Nodes Versus Stage Nodes

While Stage Nodes can be materialized as views, dedicated View Nodes have distinct characteristics:

- Only allow view materialization (no table option).
- Have view-specific configuration options like Distinct.

## The Run View Alternative

If you need a view to be recreated during refresh jobs—for example, to utilize dynamic parameters for source selection—you can use the Run View Node type from the [Incremental Loading Package][].

Unlike standard View Nodes, Run View Nodes include view creation logic in the Run tab. When a job executes using Run View Nodes, the views are recreated as part of the job execution and utilize whatever parameter values are passed in at runtime.

## Delete View Nodes

If you delete a View Node from a Workspace and deploy that change, the view will be dropped from the target environment.

## Enabling View Nodes

View Nodes are disabled by default in Coalesce. You need to manually enable them by navigating to **Build Settings > Node Types** and toggling the View Node type option. This is because Stage Nodes can also be materialized as views and provide the same functionality with more flexibility.

[Incremental Loading Package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading#run-view

---

## Workspaces

A Workspace is a sandbox environment where you can complete the development of your data, refine it, and perform preliminary validation before you merge your work into the codebase. Each Workspace has its own graph, storage locations, macros, node types, connection configuration, and Git branch. You can create multiple Workspaces to work on different tasks and merge them into the codebase using Environments. For example, a data engineer may be tasked with creating a new data mart. They could create a new Workspace with its own Git branch, separate from the main Workspace. Once the feature is done, its code can be merged back into the main branch.

Workspaces are where you:

- Complete all Coalesce development in the Coalesce App.
- Create objects and load or refresh data manually on-demand, for individual objects, for part of the data pipeline, or for the full data pipeline.
- When a change is made to an object, such as adding or dropping a column or renaming a column, the object is dropped and recreated rather than altered with `ALTER`.
- Deleted objects are not dropped in the backend instance.

## Managing Workspaces

Access to a project's Workspaces follows project access. People you add under **Project Settings > Members** can work in that project's Workspaces according to their project role. For steps and role details, see [How to Assign User Roles][] and [RBAC permissions and roles][].

For an in-depth guide on creating Workspaces, take a look at [Step 3: Create a Workspace][].

### Create a Workspace

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. Give your Workspace a Name and Description.
4. Select a current branch to make a new branch from. For example, if you want to create a branch off main, select main, then the commit you want to start from, and then in **New Branch Name**, enter the new branch name.
5. Then click **Create**.
6. Your new Workspace will be created with the new branch.

### Edit a Workspace

There are two ways to edit a Workspace:

- Go to **Build Settings** > **Workspace** and click the cog.
- From the Project page, click the cog next to the Workspace.

You can update the following in a Workspace:

- Settings includes Workspace name, description, and tag color.
- User Credentials including OAuth
- Storage Mappings
- Parameters

### Workspace Settings Overview

To edit or create a Workspace, go to **Build Settings > Workspace**.

1. **Settings** - Includes the Workspace name, description, tag color, creator, creation date, and account URL. For Workspace and other IDs you use in API calls, see [Workspace and Environment IDs][].
2. **User Credentials** - Configure how you connect to your Data Platform. For Snowflake, key pair authentication is recommended. See [Setting Up a Snowflake Service Account for Coalesce][].
3. **Storage Mappings** - To deploy or refresh, you need Storage Mappings set up. Storage Mappings represent the physical locations in your Data Platform where your data pipeline deploys. See [Storage Locations and Storage Mappings][].
4. **Parameters** - Parameters are defined as a JSON blob you set that can be accessed in the metadata during template rendering. Learn more about [Parameters][].
5. **OAuth Settings** - Enter the OAuth configuration information for your data platform

### Duplicate or Copy a Workspace

To duplicate a Workspace, go to the Projects page and click the **Duplicate Settings** button, <DuplicateSVG />. Then follow the on-screen prompts.

To copy a Workspace, click the three dots next to the Workspace and select **Copy Objects from Workspace**.

Review [When To Use Copy Objects, Duplicate, or Create New Project][] to learn more about duplicating versus copying and what information will migrate to the Workspace.

:::info[Passwords]

Passwords aren't copied.

:::

### Delete a Workspace

There are two ways to open Workspace Settings:

- Go to **Build Settings** > **Workspace** and click the cog.
- From the Project page, click the cog next to the Workspace.

In Workspace Settings, click **Delete Workspace**.

If you delete all Workspaces, Coalesce creates a new blank Workspace named DEV, because there must always be at least one development Workspace.

## Share Workspace Links

In **Build**, your browser URL can include **Project** and **Workspace** identifiers. Copy the full address bar URL so teammates with access open the same context.

1. Open **Build** for the Project and Workspace you want.
2. Copy the URL from your browser and send it.

:::warning[Project access]

A shared link does not grant access. The recipient needs project membership and a role that allows them to see the information. Check **Project Settings > Members** if something fails.

:::

---

## What's Next?

- [Step 3: Create a Workspace][] - Learn how to create a Workspace

---

[When To Use Copy Objects, Duplicate, or Create New Project]: /docs/get-started/coalesce-fundamentals/copying-workspace-and-project-data
[Step 3: Create a Workspace]: /docs/setup-your-project/create-a-workspace
[Parameters]: /docs/deploy-and-refresh/parameters/
[Workspace and Environment IDs]: /docs/reference/whats-my-id
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[How to Assign User Roles]: /docs/organization-and-accounts/organization-management/role-based-access-control/assign-rbac-roles
[RBAC permissions and roles]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Setting Up a Snowflake Service Account for Coalesce]: /docs/setup-your-project/connection-guides/snowflake/snowflake-service-accounts

---

## Coalesce Tips and Tricks

Learn how to get the best out of the Coalesce App.

## Editing Code

In any code editor in Coalesce, you have access to a number of functions. Right click in the box to bring a sub-menu that has different actions, along with the available keyboard shortcuts.

Selecting Command Palette(F1) brings up a full menu of options, along with the keyboard shortcuts.

## Find and Replace

You can use `CTRL+F` on Windows or `CMD+F`  Mac to search, find and replace text any location in Coalesce that has a code editor.

- Node Configuration
- Join Tab
- Column Editor
- Column Testing
- Pre-SQL and Post-SQL
- Git Modal (read only)

## Searching Nodes

You can search for the name of a Node in the Build Interface sidebar.

## Drag and Drop Columns

You can drag and drop columns in multiple ways in Coalesce.

- Multi Source Nodes
- Ordering columns in a Node
- Adding columns to a Node
- Add Nodes to Jobs

## Information Bar

At the bottom of the screen, an information bar is shown. It contains the current information about your Workspace.

From left to right:

- **Branch**: The current Git branch. You can click this to open the Git modal.
- **Last committed**: Date of the last commit.
- **Data Platform**: The data platform account name.
- **User**: The email of the data platform account.
- **Role**: The role of the data platform account.
- **Email**: The email of the user logged into Coalesce.
- **Org**: The organization and user name logged into Coalesce.

If you are on the Project page, it will show the Git URL and the Git account name given during setup.

## Tooltips

Throughout the app, you'll see the information icon. Hover over it to find useful information related to the section.

## Close All Tabs

Quickly close all open tabs while working by clicking on the folder.

## Viewing Your DAG

On the Build page, there are several options that can help you view your DAG. From left to right:

1. **Node View Options**
   1. **All** - View all Nodes
   2. **Upstream** - Shows all Nodes that feed into the selected Node.
   3. **Downstream** - Shows all Nodes that depend on the selected Node.
   4. **Related** - Shows both upstream and downstream Nodes for the selected Node.
   5. **Spotlight** - Shows both upstream and downstream Nodes up to four levels deep.
2. **Mini map** - The mini map shows a view of the entire graph and highlights your current view port so you can jump quickly to any area.
3. **Draw optimized graph** - This will attempt to adjust the DAG view so the connections between nodes are easier to see at first glance. It doesn't change the order of execution.
4. **Zoom out and Zoom in** - Zoom in and out on the DAG.
5. **Full screen**- Full screen attempts to fit the entire DAG within the given space.

## Sort Table Data

You can sort column data by a single column or multiple columns.

### Single Column Sort

Click a column header to sort by ascending, descending, and to remove the sort and rest the order.

### Multi-Column Sort

Multi-column sort lets you select multiple columns to first sort your data by one column and then further sort it by another.

To select multiple columns, use the keyboard shortcuts:

- macOS: `CMD+SHIFT`
- Windows: `CTRL+SHIFT`

As you select columns, they have will have 1, 2, 3, etc appear next to them indicating the order they were selected and will sort by.

- **Primary Sort**: Begin by sorting your table based on the main column. For example, if you have a lot of sources, and you want to first group by source.
- **Secondary Sort**: This is the second column you want to sort on. Using the same example, After sorting by sources, then you might sort by Storage Location.
- **Tertiary and Additional Sorting** - You can continue to sort data with the selected columns. Continuing the previous example, after sorting by sources, Storage Locations, you can sort by column name to see them alphabetically.

<Wistia id="dz1pcd61ae" />

---

## Feature Availability

## General Availability

Product features are enabled by default for all customers. Preview tags are removed, support is available based on your plan, and documentation is available.

## Private Preview

Coalesce may select a set of customers for early testing of certain features. Private preview can include:

- Bug Fixes
- Updates to existing features or new features
- New features

### Understanding Private Preview Releases

- Features and updates in private preview might not be available for GA.
- Documentation is available with a preview tag and subject to change.
- Customers in private preview can reach out to support for any questions they have.
- Features in private preview are subject to change.
- Private preview releases are marked by a preview tag in the documentation and in the release notes.

<PrivatePreview />

---

## Get Started

Get up and running quickly with Coalesce using our Quick Start guide.

<!-- markdownlint-disable -->
<!-- vale off -->

<CardWithIcon
  title="Transform Onboarding"
  icon="/img/card-icons/colored-bg/bright-blue/quick-start-icon.svg"
  description="Follow the Transform Onboarding Guide for a step-by-step checklist from account creation through deployment. Covers prerequisites, data platform connection, Project and Workspace setup, building your first pipeline, and deployment."
  link="/docs/get-started/transform-onboarding-guide"
  linkText="Start the Onboarding Guide"
/>

<CardWithIcon
  title="Quick Start Guides"
  icon="/img/card-icons/colored-bg/bright-blue/speed-icon.svg"
  description="Get started with Coalesce by using one quickstarts to learn the basics of Coalesce with a variety of data platforms."
  link="/docs/get-started/quick-start"
  linkText="View Guides"
/>

<CardWithIcon
  title="AI in Coalesce"
  icon="/img/card-icons/colored-bg/light-blue/ai-ml-icon.svg"
  description="Discover how Coalesce leverages artificial intelligence to streamline data transformation workflows, enhance documentation, and scale machine learning operations with Snowflake Cortex integration. Learn about intelligent documentation assistants, automated commit messages, and enterprise-scale AI operations for modern data pipelines."
  link="/docs/coalesce-ai/"
  linkText="Find Out More"
/>

<CardWithIcon
  title="Coalesce Fundamentals"
  icon="/img/card-icons/colored-bg/orange/life-sciences-icon.svg"
  description="These guides introduce fundamental concepts you need to understand in order to use Coalesce effectively."
  link="/docs/get-started/coalesce-fundamentals"
  linkText="View Guides"
/>

<CardWithIcon
  title="Coalesce Best Practices"
  icon="/img/card-icons/colored-bg/teal/101-icon.svg"
  description="Learn best practices for setting up and managing your Coalesce projects, including environment configuration, Git integration, workspace strategies, and deployment guidelines. Follow this checklist to optimize your Coalesce workflows for a more efficient and reliable data pipeline."
  link="/docs/get-started/coalesce-best-practices"
  linkText="Read Best Practices"
/>

<CardWithIcon
  title="Coalesce Tips and Tricks"
  icon="/img/card-icons/colored-bg/magenta/speed-icon-2.svg"
  description="Master Coalesce application productivity with essential tips and tricks for code editing, node searching, drag-and-drop functionality, DAG visualization, and interface optimization. Enhance your data transformation development workflow with these expert techniques and shortcuts."
  link="/docs/get-started/coalesce-tips-and-tricks"
  linkText="View Tips and Tricks"
/>

<CardWithIcon
  title="Contacting Support"
  icon="/img/card-icons/colored-bg/red/support-icon.svg"
  description="Connect with Coalesce support through multiple channels including Slack, Microsoft Teams, and email assistance. Access expert help for data transformation challenges, technical issues, and collaboration with Coalesce customer success teams."
  link="/docs/get-started/support-information"
  linkText="Find Out More"
/>

<CardWithIcon
  title="Feature Availability"
  icon="/img/card-icons/colored-bg/bright-blue/event-icon.svg"
  description="Understand Coalesce feature availability including General Availability releases and Private Preview programs. Learn about feature maturity levels, documentation status, and early access opportunities for cutting-edge data transformation capabilities."
  link="/docs/get-started/feature-availability"
  linkText="Find Out More"
/>

---

## Quick Start Guides

Get started with Coalesce by using one of our quickstart guides to learn the basics of Coalesce with a variety of data platforms.

<!-- markdownlint-disable -->
<!-- vale off -->

  
    <QuickStartCard
      img="/img/cards/card-snowflake.png"
      imgAlt="Snowflake"
      title="Snowflake"
      description="Learn to connect your Snowflake account to Coalesce, configure storage mappings, and build Source, Stage, Dimension, and Fact Nodes to transform and join your data."
      link="/docs/guides/snowflake-quickstart"
      linkText="Start Building"
    />
  
  
    <QuickStartCard
      img="/img/cards/card-databricks.png"
      imgAlt="Databricks"
      title="Databricks"
      description="Learn to set up Databricks data pipelines in Coalesce, integrate AccuWeather data, and build Stage, Dimension, and Fact Node."
      link="/docs/guides/databricks-quickstart"
      linkText="Start Building"
    />
  

  
    <QuickStartCard
      img="/img/cards/card-catalog.png"
      imgAlt="Catalog"
      title="Catalog"
      description="Quickly set up your Catalog by signing up, connecting your data warehouse and BI tool, and configuring integrations and settings for seamless metadata management."
      link="/docs/catalog/quick-set-up"
      linkText="Start Building"
    />
  
  
    <QuickStartCard
      img="/img/cards/card-coalesce.png"
      imgAlt="Coalesce"
      title="Coalesce"
      description="Master the basics of Coalesce by exploring its intuitive interface, transforming data with Stage Nodes, and building a star-schema using Dimension and Fact Nodes in Snowflake."
      link="/docs/guides/coalesce-foundational-hands-on-guide/"
      linkText="Start Building"
    />

---

## Contacting Support

Connecting with our team is simple. Choose the option that works best for you.

## Shared Slack or Teams Channel

We create dedicated Slack Connect or Microsoft Teams channels for communication between your team and our Customer Success experts. Our Product Managers and Solution Engineers frequently join these channels, enabling direct collaboration.

If you don't have a channel set up, email us at [support@coalesce.io](mailto:support@coalesce.io) to get started.

## Support Portal

Open new support requests and manage existing ones through our Support Portal. Visit [support.coalesce.io](https://support.coalesce.io) and log in with your email address to review your current and past support tickets.

## Email Support

For quick assistance, reach us directly by emailing [support@coalesce.io](mailto:support@coalesce.io). Our team is ready to help with any questions or issues.

## In-App Support

In the app, you can access support by:

- Clicking the question mark icon to bring up our AI Assistant.
- Clicking **Get Help** to open your default email client with [support@coalesce.io](mailto:support@coalesce.io) pre-populated as the recipient.

## Providing Information

To help our team provide fast support, include a description of your request along with your system information. You can provide all of the information by clicking **Copy All to Clipboard**. You can also copy individual fields using the copy icons.

- Email
- User ID
- User Role
- Org ID
- Org Name
- Software Version

---

## Transform Onboarding Guide

This guide walks you through everything you need to set up Coalesce Transform and deploy your first pipeline. Work through each section in order. For detailed instructions on any step, follow the linked guides.

## Who This Guide Is For

Transform onboarding involves two main roles:

- **Admins and setup owners:** Configure accounts, connect data platforms, create Projects and Workspaces, and manage Environments.
- **Developers:** Build pipelines, add sources, create transformations, deploy, and run refreshes.

## Prerequisites Checklist

Before starting, confirm you have:

- **Cloud data warehouse access:** Snowflake, Databricks, BigQuery, or Fabric. See [Connection guides][].
- **Git repository:** For version control (or plan to create one). See [Coalesce Git Requirements][].
- **Basic SQL and data transformation concepts:** Familiarity with SQL and data modeling.
- **Admin access:** To configure integrations and add team members. See [Administrative Tasks][].
- **Google Chrome:** The only supported browser. See [System Requirements][].

For network allowlisting, setup tasks, and more, see [Setup Requirements][].

---

## Phase 1: Account and Organization Setup

### Create Your Account

- Sign up [through a trial][] or [contact Coalesce][] for an enterprise account.
- Add team members and assign roles. See [Add Users and Set Permissions][].
- Configure SSO if needed. See [Authentication][].

### Configure Network Access

- Allow [inbound traffic][] from Coalesce IP addresses.
- Allow [outbound traffic][] to Coalesce domains.
- For Snowflake: Configure network policies. For Databricks: Configure egress policies. See [Network Requirements][].

### Connect Your Data Platform

- Choose your platform and follow the connection guide:
  - [Snowflake][]: Username/password, key pair, or OAuth
  - [Databricks][]: Unity Catalog required
  - [BigQuery][]: Service account authentication
  - [Fabric][]: See Fabric connection guide
- Test the connection before proceeding.

---

## Phase 2: Project and Workspace Setup

### Create Your Project

- Create a new Project from the Coalesce dashboard.
- Configure Git integration: choose provider ([GitHub][], [GitLab][], [Bitbucket][], or [Azure DevOps][]).
- Add your repository URL.
- Create a [personal access token][] for authentication.
- Use one Git repository per Project. See [Create Your Project][].

### Set Up Version Control

- Each user has their own Git provider account.
- Each user creates a [personal access token][] for Coalesce.
- Each user belongs to the organization's Git account.
- Decide your [branch strategy][] (for example, feature branches, main for deployment).
- See [Set Up Version Control][] for full details.

### Create Your Workspace

- From your Project, click **Create Workspace**.
- Complete the Onboarding Wizard: name, description, connection.
- Configure Storage Locations and mappings.
- Connect the Workspace to your data platform using credentials from the Connect Your Data Platform section above.
- See [Create a Workspace][] for step-by-step instructions.

---

## Phase 3: Build Your First Pipeline

### Add Sources

- From the Build screen, click **+** then **Add Sources**.
- Select the tables you want to add.
- Preview each source before adding.
- See [Add a Data Source][] for details.

### Build Transformations

- Add Nodes (Stage, Dimension, Fact, View, or Custom).
- Configure column transforms, joins, and filters.
- Check the [Marketplace][] for pre-built Node types that fit your use case.
- See [Transforms][] and [Nodes][] for details.

### Validate Your Pipeline

- Run the pipeline (or a subset) to populate tables.
- Preview data in each Node to verify transformations.
- Use the [Problem Scanner][] to catch errors before deploying.

---

## Phase 4: Environments and Deployment

### Create Environments

- Go to **Build Settings > Environments** in your Workspace.
- Create Environments for DEV, QA, and Production (or your naming convention).
- For each Environment:
  - Configure authentication (username/password, OAuth, or key pair).
  - Set Storage Mappings (database and schema).
  - Add Parameters if needed.
- Each Environment should map to a distinct database and schema. See [Create Your Environments][].

### Deploy

:::info[Governance before production]

Before you deploy to production, align with your organization's Git and CI/CD standards. Branch protection, required reviews, and deployment gates are enforced on your Git platform or in automation. Coalesce deploys the branch and commit you select. Read [Governance and Production Deployments][] and [DataOps Best Practices with Git and Coalesce][] for the full pattern.

:::

- Ensure your Workspace is on the main branch with no uncommitted changes (for production).
- Go to **Deploy** and select your target Environment.
- Review the deployment plan.
- Deploy using the [Coalesce App][], [CLI][], or [third-party tools][].
- See [Deployment Overview][] and [Deploying to an Environment][].

### Set Up Refresh and Jobs

- Create Jobs by selecting the Nodes to include.
- Deploy before refreshing (Jobs run only on deployed Nodes).
- Schedule refreshes using the [Coalesce Scheduler][], [CLI][], [Jobs API][], or external tools.
- See [Refresh Your Pipeline][] and [Scheduling Jobs][].

---

## Phase 5: Team Rollout

- Add developers to the Project.
- Establish a [Workspace strategy][] (one per branch, one per user, or one per feature).
- Designate a single developer to commit to the main branch.
- See [Coalesce Best Practices][] for ongoing guidance.

---

## Optional: Advanced Paths

### AI Features

- **Copilot:** Use natural language to generate transformations. See [Copilot][] and [Migrating SQL to Coalesce with Copilot][].
- **AI-generated descriptions:** Add descriptions to Nodes and columns for documentation and lineage.

See [Coalesce AI][] for the full set of AI capabilities.

### Programmatic Setup

- **Project APIs:** Automate project, Workspace, and Environment creation. See [API documentation][].
- **CLI:** Deploy and refresh from the command line. See [CLI][].
- **Automation:** Use APIs and CLI for CI/CD, Workspace provisioning, and Environment management.

### Integrations

- **Catalog:** Sync lineage and documentation to Coalesce Catalog. See [Catalog integration with Coalesce][].
- **Marketplace packages:** Add pre-built Node types and patterns. See [Marketplace][].
- **External orchestrators:** Integrate with Airflow, GitHub Actions, GitLab, and others. See [Third-Party DevOps Tools][].

---

## Get Help

### Support Channels

- **Shared Slack or Teams channel:** Dedicated channel for your team and Coalesce Customer Success.
- **Email:** [support@coalesce.io](mailto:support@coalesce.io) for quick assistance.
- **In-app support:** Click the question mark icon for the AI Assistant, or **Get Help** to open an email to support.

When contacting support, include your Environment ID, run ID, and error details. Use **Copy All to Clipboard** in the app to capture system information. See [Contacting Support][] for full details.

### Self-Service Resources

- **Quick Starts:** [Snowflake Quick Start][], [Databricks Build Weather Analytics][]
- **Foundational guide:** [Coalesce Foundational Hands-On Guide][]
- **FAQ and troubleshooting:** [FAQ][] and [Troubleshooting Deployments and Refreshes][]

---

## What's Next?

- [Coalesce Best Practices][] for ongoing workflow and deployment guidance
- [Coalesce Catalog Onboarding Guide][] to add discovery and governance with Catalog
- [Marketplace][] to explore pre-built Node types and patterns
- [Troubleshooting Deployments and Refreshes][] if you run into issues

---

[Connection guides]: /docs/category/connection-guides
[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements
[Administrative Tasks]: /docs/setup-your-project/setup-requirements/administrative-tasks
[System Requirements]: /docs/setup-your-project/setup-requirements/system-requirements
[Setup Requirements]: /docs/category/setup-requirements
[through a trial]: https://coalesce.io/product/transform/start-free/
[contact Coalesce]: https://coalesce.io/product/request-demo/
[Add Users and Set Permissions]: /docs/setup-your-project/add-users-set-permissions
[Authentication]: /docs/organization-and-accounts/organization-management/single-sign-on/
[inbound traffic]: /docs/setup-your-project/setup-requirements/network-requirements
[outbound traffic]: /docs/setup-your-project/setup-requirements/network-requirements
[Network Requirements]: /docs/setup-your-project/setup-requirements/network-requirements
[Snowflake]: /docs/setup-your-project/connection-guides/snowflake/snowflake-username-and-password
[Databricks]: /docs/setup-your-project/connection-guides/databricks
[BigQuery]: /docs/setup-your-project/connection-guides/bigquery
[Fabric]: /docs/setup-your-project/connection-guides/fabric
[Create Your Project]: /docs/setup-your-project/create-your-project
[GitHub]: /docs/git-integration/set-up-your-git-integration
[GitLab]: /docs/git-integration/set-up-your-git-integration
[Bitbucket]: /docs/git-integration/set-up-your-git-integration
[Azure DevOps]: /docs/git-integration/set-up-your-git-integration
[personal access token]: /docs/git-integration/set-up-your-git-integration
[branch strategy]: /docs/git-integration/git-branches
[Set Up Version Control]: /docs/setup-your-project/setup-version-control
[Create a Workspace]: /docs/setup-your-project/create-a-workspace
[Add a Data Source]: /docs/setup-your-project/add-a-data-source
[Marketplace]: /docs/marketplace/
[Transforms]: /docs/build-your-pipeline/transforms
[Nodes]: /docs/get-started/coalesce-fundamentals/nodes
[Problem Scanner]: /docs/build-your-pipeline/the-build-interface/problem-scanner
[Create Your Environments]: /docs/setup-your-project/create-your-environments
[Coalesce App]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment
[CLI]: /docs/deploy-and-refresh/deploy/deploy-using-the-cli
[third-party tools]: /docs/deploy-and-refresh/third-party-devops-tools/
[Deployment Overview]: /docs/deploy-and-refresh/deploy/
[Deploying to an Environment]: /docs/deploy-and-refresh/deploy/deploying-to-an-environment
[Refresh Your Pipeline]: /docs/deploy-and-refresh/refresh/
[Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Jobs API]: /docs/api/
[Scheduling Jobs]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[Workspace strategy]: /docs/get-started/coalesce-fundamentals/workspaces
[Coalesce Best Practices]: /docs/get-started/coalesce-best-practices
[Catalog integration with Coalesce]: /docs/catalog/integrations/transformation/coalesce/
[Third-Party DevOps Tools]: /docs/deploy-and-refresh/third-party-devops-tools/
[Copilot]: /docs/coalesce-ai/copilot/
[Migrating SQL to Coalesce with Copilot]: /docs/guides/migrating-sql-to-coalesce-with-copilot
[Coalesce AI]: /docs/coalesce-ai/
[API documentation]: /docs/api/
[Contacting Support]: /docs/get-started/support-information
[Snowflake Quick Start]: /docs/guides/snowflake-quickstart
[Databricks Build Weather Analytics]: /docs/guides/databricks-build-weather-analytics
[Coalesce Foundational Hands-On Guide]: /docs/guides/coalesce-foundational-hands-on-guide
[FAQ]: /docs/category/faqs
[Troubleshooting Deployments and Refreshes]: /docs/deploy-and-refresh/troubleshooting-deployments-and-refreshes/
[Coalesce Catalog Onboarding Guide]: /docs/catalog/catalog-onboarding-guide
[Governance and Production Deployments]: /docs/deploy-and-refresh/deploy#governance-and-production-deployments
[DataOps Best Practices with Git and Coalesce]: /docs/guides/dataops-and-git-best-practices

---

## Creating a Git Service Account

When you run automated processes (such as CI/CD pipelines, deployments triggered by Job Schedule, or GitHub Actions), using a developer's personal account is risky. If that person leaves, tokens expire, or permissions change, your automation breaks.

With Coalesce, creating a dedicated version control account tied to automated functions is best practice. This approach gives you:

- **Stability**: Automation isn't tied to any individual employee.
- **Auditability**: Commits from automation are clearly attributed to a dedicated account.
- **Security**: You can scope tokens and permissions to only what automation needs.

## When to Use a Dedicated Git Account

Use a dedicated Git account for automation when:

- **GitHub Actions or CI/CD** runs deployments or refreshes. Store your Coalesce API token and other secrets in GitHub; Coalesce uses the Git account configured for that Project to perform deployments. See [Orchestrate Jobs With GitHub Actions and the API][] or [Orchestrate Deploys With GitHub Actions and the CLI][] for workflow setup.
- **Job Schedule** triggers deployments or commits. Select the automation account as the default Git account when creating the Project.
- **CLI or API** triggers deployments. The deployment uses the Git account associated with the Project and Environment.
- **Shared Workspaces** used by automation need consistent Git credentials.

For interactive development, developers can continue using their personal Git accounts in Coalesce. This guide focuses on the account used for automation.

## Before You Begin

Complete the following before you begin:

- Review [Coalesce Git Requirements][] before starting.
- Ensure you have admin access to your GitHub organization or repository.
- Have a [Project][] in Coalesce with version control configured, or plan to add it during setup.

## Create a Dedicated GitHub Account for Automation

:::info[Detailed Setup]

Review [Step: 1 Setup Version Control][] for detailed steps and videos.

:::

You can create a dedicated account for automation. Many organizations use a pattern such as:

- `coalesce-bot@yourcompany.com`
- `coalesce-automation@yourcompany.com`
- `data-platform-ci@yourcompany.com`

When setting up the account:

- Use a distribution list or shared mailbox so the account isn't tied to a single person.
- Add the account as a collaborator with the appropriate permissions (for example, Write access for the repository).

### Step 1: Generate a Fine-Grained Access Token

Generate a fine-grained token in GitHub with these steps:

1. Click on your GitHub profile and go to **Settings > Developer Settings**.
2. Click **Personal access tokens > Fine-grained tokens**.

    

3. Give the token a name and description.
4. Set the following based on your company policy.
   1. [Resource owner][]
   2. Expiration date
   3. Repository access should be either **All repositories** or **Only select repositories**.
5. Open **Permissions > Repository permissions**.
6. Set **Contents** to **Read and write**. GitHub will also automatically grant Metadata, which is required.

    

7. Save your access token somewhere secure. You won't be able to recover it.

:::info[Token Permissions]

For Coalesce, the token needs **Contents: Read and write**. This allows the account to push commits, pull changes, and manage branches. No other repository permissions are required.

:::

### Step 2: Add the Account to Coalesce

Add the automation account to your Project in Coalesce:

1. Go to the Project page. If you are on the [Build page](/docs/build-your-pipeline/the-build-interface/) , click the **back arrow**.
2. Click the **plus sign**(+) next to Projects.

   

3. Enter the Project name and description. Click **Next**.

   

4. Enter your **Version control repository URL**.

   

5. On the next page, Click **Add New Account**.
   1. Enter an account nickname. This will displayed in the interface.
   2. Enter your username. If you are using Bitbucket AND generated a repo access token, set `x-token-auth` as the username.
   3. Enter your token. This will be either the GitLab token, Git token, Azure token, or Bitbucket Token
   4. Enter the **Author Name**, which identifies the committer. This doesn't have to match your version control account.
   5. Enter the **Author Email**, which identifies the committer email. This doesn't have to match your version control account.
   6. Click **Add**.

    

6. Select the Git account you just created in the drop down, then click **Test Account**.

   

7. Then click **Finish** to create your Project.

:::info[Existing Projects]

If your Project already has version control configured, you can add a new Git account from **User Menu > User Settings > Git Accounts**. Click **Add** and fill in the same fields. Then select the new account when configuring the Project or Workspace used by automation.

:::

## Coalesce Service Accounts vs. Git Accounts

Coalesce has its own [service accounts][] for running deployments, refreshes, and scheduled jobs in the Coalesce app. Those are separate from Git accounts:

- **Coalesce service accounts**: Log into Coalesce and perform deployments, refreshes, and other operations. They are created in **Org Settings > Users** and use key pair or machine-to-machine authentication. See [Service Accounts in Coalesce][] and [Adding Users and Setting Permissions][].
- **Git accounts**: Authenticate with your Git provider (GitHub, GitLab, Bitbucket, Azure DevOps) when Coalesce commits or pushes code. They are created in **User Settings > Git Accounts** or during Project setup.

For a fully automated pipeline, you typically need both:

1. A **Coalesce service account** with Environment Admin (or equivalent) to deploy and refresh.
2. A **Git account** tied to a dedicated GitHub (or other provider) account for version control operations.

## Best Practices

Follow these practices when managing automation accounts:

- **Use a dedicated account**: Don't use a developer's personal GitHub account for automation.
- **Use distribution emails**: Tie the automation account to a shared mailbox (for example, `coalesce-bot@yourcompany.com`) so it isn't tied to one person.
- **Scope tokens narrowly**: Use fine-grained tokens with access only to the repositories Coalesce needs.
- **Rotate tokens periodically**: Set an expiration and rotate tokens before they expire. Update the token in Coalesce when you rotate.
- **Document the account**: Record who manages the account and where tokens are stored so your team can maintain it.

## What's Next

- [Set Up Version Control][]—Full setup guide for GitHub, GitLab, Bitbucket, and Azure DevOps.
- [Managing Git Accounts][]—Change repository URLs, add accounts, and update credentials.
- [Service Accounts in Coalesce][]—Configure Coalesce service accounts for deployments and refreshes.
- [Orchestrate Jobs With GitHub Actions and the API][]—Automate deployments with GitHub Actions.
- [DataOps Best Practices with Git and Coalesce][]—Branching, merging, and deployment workflows.

[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements
[Project]: /docs/setup-your-project/create-your-project
[Set Up Version Control]: /docs/git-integration/set-up-your-git-integration
[Managing Git Accounts]: /docs/git-integration/managing-git-accounts
[service accounts]: /docs/organization-and-accounts/organization-management/role-based-access-control/service-accounts
[Service Accounts in Coalesce]: /docs/organization-and-accounts/organization-management/role-based-access-control/service-accounts
[Adding Users and Setting Permissions]: /docs/setup-your-project/add-users-set-permissions
[Orchestrate Jobs With GitHub Actions and the API]: /docs/deploy-and-refresh/third-party-devops-tools/continuous-integration/schedule-jobs-with-github-actions-and-the-api
[Orchestrate Deploys With GitHub Actions and the CLI]: /docs/deploy-and-refresh/third-party-devops-tools/continuous-integration/schedule-jobs-github-actions-cli
[DataOps Best Practices with Git and Coalesce]: /docs/guides/dataops-and-git-best-practices
[Resource owner]: https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/
[Step: 1 Setup Version Control]: /docs/setup-your-project/setup-version-control

---

## Git Basics

This guide will go over some basic Git concepts you’ll use in Coalesce.

## What Is Git?

Git is a distributed version control system that helps developers track and manage changes to their code over time.

There are different version control methods including [Git][], [Apache Subversion][], and [Mercurial][]. Coalesce uses Git for version control. Git is supported by many providers including:

- GitHub
- Azure DevOps
- Bitbucket
- GitLab

:::info[Git Requirements]

Before setting up Git in Coalesce, review [Coalesce Git Requirements][].

:::

You'll see the term Git used throughout these guides to mean the version control system, and not GitHub. These methods should be applicable to any other version control system using Git.

The main functions of Git you’ll use are:

- Repository
- Stage
- Commit and Push
- Branch and Merge

## Repository

A repository is where you keep your code. There are two types:

- **Local** - On your computer and typically not accessible by others.
- **Remote** - A shared repository where others will contribute work. Such as GitHub or GitLab.

A typical workflow will mean a central **remote** repository that you have **cloned** locally. You make your changes locally then **push** them to the remote repository to contribute to the overall code.

In Coalesce, when you create a project, you’ll need to provide a URL to the remote repository so Coalesce can keep it updated with the changes you make.

## Stage

**Stage** means you’ve selected which changes you want to go into your next commit. Selecting your changes in the Git Modal means you’re staging your changes.

## Commit and Push

In Coalesce, we run your commit and push at the same time.

**Commit** means you take a record or “snapshot” of your local repository at that time. When making a commit, a commit message describes the changes made. It’s required in Coalesce to make a push.

**Push** means taking the local commits you made and sending them to your remote repository. The remote repository is the central location where all code and changes are stored.

When you click Commit and Push, Coalesce runs both of these at the same time.

## Branches and Merge

**Branches** are where you’ll do your work. Typically you’ll create a branch off the main or feature branch. A branch contains changes made. A branch can be local or remote. A good workflow is to periodically commit and push your local changes to your remote branches. Then once you’re ready, merge your branch into the main feature branch.

**Merge** means adding your branch changes into another branch. Typically you’ll merge into the main branch.

[Git]: https://git-scm.com
[Apache Subversion]: https://subversion.apache.org/
[Mercurial]: https://www.mercurial-scm.org/
[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements

---

## Git Branches

This guide will go over creating branches and reading the branches tab in the Git Modal.

## Branches

When you first open the branch tab, it will show the branch you are currently working on at the top. The Selected Branch will default to the branch you are working in. The drop down will list all available branches. You can only have one branch per Workspace.

The image shows a branch called `understanding_commits` with commits listed.

- **Message** - The commit message
- **Author** - The username of the person who made the commit
- **Commit** - The commit hash. A unique commit ID to identify the commit.
- **Date** - When the commit was done
- **Used by Workspace** - The Workspace the commit is used in.
- **Deployed In Environment** - The environment the commit was deployed in.

### Branch Actions

- **Merge Latest -** Merge the latest changes from one branch to another. The modal will show which direction the changes are happening.
- **Check Out Latest -** Create a copy of the branch for you to work on in Coalesce. A progress modal shows while checkout runs. To make the changes available for everyone you’ll need to commit and push to either the remote main branch or to the remote branch you’re working on.
- **Force Checkout -** A force checkout means it will checkout any changes made on the remote to your local branch without checking to see if they are compatible. For example, if you made changes to your branch, but run force checkout, it will overwrite the local changes if they differ making it match the remote branch. You could lose local changes, so only use this if you are sure you want to overwrite changes.
- **Fetch -** Get any changes from the remote. It doesn’t overwrite the changes you’re working on. It’s a good way to get other branches and see what others have been working on or to get the most recent changes to the main branch.
- **New Branch** - Each commit will have the New Branch option next to it. You can create a new branch from that commit. For example, in our current branch, there are three commits. A commit is a snapshot of the branch at that point in time. If you decide to create a new branch from the **Coalesce Automatic Upgrade** commit, then that means anything in the **New branch** won’t be available.

## Create a New Branch

To make a new branch, click on the New Branch link on an existing commit, type in a name for it, and a new branch will be created from the current one.

## Checkout a Branch

When you create a new branch, you will automatically check it out, but if you'd like to return to another branch, you can do so by selecting the branch from the dropdown and clicking **Check Out Latest**.

A progress modal opens while Coalesce checks out the branch and syncs files. Wait until the modal finishes before you close it or take other actions in the Git Modal. On larger Workspaces, checkout can take longer before the Workspace is ready. Don't refresh the browser while checkout is in progress.

If you haven't committed your latest changes on your current branch, you will receive a warning message uncommitted changes will be moved to the new branch. If you'd like to force the checkout, click **Force Checkout**. Forcing a checkout to a new branch means you’ll lose any changes you haven’t committed.

## Force Checkout

Force checkout lets you switch branches. Only do a force checkout when you want to discard local changes in your current branch and revert to the last committed state of the branch you are checking out. This can be helpful if your current changes are causing issues or if you want to start fresh from the last known good state.

If you have uncommitted changes, you’ll need to confirm you want to proceed. If you have committed your changes, you won’t need to confirm.

:::danger[Use Force Checkout With Caution]

Force checkout will permanently delete your uncommitted changes and they can’t be recovered. Commit your changes before proceeding.

:::

## Merge Branches

After making changes and making a commit to a branch, you can merge the changes back to another branch.

1. Check out your destination branch to make it your **Current Branch**.
2. Select the source branch with the incoming changes from the **Selected Branch** dropdown.
3. Click on **Merge Latest** or **Merge** next to the desired commit you would like to merge into the current branch.

All the changes from the **Selected Branch** will be merged into your **Current Branch**. If you get an error while trying to merge, review [Merge Conflicts][].

- Merges are only allowed when there are no uncommitted changes in the target branch. The Merge button is disabled until the changes are either discarded or committed.

### Merge and Merge Latest

- **Merge** will merge all changes and commit history into the branch.
- **Merge Latest** will only merge the most recent commit into the branch.

## Delete Branches

:::danger[Deleting Branches is Irreversible]

Before deleting a branch, make sure it's no longer needed. This action is irreversible.

:::

Branches can't be deleted within Coalesce, but they can be deleted in your version control provider.

- [GitHub][]
- [GitLab][]
- [Azure Repos][]
- [Bitbucket][]

In your Coalesce app, fetch the latest changes after deleting the branch.

[Merge Conflicts]: /docs/git-integration/solving-git-errors/merge-conflicts
[GitHub]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-and-deleting-branches-within-your-repository#deleting-a-branch
[GitLab]: https://docs.gitlab.com/ee/user/project/repository/branches/
[Azure Repos]:https://learn.microsoft.com/en-us/azure/devops/repos/git/delete-git-branch?view=azure-devops&tabs=Browser
[Bitbucket]: https://support.atlassian.com/bitbucket-cloud/docs/manage-unmerged-branches/

---

## Git Commits

This guide goes over the commit tab of the Git Modal and making a commits to your repository.

## Commits

When you open the Git Modal, it will show any changes made and automatically select all files to be staged. You'll need to deselect any files you don't want.

1. **Git Modal Header** - The header will always show the Current Branch, the Current Commit and if there are any uncommitted changes. It will also indicate if you are on the Commit or Branches screen.
2. **Changes** - Changes shows any changes that were made to the current branch. Each one shows the number of files modified. Click each file to see the changes made. Learn more in [What Gets Committed][].
3. **Manage Changes**:
    1. **Stage All** - All files under changes are selected to be part of the next commit.
    2. **Unstage All** - No files are selected to be part of the next commit. You’ll need to select the files you want to commit. At least one file needs to be selected to make a commit.
    3. **Discard All** - Discard all changes that were made to the Workspace.
4. **File Differences** - Changes in red are deletions and changes in green are additions.  
    4\. Previous will show the contents before changes were made.  
    5\. Current will show the contents after changes were made, or what the file looks like currently. See Viewing File Differences for an example.
5. **Commit Message** - A commit message describes the changes that were made. Use [Coalesce's AI][]  to generate your commit message.
6. **Fetch** - Fetch gets all changes from the repository. Including any changes made to the branch you are working on.

## Viewing File Differences

To view changes to a specific file, click on it, and the two panes on the right will populate with the file's previous state in the left pane and its current state in the right pane. Scrolling through the file will show you patches of green and red, which denote where additions and deletions have occurred. In the following example, a `LOWER()` transformation was applied and the column `N_COMMENT` was deleted without a replacement.

Next to each file there will be a red, orange, or green dot.

- **Green** - A new Node or other type created.
- **Orange or Yellow** - There were changes to the data.
- **Red** - The Node or other type was deleted.

## Making a Commit

1. Select the files you want to **Stage**. If you need to make changes, you’ll need to make them in the Workspace. At least one file needs to be selected.
2. Write your **Commit Message**. It should be short and descriptive. Use [Coalesce's AI][]  to generate your commit message.
3. **Commit and Push** your changes.

If you have no more changes to display, the list on the left won’t have any files. If you didn't commit all files, the list will be reduced to show just those files. This will allow you to make multiple commits within one use of the Git Modal, following the same process outlined above. The uncommitted code will either need to be discarded or committed so that it no longer appears as a change when opening the Git modal in the future.

## Discard Changes

- To permanently roll back all changes to the code since the last commit to the Current Branch, click on the **Discard All** button. After accepting the warning, all code will be rolled back and the list of YAML files will be empty.
- Individual YAML files can be rolled back, by clicking on the undo button.

## Understanding the Commits

**Commit Init commit (4e9ae5d1)** is available in both the jobs-testing branch and main branch. The branch jobs-testing was created from the main branch at commit **4e9ae5d1** . That means jobs-testing contains any changes made in the main branch at the commit snapshot **4e9ae5d1**.

## Git Commit Notes

- The changes to the YAML files seen in the list, will include everyone’s changes who are working in the same Workspace (connected to the same Git Branch). It's best practice that everyone has their own Workspace to prevent confusion or accidentally removing each others work.
- Opening the Git modal captures a snapshot of the code in the Workspace, at that particular time. There is no need to click on the ‘Fetch’ button, as this has already happened on entry to the page.
- While the Git modal is open, if code is being changed by another user using the same Workspace, this will NOT be reflected, even if the ‘Fetch’ button is clicked. To include any last minute recent changes, simply close and re-open the Git modal.
- When discarding changes, this will affect all other users who are working in the same Workspace. Therefore use this option cautiously and ideally only discard individual items rather than everything.

[What Gets Committed]: /docs/git-integration/what-gets-committed
[Coalesce's AI]:/docs/coalesce-ai/

---

## Version Control

Coalesce leverages Git style version control to manage changes to your project.

## Supported Providers

- [GitHub](https://github.com/)
- [Bitbucket](https://bitbucket.org/)
- [GitLab](https://gitlab.com/)
- [Azure DevOps Git](https://dev.azure.com/)

:::info[Git Requirements]

Before setting up Git in Coalesce, review [Coalesce Git Requirements][].

:::

## The Git Modal

1. **Current Branch** - This is the branch you are working on or have checked out.
2. **Commit** - This tab contains information about your commit. A commit is any changes you’ve made while working. Learn more in Commits.
3. **Branches** - This tab will list any available branches. Learn more in Branches.
4. **Fetch** - Fetch will check the repository for any recent changes.

## Resources

- Go through our interactive Git tutorial. [DataOps Best Practices with Git and Coalesce](/docs/guides/dataops-and-git-best-practices/)
- Learn the basics of version control. [What is version control | Atlassian Git Tutorial](https://www.atlassian.com/git/tutorials/what-is-version-control)
- Understand how Git branches work in this interactive walkthrough. [Learn Git Branching](https://learngitbranching.js.org/?locale=en_US)
- [Git Cheat Sheet](https://education.github.com/git-cheat-sheet-education.pdf). A handy reference from GitHub.

<DocCardList />

[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements

---

## Managing Git Accounts

## Changing a Repository

:::danger[Data Loss Can Occur]

Only use this if you are sure. Changing the repository can incur loss of data such as missing changes, branches, and nodes.

:::

On the Projects page, click **Project Settings > Git Repository**. Enter the new URL.

If the version control account in the Project doesn't have access rights to the new URL, you'll need to add a new account. Follow the instructions in [Set Up Version Control][].

## Add a Git Account

In the Coalesce App, go to **User Menu > User Settings > Git Accounts**. Click **Add**. Fill in the account information.

[Set Up Version Control]: /docs/git-integration/set-up-your-git-integration

---

## Reverting a Commit

There may arise an occasion in which you need to revert a commit you have made to the repository used by Coalesce, for example, code was promoted from your DEV branch to your test branch before it was ready.

However, reverting the problem commit is not enough to reset your repository to your expected state. Instead, what you'll see if you simply revert is that when you go to reapply the changes once you've corrected your problems, your changed objects will not be in your change tracking list as you'd expect. What you have to do is actually revert the problem commit, and then revert your revert. While this seems logically counterintuitive, reverting the revert is necessary to rebase the history on your objects to the state it was in prior to your problem commit and ready the objects for re-commit at a later stage.

The following articles provide more information on why this in necessary when this occurs:

- [Git - Revert The Revert][]
- [How to restore changes which you’ve reverted from your main git branch][]
- [Create Pull Request from a Reverted Git Branch][]

To apply this practice to Coalesce, you would need to take the following steps within a Git command line interface (CLI) on your local machine/a Git client:

1. Checkout the branch you wish to revert the commit on and pull by running the following scripts:

    ```bash
    git checkout <brand_name>
    git pull
    ```

2. Locate the commit ID of the commit you wish to revert and make note of it.
3. Revert the commit by running:

    ```bash
    git revert <the commit ID you wish to revert> -m 1
    ```

4. Now, locate the commit ID of the revert you just completed in your branch, and execute the following script to “revert the revert”:

    ```bash
    git revert <the commit ID of your revert you completed in Step 3> -m 1
    ```

5. Proceed with your development and code promotion as normal.

[Git - Revert The Revert]: https://itnext.io/git-revert-the-revert-88b1e66d71d4
[How to restore changes which you’ve reverted from your main git branch]: https://simonplend.com/how-to-restore-changes-which-youve-reverted-from-your-main-git-branch/
[Create Pull Request from a Reverted Git Branch]: https://medium.com/@shanikae/create-pull-request-from-a-reverted-git-branch-27219cadf9b5

---

## Set Up Version Control

You’ll set up version control when creating a new project in Coalesce.

:::info[First Time Using Version Control?]

Take a look at [Step: 1 Setup Version Control][], which contains videos and in-depth steps.

:::

## Before You Begin

Before you begin setting up version control, ensure you have the following prerequisites in place.

### Git Requirements

- Review the [Coalesce Git Requirements][] documentation for detailed information about supported Git providers and configurations.
- Your Git repository must be accessible via HTTPS.
- For new repositories, they must be completely empty (no README files or .gitignore files).

:::warning[`.gitignore`]

Don't add a `.gitignore` to your repo when creating it. It should be completely empty of all files. This is so Coalesce can initiate the repo correctly. After the repo is initiated, you can create a `.gitignore`.

These file paths are protected paths and files and should NOT be in the `.gitignore` and should not be used for anything else.

- data.yml
- locations.yml
- nodes/
- subgraphs/
- packages/
- macros/
- nodeTypes/
- environments/
- jobs/

:::

### Network Requirements

- If you're using allowed IP addresses or a self-hosted instance with a firewall, review the [Network Requirements][] to allow inbound traffic from Coalesce.
- Self-hosted repositories must be accessible on the public internet.

## Get a Personal Access Token (PAT)

Many cloud Git providers have transitioned from traditional username and password authentication for remote repository access to a more secure method known as token authentication or Personal Access Tokens (PAT). Coalesce requires a PAT.

:::danger[Bitbucket Cloud Removing App Passwords]

As of **September 9, 2025** Bitbucket no longer allows the creating of app passwords. New integrations will require an API token. On June 9, 2026 all existing app passwords will stop working.

Learn more in [Bitbucket Cloud enters phase two of app password deprecation](https://www.atlassian.com/blog/bitbucket/bitbucket-cloud-enters-phase-2-of-app-password-deprecation)

:::

- [GitHub - Create a fine grained access token][]. You'll need to select which repositories you want Coalesce to have access to. Use the following permissions:
  - Contents: Read and Write
- [Bitbucket User Token Generation Instructions][]. Use the following permissions:
  - `read:repository:bitbucket`
  - `write:repository:bitbucket`
- [Bitbucket Repository Generation][]: Use the following permissions:
  - `Repositories:Read`
  - `Repositories:Write`
  - Make sure to set the username as `x-token-auth` when adding a version control account in Coalesce for this method only.
- [GitLab Token Generation Instructions][]. Use the following permissions:
  - `read_repository`
  - `write_repository`
- [Azure DevOps Token Generation][]. Use the following permissions:
  - Work Items: Read, write, & manage
  - Code: Read, write, & manage
  - Packaging: Read, write, & manage

:::info[Video Walkthrough]

For video guides for each provider, go through our [step-by-step][] setup instructions.

:::

## Gather Your Version Control Information

You’ll need the following information ready to connect your account.

| Field            | Description                                                                                                                  | Example                                                 |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| Account Nickname | Description of the version control account                                                                                   | GitHub                                                  |
| Git Username     | Your version control username/account name. If using Bitbucket Repo Tokens, make sure to set the username as `x-token-auth`. | jack-bauer                                              |
| Token            | The version control user's authorization token                                                                               | Starts with `ghp`                                       |
| Author Name      | Name of the version control user                                                                                             | Jack Bauer                                              |
| Author Email     | The version control user's email                                                                                             | [jack.bauer@example.com](mailto:jack.bauer@example.com) |

### Git Username in Repo URL

By default, some Git providers include a Git username in the repo URL. Example `https://myusername@bitbucket.org/customer_projects/my-edw.git`.

While URLs like this are currently allowed, any authentication information in them is ignored since Git URLs can be shared across multiple Coalesce users.

Coalesce will instead use the value of Git Account when authenticating.

### Git Author Information

The Author Name and Email in your Git Settings do not have to match the name and email that are registered on your Git Account. No verification is completed on these values. These two values, Author Name and Author Email, are used when making commits to your Git repository to identify the person who made the commit.

The Git username you input in your Git Settings must match your username in your Git provider.

## Connect Git to Coalesce

There are multiple ways to connect Git to Coalesce, depending on where you are in the app.

### New Repos

Follow the instructions for your Version Control Provider. Make sure to skip the step to create a README or add a `.gitignore`. The repo must be empty.

- [GitLab][]
- [GitHub][]
- [Azure Repos][]
- [Bitbucket][]

:::danger[Repository Must Be Empty]

When adding a **new** repository to Coalesce, it must be empty. This includes README and `.gitignores`.

:::

When you make your first commit to Coalesce, it will include Node metadata. Do not delete this data. Review [What Gets Committed][].

### Existing Project

There are two places you can add Git to an existing project, either through the Problem Scanner or the Project Dashboard.

:::danger[What Is an Existing Project?]

An existing project is one that was created with Coalesce. Attempting to add data from other sources will result in an error.

:::

#### Add Git Using the Problem Scanner

Go to your Development Workspace, click the relevant Problem Scanner entry, then follow on-screen prompts to edit URL and select the desired Git account for that Workspace Project.

#### Add Through the Project Dashboard

Next to Create Workspace, select the ellipses and **Configure Git Account**.

[GitHub - Create a fine grained access token]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token
[Bitbucket User Token Generation Instructions]: https://support.atlassian.com/bitbucket-cloud/docs/create-an-api-token/
[GitLab Token Generation Instructions]: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
[Azure DevOps Token Generation]: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate
[Network Requirements]: /docs/setup-your-project/setup-requirements/network-requirements
[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements
[GitLab]: https://docs.gitlab.com/ee/user/project/index.html#create-a-blank-project
[GitHub]: https://docs.github.com/en/repositories/creating-and-managing-repositories/quickstart-for-repositories#create-a-repository
[Azure Repos]: https://learn.microsoft.com/en-us/azure/devops/repos/git/create-new-repo?view=azure-devops
[Bitbucket]: https://support.atlassian.com/bitbucket-cloud/docs/create-a-git-repository/
[What Gets Committed]: /docs/git-integration/what-gets-committed
[step-by-step]: /docs/setup-your-project/setup-version-control
[Step: 1 Setup Version Control]: /docs/setup-your-project/setup-version-control
[Bitbucket Repository Generation]: https://support.atlassian.com/bitbucket-cloud/docs/create-a-repository-access-token/

---

## Expired Version Control Credentials

When version control personal access tokens (PATs), app passwords, or other credentials expire or are revoked, Git operations in Coalesce will fail with authentication errors (401) errors.

## Renewing and Updating Credentials

1. [Regenerate your token][] on the provider’s website.
2. Update the token in Coalesce:
   1. Go to **User Settings > Version Control Accounts**.
   2. Locate the account with the expired token and click **Edit**.
   3. Paste the new token and click **Save**.

## Possible Errors by Platform

Below are common error messages returned by Coalesce when using expired credentials.

### GitHub

```txt
Can't connect to git: Error: Issue cloning repository: Support for password authentication was removed on August 13, 2021. Please see https://docs.github.com/... for recommended authentication methods.
```

```txt
Error pushing to remote: HTTP Error: 401 Support for password authentication was removed on August 13, 2021. Please see https://docs.github.com/... for recommended authentication methods.
```

### Bitbucket

```txt
Can't connect to git: Error: Issue cloning repository: Invalid credentials
```

### GitLab

```txt
Can't connect to git: Error: Issue cloning repository: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://gitlab.com/help/topics/git/troubleshooting_git.md#error-on-git-fetch-http-basic-access-denied
```

### Azure DevOps

```txt
Can't connect to git: Error: Issue cloning repository: HTTP Error: 401 Unauthorized – the Personal Access Token has expired.
```

---

## What's Next?

After updating the token in Coalesce, retry your failed action. If problems persist, verify token scopes and expiration date on the provider's site.

* [Setup Version Control](/docs/setup-your-project/setup-version-control)

---

[Regenerate your token]: /docs/setup-your-project/setup-version-control

---

## Git Sync

Learn how to handle Git sync messages in your Coalesce App.

## Unable To Update Branch Out of Sync

This error is usually caused by changes being made off platform. To prevent sync errors, we recommend periodically updating your local copy by:

* Always checking out the latest version of a branch to work from.  
* Periodically fetching the repo.  
* Doing frequent commits.

You can solve it by either discarding your changes or creating a new branch to keep your work.

### Discard All Changes and Update

This will delete any changes you’ve worked on and update your branch to match remote.

### Create a New Branch To Keep Work

You can create a new branch to hold your changes, then merge the new branch into the synced branch.

Let’s review the example, you’ll be walking through. You made a change locally, in this case, the description of the node `CUSTOMER_VIEW` was changed to `Coalesce App.` The remote has the same node, but the description was changed to `GitHub description`. In the app, the branch is called `main`.

### Step 1: Create a New Branch

If you try to merge these changes, you’ll get the error message: **Unable to Update Branch out of Sync**.

1. While in the modal, click the **Branches** tab. The Unable to Update Branch out of Sync will disappear and you’ll be on the Branches tab.

    

2. Create a new branch from the commit you are working on. In this example, the commit is `GitHub Customer View Node Desc Change`. The new branch in this example is named `save_uncommited_changes`.

    

3. You’ll get a new message that says:  **The branch `save_uncommited_changes` that's attached to this workspace is out of sync. Certain actions are unavailable until the current branch is synced with the latest version**. Go to the Commit tab to sync the branch.
4. Click on the Commit tab. The branch is now `save_uncommited_changes`. You have a new message advising your branch is out of sync. Click **Update Branch**.

    

### Step 2: Commit Your New Branch

1. After creating a new branch to save your changes and then updating, now you’ll commit your changes.  
2. Still on the Commit tab, review your changes and commit. In this example, you can see that the description that was changed locally is being committed to the new branch `save_uncommited_changes`. The commit message is `changed description.`

    

3. Go back to the Branches tab and switch to the `main` branch. Click **Check Out Latest**. This will pull the changes from the remote main to your local main branch. A progress modal shows while checkout runs. Wait until it finishes before you close the modal, use other Git Modal actions, or refresh the browser. Checkout can take time on larger Workspaces.
4. After checkout completes, **refresh your page**. This is important to make sure that the changes from main are applied in the Coalesce App. Your changes are still saved on the new branch you created. Review the changes to be sure. You are looking for the description from main, in this example it’s `GitHub description.`

:::info[Don’t Forget To Refresh]

Refresh only after the checkout progress modal finishes, so the changes from main are applied correctly.

:::

### Step 3: Merge Your Branch

The branch that holds the uncommitted changes needs to be merged into main. In this example it’s `save_commited_changes`.

1. Go back to the **Branches tab** and switch to the `main` branch if you already aren’t on main.  
2. In the dropdown select the `save_uncommited_changes` branch.
3. Select **Merge** next to your commit that saved the change. In this example, it was `changed description`.

    

4. If you only had a few changes, then you’ll be presented with a **Merge Conflict** screen. Accept your changes. In this example, click **Accept Incoming** to accept the description change to the node. We are using **Accept Incoming** because the branch we saved the changes to is incoming to the main branch. If you have too many conflicts, you’ll need to resolve them outside of Coalesce. Review our [Merge Conflicts][] documentation.

    

5. Your saved changes were merged into main. Refresh the app to see your changes.

## Branch Out of Sync

This can happen if changes were made to the branch outside of Coalesce. Update the branch with the remote changes. If it can’t update, you’ll be able to pick the changes to keep using the [Merge Conflicts][] processes.  

## Video Resolving Git Sync Errors in Coalesce

<YouTube id="TLBxCm25Nvw" />

[Merge Conflicts]: /docs/git-integration/solving-git-errors/merge-conflicts

---

## Solving Git Errors

There will be times when you’ll need to fix errors before you can merge your changes. This section will help you understand and resolve frequent issues that may happen.

<DocCardList />

---

## Merge Aborted Errors

## Error: Merge Aborted

When attempting to merge branches in the Coalesce UI, you may see the following error:

`Merge Aborted`

When this happens, your Workspace is reset to its state before the merge.

## Why This Happens

A merge operation may fail in the following situations:

- The branches do not share a common base commit  
- The branches share two or more common commits

## How To Fix Merge Aborted Errors

If you receive a merge aborted error:

1. Complete the merge outside of Coalesce using your normal Git workflow or pull request process.  
2. After finishing the merge in your repository, return to Coalesce.  
3. Open the **Git** modal in your Workspace.  
4. Click **Update Branch** to pull the latest changes into Coalesce.  

## Next Steps

If merge issues continue after following these steps, confirm that your repository history is consistent and that merges are completed successfully in your Git provider before syncing changes back to Coalesce.

---

## Merge Conflicts

Learn what merge conflicts are and ways to solve them.

## What is a Merge Conflict?

A merge conflict in Git occurs when more than one change has been made to the same line of code or section of a file, and Git cannot automatically decide which change to keep and which to discard. This typically happens when at least two people have made changes to the same area in a file.

## Merge Conflict Modal

When a merge conflict happens, you’ll resolve it in this modal.

1. The name of the file the conflict happened in.
2. The option to either accept or reject all changes at once. **Green** is your changes and **Blue** are the changes that are coming from the remote or other branch.
3. Navigation arrows that allow you to move between each merge conflict.
4. The main window where you can edit the file.
5. You can either Abort the change or Merge. You can only merge after all conflicts are resolved.

## What to Know Before Solving Merge Conflicts

- **Local Changes** - These are changes you made locally that haven’t been pushed to your Git repository. They only exist locally for you.
- **Remote Changes** - Remote is the term used for the location of your “main” repository. These changes have been made by others and are present in the remote repository. You encounter these changes when you fetch or pull updates from the remote repository.
- **Keep Current** - These are all changes highlighted in green. These are the changes you made or the current branch you are working on.
- **Accept Incoming** - These changes are highlighted in blue. These are the changes you want to bring in.

## Understanding a Merge Conflict

If a merge conflict occurs, Coalesce will pause the merging process and mark the file where the conflict exists. Understanding conflict markers is crucial to resolving conflicts.

- **Conflict Start Marker** `(<<<<<<<)`: This marker indicates the beginning of the conflicting area and is followed by your local changes.
- **Base Marker** `(=======):` This marker separates your changes from the changes in the remote branch.
- **Conflict End Marker** `(>>>>>>>)`: This marker indicates the end of the conflicting area and is followed by the incoming changes from the remote branch.

```text
deployEnabled: true
<<<<<<< refs/heads/main
description: Main description
=======
description: Lorem Ipsum
>>>>>>> refs/remotes/origin/merge_conflict_three
isMultisource: false
```

In this example, your local changes are `description: Main description` and you want to merge into the remote. But there is a conflict because the remote has `description: Lorem Ipsum`.

To fix this error, you would remove all the other lines except for your changes. You can do this manually by deleting lines or if you are fine with all the changes, you can select Keep Current Changes. Watch out for empty lines.

:::info[Searching for Merge Conflicts]

You can use your operating system's search functionality within the Merge Conflict Window by clicking CTRL + F or Command + F once your cursor is in the text box.

:::

## Large Merge Conflicts

There are times that conflicts are too complex to resolve within Coalesce and require going through the Git provider's platform to resolve before merging the branches/commits. Our system will make you aware if this is the case and warn you accordingly. If you have a use case where you're running into this situation often, please reach out to our Support team.

## General Steps to Resolve Merge Conflicts

1. **Identify the Conflicted Files**: Coalesce will return all the files with conflicts.
2. **Review the Markers**: Use the conflict markers to understand the differences between your local changes and the remote changes. Look at the code between the start and base markers to see your changes, and between the base and end markers to see the remote changes.
3. **Decide on the Changes to Keep**: You need to decide whether to keep your local changes, accept the remote changes, or combine both. Ask your team to make sure you aren’t removing important work.
4. **Edit the File**: Remove the conflict markers and make the necessary edits to the file to incorporate the chosen changes. Ensure that the code's functionality and syntax are correct.
5. **Complete the Merge**: Continue with the merging process. Once all conflicts are resolved and the merge is successful, push your changes to the remote repository to share them with others.

---

## Parsing Errors

Parsing errors occur when there are formatting errors. For example missing commas, extra quotes, and indenting. These usually occur off platform and happen when trying to merge a remote branch into your current local branch.

When a parsing error occurs, the Git Modal will open a Parsing Error screen and jump to the error message. There you can fix the error then merge your changes.

## Read Parsing Errors

In this example, you can see there is a parsing error on line 3, it's highlighted. You can hover over it for more information.

This tells you the error, bad indentation, and the line and character. Around line 3 and column 8 there should be an error. The system is giving you the best estimate where the error is. The line and column is a starting point to find the error.

To solve the error in the example, you can remove the commas. We know it's the commas because YAML files don't have commas in them. Go to your Git provider and remove the commas. Then resync the branch in Coalesce.

You'll need to resync as many times as needed to solve the error.

## Common Parsing Errors

Bad indentation of mapping entry can occur because:

- Missing quotation marks.
- Mismatched quotation marks. Single versus double quotes.
- Incorrect indentation.
- Extra or missing commas.

Block mapping entry or implicit mapping pair can occur because:

- Missing colon.

## Other Parsing Errors

These are some other parsing errors you may encounter and how to solve them.

- `Cannot destructure property 'type' of 'parsedFileData' as it is undefined`. To fix:
  - Delete the file.
  - Restore the file from the previous commit

---

## Repo Format Upgrade

When making a Git commit for the first time in a Workspace, you may see **Repository format upgrade**.

To solve, click the **Upgrade** button. When you click upgrade, there is a new commit made automatically: `Coalesce: Automatic Upgrade Commit`. Your changes are still there and need to be committed.

In this example, you can see the branch and the current commit is the automatic upgrade, with the Environment changes still waiting to be committed.

---

## Validation Errors

Validation errors occur when there are semantic errors. Such an something being an array when it should be an object. These types of errors tend to be introduced outside of Coalesce.

When you get a validation error, the Coalesce App will open a drawer showing a list of where the errors are occurring in your repository. You can resolve them by going to the repository and fixing the errors listed.

## Reading a YAML File

It's important to understand how to read a YAML(.yml) before solving errors.

When reading a YAML file, Coalesce will use dot notation like `operation.dataset` or `operation.metadata.columns.0.acceptedValues.values`, the notation refers to specific paths within the hierarchical structure of the YAML file.

```yaml title="YAML File Example"
fileVersion: 1
id: 45ef876a-91d8-4db3-8bc6-27d96889be53
name: LINEITEM
operation: <------
  database: ""
  dataset: [] <----- operation.dataset
  deployEnabled: true
  description: Lineitem data as defined by TPC-H
  locationName: SAMPLE
  metadata: <---------
    columns: <--------
      - acceptedValues: <-------
          strictMatch: true
          values: {} <-------- operation.metadata.columns.0.acceptedValues.values
        appliedColumnTests: {}
        columnReference:
          columnCounter: e9ea1254-edec-460e-895c-c3439fcf6b51
          stepCounter: 45ef876a-91d8-4db3-8bc6-27d96889be53
        config: {}

```

- `operation.dataset`: This refers to the dataset key under the operation section. In this example, dataset is an empty list (`[]`).
- `operation.metadata.columns.0.acceptedValues.values`: `operation.metadata` refers to the metadata section within operation.
  - `columns.0` indicates the first item in the columns list (index 0).
  - `acceptedValues.values` refers to the values key within `acceptedValues`. In this example, this is an empty dictionary (`{}`).

## Solving Validation Errors

Using the example again, let's solve these errors.

These examples use GitHub, but can be applied to any Git provider.

We can see that there are two errors in the `LINEITEM` node.

1. `operation.dataset Expected string, received array`
2. `operation.metadata.columns.0.acceptedValues.values Expected array, received object`.

The errors tell you where the issue is occurring and the data type it expects.

**For the first error**:`operation.dataset Expected string, received array`. In GitHub, go to the node and review the file. The first part of the error `operation` points to the operation on line 4, then within operation, the second part, dataset, points to line 6. Finally, the message, Expected string, received array, means that the array `[]` should be changed to a string value, or `""`.

**The second error**: `operation.metadata.columns.0.acceptedValues.values Expected array, received object`. In GitHub, go the node and review the file. Starting in `operation` on line 4, then `metadata` on line 10, then `columns` on line 11, then `acceptedValues`, and then `values` on line 14. You can see the value is an empty object, `{}` and it should be changed to an array, `[]`.

## Protected File Paths

In Coalesce, certain file paths are protected and should not be modified directly by users. These files are critical to the system’s operation, and altering them can lead to unexpected behavior or errors. The protected file paths are:

```shell
./environments/*
./jobs/*
./macros/*
./nodeTypes/*
./nodes/*
./subgraphs/*
./data.yml
./locations.yml
./packagesV1.yml
```

Attempting to modify these files directly can result in validation errors or system instability.

### Understanding Protected Files

Protected files contain essential configurations, definitions, and metadata that Coalesce relies on for proper functioning. For example:

- **Nodes and Node Types (`./nodes/*`, `./nodeTypes/*`):** Define the structure and behavior of data transformations.
- **Macros (`./macros/*`):** Contain reusable code snippets or functions.
- **Environments and Locations (`./environments/*`, `./locations.yml`)**: Specify different runtime environments and data locations.
- **Jobs and Subgraphs (`./jobs/*`, `./subgraphs/*`):** Define job sequences and data processing workflows.
- **Configuration Files (`./data.yml`, `./packagesV1.yml`):** Hold global settings and package information.

### Resolving Errors Related to Protected Files

If you encounter validation errors due to modifications of protected file paths:

1. **Do Not Edit Protected Files Directly:** Refrain from making manual changes to the files listed above.
2. **Revert Unauthorized Changes:** If changes were made, revert them to the original state from the repository’s history.
3. **Use the Coalesce Interface:** Make any necessary changes through the Coalesce application, which ensures that modifications comply with system requirements.
4. **Consult Documentation**: Refer to specific sections of the documentation for guidance on how to perform desired actions without altering protected files.
5. **Contact Support:** If you’re unsure how to proceed, reach out to the Coalesce support team for assistance.

---

## Version Control Best Practices

These are some best practices that can make using Git go smoothly for your organization.

## Commits

- Make frequent commits using meaningful commit descriptions.
- Keep your commits simple. They should be for solving a single unit of work or single task. This makes it easier to revert changes.
- Ensure that all members of the development team are aware that a commit is planned, so that all code for the Workspace is in a ‘commit ready' state.
- Consider allocating a single developer the task of performing commits to the Main branch, to avoid any possible conflicts.
- Avoid making breaking commits to your main Git branch.

:::info[Having Trouble Making Commits]

If you cannot commit your changes in a Workspace to Git, contact Coalesce Support for help. Potentially overwriting your live metadata with a previous commit will cause you to lose all recent uncommitted development.

If the error includes **HTTP Error: 413**, the push payload is too large. Split the commit or see [HTTP 413 Request Entity Too Large During Git Commit](/docs/reference/error-library/413).

:::

## Branches

- Use branches for development to keep work in progress away from the main branch.
- Use a branching strategy
- Only deploy to target Environments from a ‘Main' branch, rather than from feature branches.

## Workspace

- Do not have more than one instance of the Git modal open for a single Workspace, at the same time.
- Never develop directly in your main development Workspace.
- Never merge into a Workspace which has uncommitted changes.
- Never change the Git repo settings when having uncommitted changes in ANY Workspace.
- All development Workspaces should be ideally mapped to different schemas (except perhaps source nodes). You can't have a Workspace AND Environment on the same set of Data Platform schemas, unless the Workspace is used as read-only (which cannot be enforced).
- All Environments should be mapped to different schemas, except for source nodes in some cases.

## Resources

- Read our tutorial on setting up Git, merging changes and the deployment process in [DataOps Best Practices with Git and Coalesce][]
- Watch a video on [Using Best Practices in Git][].

[DataOps Best Practices with Git and Coalesce]: https://guides.coalesce.io/dataops-and-git-best-practices/index.html#4
[Using Best Practices in Git]: https://www.youtube.com/watch?v=rCBOuQOTw_k

---

## What Gets Committed

<!-- markdownlint-disable MD046 -->

## What Gets Committed

When you make a commit, we’ll commit the following:  

- Environment Mappings (folder)  
- Subgraphs (folder)  
- Jobs (folder)  
- Macros (folder)  
- Node Types (folder)
- Nodes (folder), as `.yml` for Nodes on V1 node types and `.sql` for Nodes on V2 node types
- Packages (folder)
- data.yml
- locations.yml

## Environment Mappings

Contains the [Storage Mappings][] and environment ID. Each environment will have it's own file.

<details>
  <summary>PROD-ENV.yml (sample)</summary>

    ```yaml
    fileVersion: 1
    id: "3"
    mappingDefinitions:
    SOURCE:
        database: SNOWFLAKE_SAMPLE_DATA
        schema: TPCH_SF10
    TARGET:
        database: COA_TESTING
        schema: QA
    name: Dev Env
    type: Environment
    ```

</details>

## Subgraphs

For how Subgraphs behave in the Build Interface, how they map to Jobs, and what gets stored in Git, see [Using Subgraphs to Build Your Pipeline][].

Each subgraph created will have it's own file. Each node added to a subgraph is a **step**.

<details>
<summary> QA_Subgraph.yml (sample)</summary>

    ```yaml
    fileVersion: 1
    id: "1"
    name: New Subgraph
    steps:
    - 5b8dea41-4e27-4064-80d3-41177e88fd78
    type: Subgraph
    ```

</details>

## Jobs

Each Job created will have it's own file.

<details>
<summary> QA_JOBS.yml (sample)</summary>

    ```yaml
    excludeSelector: ""
    fileVersion: 1
    id: "1"
    includeSelector: "{ location: SAMPLE name: NATION }"
    name: New Job
    type: Job
    ```

</details>

## Macros

Each Macro created will have it's own file.

<details>
<summary> Macros.yml (sample)</summary>

    ```yaml
    fileVersion: 1
    id: "1"
    macroString: |-
    {%- macro even_odd(column) -%}
        CASE WHEN MOD({{ column }}, 2) = 0 THEN 'EVEN' ELSE 'ODD' END
    {%- endmacro %}
    name: macro
    type: Macro
    ```

</details>

## Nodes

Each Node is stored as its own file under the **Nodes** folder. Nodes on V1 node types use `.yml`. Nodes on [Node Type V2][] use `.sql` (SQL-first authoring with annotations). V1 and V2 Nodes can live in the same repository.

<details>
    <summary> Nodes.yml (sample)</summary>
    <Tabs>
        <TabItem value="source" label="Source node example" default>

```yaml
fileVersion: 1
id: fa3ae643-55c3-4cd8-a860-851c13d5b240
name: SUPPLIER
operation:
  database: ""
  dataset: ""
  deployEnabled: true
  description: Supplier data as defined by TPC-H
  locationName: SOURCE
  metadata:
    columns:
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: c2595953-a53d-43b1-bb67-f43c99b11a7e
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: NUMBER(38,0)
        defaultValue: ""
        description: ""
        name: S_SUPPKEY
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 0e83d34f-616e-4ef9-a68f-27335ae9becf
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: VARCHAR(25)
        defaultValue: ""
        description: ""
        name: S_NAME
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 52436bcb-fc31-4085-b45c-b46c3c477e9c
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: VARCHAR(40)
        defaultValue: ""
        description: ""
        name: S_ADDRESS
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: a5a8b8c9-d145-4246-8db3-2afbb70779b7
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: NUMBER(38,0)
        defaultValue: ""
        description: ""
        name: S_NATIONKEY
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 5b1f4abf-f2b0-4eac-ac28-6628b98c84ff
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: VARCHAR(15)
        defaultValue: ""
        description: ""
        name: S_PHONE
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 7a62e69a-da5a-4311-b81c-202e300ad389
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: NUMBER(12,2)
        defaultValue: ""
        description: ""
        name: S_ACCTBAL
        nullable: false
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 200972c1-d133-47bc-8fac-f9ea4d07cb1a
          stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
        config: {}
        dataType: VARCHAR(101)
        defaultValue: ""
        description: ""
        name: S_COMMENT
        nullable: true
        primaryKey: false
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
        transform: ""
        uniqueKey: false
    join:
      joinCondition: FROM {{ ref('SOURCE', 'SUPPLIER') }}
  name: SUPPLIER
  schema: ""
  sqlType: Source
  table: SUPPLIER
  type: sourceInput
  version: 1
type: Node
```

        </TabItem>
        <TabItem value="dimension" label="Dimension Node Example">

```yaml
fileVersion: 1
id: 20ef9752-7e2c-411b-9fa9-10bed025d824
name: DIM_SUPPLIER
operation:
  config:
    postSQL: ""
    preSQL: ""
    testsEnabled: true
  database: ""
  deployEnabled: true
  description: Supplier data as defined by TPC-H
  isMultisource: false
  locationName: SOURCE
  materializationType: table
  metadata:
    appliedNodeTests: []
    columns:
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 228d0101-18f7-43a0-bc75-3132232cc39d
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: NUMBER
        defaultValue: ""
        description: ""
        isSurrogateKey: true
        name: DIM_SUPPLIER_KEY
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 7f74c456-386a-4387-be69-787da7a4f451
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_SUPPKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: c2595953-a53d-43b1-bb67-f43c99b11a7e
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 8f459f3a-7b1f-4647-93b9-fb8d43618330
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: VARCHAR(25)
        description: ""
        name: S_NAME
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 0e83d34f-616e-4ef9-a68f-27335ae9becf
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: b0b93725-2958-4aeb-82cb-615a48166432
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: VARCHAR(40)
        description: ""
        name: S_ADDRESS
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 52436bcb-fc31-4085-b45c-b46c3c477e9c
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: dbee00ec-48ff-4721-982e-53fa403948cc
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_NATIONKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: a5a8b8c9-d145-4246-8db3-2afbb70779b7
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 2c0d7531-52a3-4309-a54d-2a457c6b4ba8
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: VARCHAR(15)
        description: ""
        name: S_PHONE
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 5b1f4abf-f2b0-4eac-ac28-6628b98c84ff
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: af228c16-14c0-4d6d-9796-080af7b56b59
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: NUMBER(12,2)
        description: ""
        name: S_ACCTBAL
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 7a62e69a-da5a-4311-b81c-202e300ad389
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: e272311f-cabf-4156-87fe-56cfe7eb03f5
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: VARCHAR(101)
        description: ""
        name: S_COMMENT
        nullable: true
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 200972c1-d133-47bc-8fac-f9ea4d07cb1a
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: c938b844-02c0-4a81-93d7-e766e8ac3e07
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: NUMBER
        defaultValue: ""
        description: ""
        isSystemVersion: true
        name: SYSTEM_VERSION
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: ee35f28c-12e3-4f24-8adf-661bf8e0cd5e
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: VARCHAR
        defaultValue: ""
        description: ""
        isSystemCurrentFlag: true
        name: SYSTEM_CURRENT_FLAG
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 8aae0912-1027-46e4-9b9d-f66259214724
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemStartDate: true
        name: SYSTEM_START_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 988f7659-976d-4697-ac46-b08583932f40
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemEndDate: true
        name: SYSTEM_END_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 24043104-eef1-4493-a255-0f3cce3cccef
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemCreateDate: true
        name: SYSTEM_CREATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 2dff18f7-8f8f-486e-bc0c-ee2ccfe11d2d
          stepCounter: 20ef9752-7e2c-411b-9fa9-10bed025d824
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemUpdateDate: true
        name: SYSTEM_UPDATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
    cteString: ""
    enabledColumnTestIDs: []
    sourceMapping:
      - aliases: {}
        customSQL:
          customSQL: ""
        dependencies:
          - locationName: SOURCE
            nodeName: SUPPLIER
        join:
          joinCondition: FROM {{ ref('SOURCE', 'SUPPLIER') }} "SUPPLIER"
        name: DIM_SUPPLIER
        noLinkRefs: []
  name: DIM_SUPPLIER
  overrideSQL: false
  schema: ""
  sqlType: Dimension
  type: sql
  version: 1
type: Node
```

        </TabItem>
        <TabItem value="fact" label="Fact Node Example">

```yaml
fileVersion: 1
id: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
name: FCT_SUPPLIER
operation:
  config:
    postSQL: ""
    preSQL: ""
    testsEnabled: true
  database: ""
  deployEnabled: true
  description: Supplier data as defined by TPC-H
  isMultisource: false
  locationName: SOURCE
  materializationType: table
  metadata:
    appliedNodeTests: []
    columns:
      - appliedColumnTests: {}
        columnReference:
          columnCounter: a923c827-230b-4331-9258-4f194d73800a
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_SUPPKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: c2595953-a53d-43b1-bb67-f43c99b11a7e
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 1e307d0b-d4a8-4da3-9d7c-00651d29e028
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: VARCHAR(25)
        description: ""
        name: S_NAME
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 0e83d34f-616e-4ef9-a68f-27335ae9becf
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: b7985532-d01f-4481-9a15-40ff02f8dd9a
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: VARCHAR(40)
        description: ""
        name: S_ADDRESS
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 52436bcb-fc31-4085-b45c-b46c3c477e9c
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: cc210856-e92d-4430-ad11-44c0611f3c16
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_NATIONKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: a5a8b8c9-d145-4246-8db3-2afbb70779b7
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: e345fdb3-9b7d-48b4-a24e-dba7050c935f
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: VARCHAR(15)
        description: ""
        name: S_PHONE
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 5b1f4abf-f2b0-4eac-ac28-6628b98c84ff
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 74a26d8a-7c54-4218-bcb2-40894f32fb3e
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: NUMBER(12,2)
        description: ""
        name: S_ACCTBAL
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 7a62e69a-da5a-4311-b81c-202e300ad389
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: c76320ab-8ac7-4c84-8fe6-4606be66c527
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: VARCHAR(101)
        description: ""
        name: S_COMMENT
        nullable: true
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 200972c1-d133-47bc-8fac-f9ea4d07cb1a
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 92727293-9273-47e4-ab15-cf95cf32459d
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemCreateDate: true
        name: SYSTEM_CREATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: a8cf14ec-7a3e-4629-a056-66e034adb18c
          stepCounter: e20ca58d-ad59-4394-85f5-dc7a9cdeaeeb
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemUpdateDate: true
        name: SYSTEM_UPDATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
    cteString: ""
    enabledColumnTestIDs: []
    sourceMapping:
      - aliases: {}
        customSQL:
          customSQL: ""
        dependencies:
          - locationName: SOURCE
            nodeName: SUPPLIER
        join:
          joinCondition: FROM {{ ref('SOURCE', 'SUPPLIER') }} "SUPPLIER"
        name: FCT_SUPPLIER
        noLinkRefs: []
  name: FCT_SUPPLIER
  overrideSQL: false
  schema: ""
  sqlType: Fact
  type: sql
  version: 1
type: Node
```

        </TabItem>

        <TabItem value="stage" label="Stage Node Example">

```yaml

fileVersion: 1
id: b3fe70c0-5b54-48e5-974d-4699e1fdee69
name: STG_SUPPLIER
operation:
  config:
    insertStrategy: INSERT
    postSQL: ""
    preSQL: ""
    testsEnabled: true
    truncateBefore: true
  database: ""
  deployEnabled: true
  description: Supplier data as defined by TPC-H
  isMultisource: false
  locationName: SOURCE
  materializationType: table
  metadata:
    appliedNodeTests: []
    columns:
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 4ccc3937-4514-4af2-ae21-60d87adb0290
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_SUPPKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: c2595953-a53d-43b1-bb67-f43c99b11a7e
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: a4c44bbf-e44c-4757-ab58-f2211b816acb
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: VARCHAR(25)
        description: ""
        name: S_NAME
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 0e83d34f-616e-4ef9-a68f-27335ae9becf
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 13602029-532d-436a-a648-093389a4eafb
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: VARCHAR(40)
        description: ""
        name: S_ADDRESS
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 52436bcb-fc31-4085-b45c-b46c3c477e9c
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: e2c777ee-d110-4e05-8c58-a8bc3581101c
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_NATIONKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: a5a8b8c9-d145-4246-8db3-2afbb70779b7
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 18f59f49-39df-483d-a902-74b857e13cb8
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: VARCHAR(15)
        description: ""
        name: S_PHONE
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 5b1f4abf-f2b0-4eac-ac28-6628b98c84ff
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 25316571-032e-4021-ab2a-ee5519cce874
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: NUMBER(12,2)
        description: ""
        name: S_ACCTBAL
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 7a62e69a-da5a-4311-b81c-202e300ad389
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: a0523871-16e7-4a36-91a8-5d8b801473c7
          stepCounter: b3fe70c0-5b54-48e5-974d-4699e1fdee69
        config: {}
        dataType: VARCHAR(101)
        description: ""
        name: S_COMMENT
        nullable: true
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 200972c1-d133-47bc-8fac-f9ea4d07cb1a
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
    cteString: ""
    enabledColumnTestIDs: []
    sourceMapping:
      - aliases: {}
        customSQL:
          customSQL: ""
        dependencies:
          - locationName: SOURCE
            nodeName: SUPPLIER
        join:
          joinCondition: FROM {{ ref('SOURCE', 'SUPPLIER') }} "SUPPLIER"
        name: STG_SUPPLIER
        noLinkRefs: []
  name: STG_SUPPLIER
  overrideSQL: false
  schema: ""
  sqlType: Stage
  type: sql
  version: 1
type: Node
```

        </TabItem>

        <TabItem value="persistant" label="Persistent Node Example">

```yaml
fileVersion: 1
id: 26bb2aa9-b030-4378-afe1-5e15117dd217
name: PSTG_SUPPLIER
operation:
  config:
    postSQL: ""
    preSQL: ""
    testsEnabled: true
  database: ""
  deployEnabled: true
  description: Supplier data as defined by TPC-H
  isMultisource: false
  locationName: SOURCE
  materializationType: table
  metadata:
    appliedNodeTests: []
    columns:
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: d3eed09f-b194-4615-be36-737716bd8ef8
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: NUMBER
        defaultValue: ""
        description: ""
        isSurrogateKey: true
        name: PSTG_SUPPLIER_KEY
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 79c970ac-1115-426a-b7fd-152581af389d
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_SUPPKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: c2595953-a53d-43b1-bb67-f43c99b11a7e
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 2efce84e-5aa1-4dab-8571-0f09ec03a451
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: VARCHAR(25)
        description: ""
        name: S_NAME
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 0e83d34f-616e-4ef9-a68f-27335ae9becf
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: c5f4e804-0c02-4fc5-9c23-a93a5ac1b72a
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: VARCHAR(40)
        description: ""
        name: S_ADDRESS
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 52436bcb-fc31-4085-b45c-b46c3c477e9c
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 25142ca7-8f5e-4ddb-9feb-04d447cbf08b
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: NUMBER(38,0)
        description: ""
        name: S_NATIONKEY
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: a5a8b8c9-d145-4246-8db3-2afbb70779b7
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 04691fcb-738e-4b65-a1b8-72385f5a8826
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: VARCHAR(15)
        description: ""
        name: S_PHONE
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 5b1f4abf-f2b0-4eac-ac28-6628b98c84ff
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 655db749-69c4-41a3-a4f6-3a3e0d3d1699
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: NUMBER(12,2)
        description: ""
        name: S_ACCTBAL
        nullable: false
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 7a62e69a-da5a-4311-b81c-202e300ad389
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - appliedColumnTests: {}
        columnReference:
          columnCounter: 555dde70-2cd1-49c4-854e-40d15a389f97
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: VARCHAR(101)
        description: ""
        name: S_COMMENT
        nullable: true
        sourceColumnReferences:
          - columnReferences:
              - columnCounter: 200972c1-d133-47bc-8fac-f9ea4d07cb1a
                stepCounter: fa3ae643-55c3-4cd8-a860-851c13d5b240
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: c1feba77-82e6-4e3b-82ac-21b250e0f244
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: NUMBER
        defaultValue: ""
        description: ""
        isSystemVersion: true
        name: SYSTEM_VERSION
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 35f93100-d008-44dc-974b-d56097d7e984
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: VARCHAR
        defaultValue: ""
        description: ""
        isSystemCurrentFlag: true
        name: SYSTEM_CURRENT_FLAG
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: ""
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 621e084f-ca1c-46da-8ebb-0b40cdc61528
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemStartDate: true
        name: SYSTEM_START_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: d20370e0-0db6-4d47-94d9-5fbd2fa948a5
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemEndDate: true
        name: SYSTEM_END_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 88e333b3-8a59-4626-b121-dd2b23c34393
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemCreateDate: true
        name: SYSTEM_CREATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
      - acceptedValues:
          strictMatch: true
          values: []
        appliedColumnTests: {}
        columnReference:
          columnCounter: 09f296b8-5240-4a41-b76f-dea92b245997
          stepCounter: 26bb2aa9-b030-4378-afe1-5e15117dd217
        config: {}
        dataType: TIMESTAMP
        defaultValue: ""
        description: ""
        isSystemUpdateDate: true
        name: SYSTEM_UPDATE_DATE
        nullable: true
        sourceColumnReferences:
          - columnReferences: []
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
    cteString: ""
    enabledColumnTestIDs: []
    sourceMapping:
      - aliases: {}
        customSQL:
          customSQL: ""
        dependencies:
          - locationName: SOURCE
            nodeName: SUPPLIER
        join:
          joinCondition: FROM {{ ref('SOURCE', 'SUPPLIER') }} "SUPPLIER"
        name: PSTG_SUPPLIER
        noLinkRefs: []
  name: PSTG_SUPPLIER
  overrideSQL: false
  schema: ""
  sqlType: persistentStage
  type: sql
  version: 1
type: Node

```

        </TabItem>
    </Tabs>
</details>

<details>
    <summary>V2 Node Types</summary>

    ```sql
    @id("ea59b792-01e9-42a2-98c8-f760ce8caa76")
    @nodeType("d75202b8-eefb-4aa5-a200-3e42276bb5e4")
    @materializationType("view")
    SELECT
        "O_ORDERKEY" AS "O_ORDERKEY",
        "O_CUSTKEY" AS "O_CUSTKEY",
        "O_ORDERSTATUS" AS "O_ORDERSTATUS",
        "O_TOTALPRICE" AS "O_TOTALPRICE",
        "O_ORDERDATE" AS "O_ORDERDATE",
        "O_ORDERPRIORITY" AS "O_ORDERPRIORITY",
        "O_CLERK" AS "O_CLERK",
        "O_SHIPPRIORITY" AS "O_SHIPPRIORITY",
        "O_COMMENT" AS "O_COMMENT"
    FROM {{ ref('SOURCE', 'ORDERS') }} "ORDERS"
    ```

</details>

## Node Types

Node type definitions are stored under the **Node Types** folder in the repository for both V1 and V2 node types. The expandable examples in this section show V1 structure: definition YAML that includes a **Config** section, plus Jinja templates (Create, Run, Join, and related files). V2 node types use the same kinds of template files, but the type definition sets `version: 2` and follows the V2 spec (no **Config** items in the definition YAML). For how V2 authoring and Node files differ from V1, see [Node Type V2][].

Coalesce-managed built-in templates cannot be changed arbitrarily. If a disallowed change is detected, Coalesce automatically attempts to revert it on the next commit. You can duplicate managed types and use them as the basis for a [custom node][].

<details>
    <summary>Fact-Fact</summary>
    <Tabs>
        <TabItem value="create" label="create.sql.j2" default>

            ```sql
            {% if node.materializationType == 'table' %}
            {{ stage('Create Fact Table') }}

            CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
            (
            {% for col in columns %}
            "{{ col.name }}" {{ col.dataType }}
            {%- if not col.nullable %} NOT NULL
            {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
            {% endif %}
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%}, {% endif %}
            {% endfor %}
            )
            {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

            {% elif node.materializationType == 'view' %}
            {{ stage('Create Fact View') }}

            CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
            (
            {% for col in columns %}
            "{{ col.name }}"
            {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
            {%- if not loop.last -%},{% endif %}
            {% endfor %}
            )
            {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
            AS
            {% for source in sources %}

            {% if loop.first %}SELECT {% endif %}

            {% for col in source.columns %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"
            {%- if not loop.last -%}, {% endif %}
            {% endfor %}
            {{ source.join }}

            {% if not loop.last %} UNION ALL {% endif %}
            {% endfor %}

            {% endif %}
            ```

        </TabItem>
        <TabItem value="definition" label="definition">
            ```yaml
            fileVersion: 1
            id: Fact
            isDisabled: false
            metadata:
            defaultStorageLocation: null
            error: null
            nodeMetadataSpec: |
            capitalized: Fact
            plural: Facts
            short: FCT
            tagColor: '#D9A438'

            config:
            - groupName: Options
            items:
            - type: materializationSelector
            isRequired: true
            options:
            - table
            - view

            - type: businessKeyColumns
            isRequired: false

            - displayName: Enable Tests
            attributeName: testsEnabled
            type: toggleButton
            default: true

            - displayName: Pre-SQL
            attributeName: preSQL
            type: textBox
            syntax: sql
            isRequired: false

            - displayName: Post-SQL
            attributeName: postSQL
            type: textBox
            syntax: sql
            isRequired: false

            systemColumns:
            - displayName: SYSTEM_CREATE_DATE
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemCreateDate

            - displayName: SYSTEM_UPDATE_DATE
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemUpdateDate
            name: Fact
            type: NodeType
            ```
        </TabItem>
        <TabItem value="run" label="run.sql.j2">
            ```sql
            {% for test in node.tests if config.testsEnabled %}
                {% if test.runOrder == 'Before' %}
                    {{ test_stage(test.name, test.continueOnFailure) }}
                    {{ test.templateString }}
                {% endif %}
            {% endfor %}

            {% if node.materializationType == 'table' %}
                {% if config.preSQL %}
                    {{ stage('Pre-SQL') }}
                    {{ config.preSQL }}
                {% endif %}
                
                {% set has_business_key = columns | selectattr("isBusinessKey") | list | length > 0 %}
                
                {% for source in sources %}
                
                    {% if has_business_key %}
                
                        {{ stage('MERGE ' + source.name | string ) }}
                        MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
                        USING (
                            SELECT
                            {% for col in source.columns %}
                                {{ get_source_transform(col) }} AS "{{ col.name }}"
                                {%- if not loop.last %}, {% endif %}
                            {% endfor %}
                            {{ source.join }})
                            AS "SRC"
                        ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            "SRC"."{{ col.name }}" = "TGT"."{{ col.name }}"
                        {% endfor %}
                        WHEN MATCHED
                        {% for col in source.columns if not (   col.isBusinessKey or
                                                                col.isSystemUpdateDate or
                                                                col.isSystemCreateDate) %}
                            {% if loop.first %}
                                AND (
                            {% else %}
                                OR
                            {% endif %}
                            NVL( CAST("SRC"."{{ col.name }}" as STRING), '**NULL**') <> NVL( CAST("TGT"."{{ col.name }}" as STRING), '**NULL**')
                            {% if loop.last %}
                                )
                            {% endif %}
                        {% endfor %}
                        THEN UPDATE SET
                        {%- for col in source.columns if not (col.isBusinessKey or col.isSystemCreateDate) %}
                                "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor %}
                        WHEN NOT MATCHED THEN
                        INSERT (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )
                        VALUES (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )
                
                    {% else %}
                
                        {{ stage('Insert ' + source.name | string ) }}
                
                            INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                            (
                                {% for col in source.columns %}
                                    "{{ col.name }}"
                                    {%- if not loop.last -%},{% endif %}
                                {% endfor %}
                            )
                
                            SELECT
                            {% for col in source.columns %}
                                {{ get_source_transform(col) }} AS "{{ col.name }}"
                                {%- if not loop.last -%}, {% endif %}
                            {% endfor %}
                            {{ source.join }}
                    {% endif %}
                {% endfor %}
                {% if config.postSQL %}
                    {{ stage('Post-SQL') }}
                    {{ config.postSQL }}
                {% endif %}
            {% endif %}

            {% if config.testsEnabled %}
                {% for test in node.tests %}
                    {% if test.runOrder == 'After' %}
                        {{ test_stage(test.name, test.continueOnFailure) }}
                        {{ test.templateString }}
                    {% endif %}
                {% endfor %}

                {% for column in columns %}
                    {% for test in column.tests %}
                        {{ test_stage(column.name + ": " + test.name) }}
                        {{ test.templateString }}
                    {% endfor %}
                {% endfor %}
            {% endif %}
            ```
            

        </TabItem>
    </Tabs>
</details>

<details>
    <summary>Dimension-Dimension</summary>
    <Tabs>
        <TabItem value="create" label="create.sql.j2" default>

            ```sql
            {% if node.materializationType == 'table' %}
                {{ stage('Create Dimension Table') }}

                CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}" {{ col.dataType }}
                        {% if col.isSurrogateKey %}
                            identity
                        {% endif %}
                        {%- if not col.nullable %} NOT NULL
                            {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
                        {% endif %}
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

            {% elif node.materializationType == 'view' %}
                {{ stage('Create Dimension View') }}

                CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%},{% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
                AS
                {% for source in sources %}

                    {% if loop.first %}SELECT {% endif %}

                    {% for col in source.columns %}
                        {% if col.isSurrogateKey or col.isSystemUpdateDate or col.isSystemCreateDate %}
                            NULL
                        {% else %}
                            {{ get_source_transform(col) }}
                        {% endif %}
                        AS "{{ col.name }}"
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                    {{ source.join }}

                    {% if not loop.last %} UNION ALL {% endif %}
                {% endfor %}

            {% endif %}
            ```

        </TabItem>
        <TabItem value="definition" label="definition">

        ```yaml
        fileVersion: 1
        id: Dimension
        isDisabled: false
        metadata:
        defaultStorageLocation: null
        error: null
        nodeMetadataSpec: |-
            capitalized: Dimension
            short: DIM
            tagColor: '#1E339A'
            plural: Dimensions

            config:
            - groupName: Options
            items:
            - type: materializationSelector
                isRequired: true
                default: table
                options:
                - table
                - view

            - type: multisourceToggle
                enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}"

            - type: businessKeyColumns
                isRequired: true

            - type: changeTrackingColumns
                isRequired: false

            - type: overrideSQLToggle
                enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"

            - displayName: Enable Tests
                attributeName: testsEnabled
                type: toggleButton
                default: true
                
            - displayName: Pre-SQL
                attributeName: preSQL
                type: textBox
                syntax: sql
                isRequired: false

            - displayName: Post-SQL
                attributeName: postSQL
                type: textBox
                syntax: sql
                isRequired: false

            systemColumns:
            - displayName: '{{NODE_NAME}}_KEY'
            transform: ''
            dataType: NUMBER
            placement: beginning
            attributeName: isSurrogateKey

            - displayName: SYSTEM_VERSION
            transform: ''
            dataType: NUMBER
            placement: end
            attributeName: isSystemVersion

            - displayName: SYSTEM_CURRENT_FLAG
            transform: ''
            dataType: VARCHAR
            placement: end
            attributeName: isSystemCurrentFlag

            - displayName: SYSTEM_START_DATE
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemStartDate

            - displayName: SYSTEM_END_DATE
            transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemEndDate

            - displayName: SYSTEM_CREATE_DATE
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemCreateDate

            - displayName: SYSTEM_UPDATE_DATE
            transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
            dataType: TIMESTAMP
            placement: end
            attributeName: isSystemUpdateDate
        name: Dimension
        type: NodeType
        ```

        </TabItem>
        <TabItem value="run" label="run.sql.j2">

            ```sql
            {% set is_type_2 = columns | selectattr("isChangeTracking") | list | length > 0 %}

                {% for test in node.tests if config.testsEnabled %}
                    {% if test.runOrder == 'Before' %}
                        {{ test_stage(test.name, test.continueOnFailure) }}
                        {{ test.templateString }}
                    {% endif %}
                {% endfor %}

            {% if node.materializationType == 'table' %}

                {% if config.preSQL %}   
                    {{ stage('Pre-SQL') }}
                    {{ config.preSQL }}
                {% endif %}
                
                {% if is_type_2 %}

                    {% for source in sources %}
                        {{ stage('MERGE ' + source.name | string) }}
                        MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
                        USING (
                        /* New Rows That Don't Exist */
                        SELECT
                        {% for col in source.columns if not col.isSurrogateKey %}
                            {% if col.isSystemVersion %}
                                1
                            {% elif col.isSystemCurrentFlag %}
                                'Y'
                            {% else %}
                            {{ get_source_transform(col) }}
                            {% endif %}
                            AS "{{ col.name }}",
                        {% endfor %}
                            'INSERT_INITAL_VERSION_ROWS' AS "DML_OPERATION"
                        {{ source.join }}
                        LEFT JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                                {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                        {% endfor %}
                        WHERE
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            "DIM"."{{ col.name }}" IS NULL
                        {% endfor %}
                        UNION ALL
                        /* New Row Needing To Be Inserted Due To Type-2 Column Changes */
                        SELECT
                        {% for col in source.columns if not col.isSurrogateKey %}
                            {% if col.isSystemVersion %}
                                "DIM"."{{ col.name }}" + 1
                            {% elif col.isSystemCurrentFlag %}
                                'Y'
                            {% else %}
                            {{ get_source_transform(col) }}
                            {% endif %}
                            AS "{{ col.name }}",
                        {% endfor %}
                            'INSERT_NEW_VERSION_ROWS' AS "DML_OPERATION"
                        {{ source.join }}
                        INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                        {% endfor %}
                        WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                        {% for col in source.columns if (col.isChangeTracking == true) %}
                            {% if loop.first %}
                                AND (
                            {% else %}
                                OR
                            {% endif %}
                            ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                            {% if loop.last %}
                                )
                            {% endif %}
                        {% endfor %}
                        UNION ALL
                        /* Rows Needing To Be Expired Due To Type-2 Column Changes
                        In this case, only two columns are merged (version and end date) */
                        SELECT
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            {% if col.isSystemEndDate %}
                                DATEADD(MILLISECONDS, -1, CAST(CURRENT_TIMESTAMP AS TIMESTAMP))
                            {% elif col.isSystemCurrentFlag %}
                                'N'
                            {% else %}
                                "DIM"."{{ col.name }}"
                            {% endif %}
                            AS "{{ col.name }}",
                        {% endfor -%}
                            'update_expired_version_rows' AS "DML_OPERATION"
                        {{ source.join }}
                        INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                        {% endfor %}
                        WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                        {% for col in source.columns if (col.isChangeTracking == true) %}
                            {% if loop.first %}
                                AND (
                            {% else %}
                                OR
                            {% endif %}
                            ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                            {% if loop.last %}
                                )
                            {% endif %}
                        {% endfor %}
                        {# The if-block below avoids unnecessary updates when no type 2 column changes are present #}
                        {% if source.columns 
                            | rejectattr('isSurrogateKey')
                            | rejectattr('isBusinessKey')
                            | rejectattr('isChangeTracking')
                            | rejectattr('isSystemVersion')
                            | rejectattr('isSystemCurrentFlag')
                            | rejectattr('isSystemStartDate')
                            | rejectattr('isSystemEndDate')
                            | rejectattr('isSystemCreateDate')
                            | rejectattr('isSystemUpdateDate') 
                            | list | length == 0 
                        %}
                            {# Skip Section #}
                        {% else %}
                        UNION ALL
                        /* Rows Needing To Be Updated Due To Changes To Non-Type-2 columns
                        This case merges only when there are changes in non-type-2 column updates, but no changes in type-2 columns*/
                        SELECT
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            {% if col.isSystemVersion or col.isSystemCreateDate or col.isSystemStartDate or col.isSystemEndDate %}
                                "DIM"."{{ col.name }}"
                            {% elif col.isSystemCurrentFlag %}
                                'Y'
                            {% else %}
                                {{ get_source_transform(col) }}
                            {% endif %}
                            AS "{{ col.name }}",
                        {% endfor -%}
                            'UPDATE_NON_TYPE2_ROWS' AS "DML_OPERATION"
                        {{ source.join }}
                        INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                        {% endfor %}
                        WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                        AND (
                        {% for col in source.columns if (col.isChangeTracking) -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                        {% endfor %} )
                        {% for col in source.columns if not (   col.isBusinessKey or
                                                                col.isChangeTracking or
                                                                col.isSurrogateKey or
                                                                col.isSystemVersion or
                                                                col.isSystemCurrentFlag or
                                                                col.isSystemStartDate or
                                                                col.isSystemEndDate or
                                                                col.isSystemUpdateDate or
                                                                col.isSystemCreateDate) -%}
                            {% if loop.first %}
                                AND (
                            {% endif %}
                            {% if not loop.first %}
                                OR
                            {% endif %}
                            NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**')
                            {% if loop.last %}
                                )
                            {% endif %}
                        {% endfor %}
                        {% endif %}
                    ) AS "SRC"
                    ON
                    {% for col in source.columns if col.isBusinessKey -%}
                        {% if not loop.first %}
                            AND
                        {% endif %}
                        "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                    {% endfor %}
                    AND "TGT"."{{ get_value_by_column_attribute("isSystemVersion") }}" = "SRC"."{{ get_value_by_column_attribute("isSystemVersion") }}"
                    WHEN MATCHED THEN UPDATE SET
                    {%- for col in source.columns if not (col.isBusinessKey or col.isSurrogateKey or col.isSystemCreateDate) %}
                        "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                        {% if not loop.last %}, {% endif %}
                    {% endfor -%}
                    WHEN NOT MATCHED THEN INSERT (
                    {%- for col in source.columns if not col.isSurrogateKey %}
                        "{{ col.name }}"
                        {% if not loop.last %}, {% endif %}
                    {% endfor -%}
                    )
                    VALUES (
                    {%- for col in source.columns if not col.isSurrogateKey %}
                        "SRC"."{{ col.name }}"
                        {% if not loop.last %}, {% endif %}
                    {% endfor -%}
                    )

                {% endfor %}

                {% else %}
                    {% for source in sources %}
                        {{ stage('MERGE ' + source.name | string ) }}
                        MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
                        USING (
                            SELECT
                            {% for col in source.columns if not col.isSurrogateKey %}
                                {% if col.isSystemVersion %}
                                    1
                                {% elif col.isSystemCurrentFlag %}
                                    'Y'
                                {% else %}
                                    {{ get_source_transform(col) }}
                                {% endif %}
                                AS "{{ col.name }}"
                                {%- if not loop.last %}, {% endif %}
                            {% endfor %}
                            {{ source.join }})
                            AS "SRC"
                        ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            "SRC"."{{ col.name }}" = "TGT"."{{ col.name }}"
                        {% endfor %}
                        WHEN MATCHED
                        {% for col in source.columns if not (   col.isBusinessKey or
                                                                col.isSurrogateKey or
                                                                col.isSystemVersion or
                                                                col.isSystemCurrentFlag or
                                                                col.isSystemStartDate or
                                                                col.isSystemEndDate or
                                                                col.isSystemUpdateDate or
                                                                col.isSystemCreateDate) %}
                            {% if loop.first %}
                                AND (
                            {% else %}
                                OR
                            {% endif %}
                            NVL( CAST("SRC"."{{ col.name }}" as STRING), '**NULL**') <> NVL( CAST("TGT"."{{ col.name }}" as STRING), '**NULL**')
                            {% if loop.last %}
                                )
                            {% endif %}
                        {% endfor %}
                        THEN UPDATE SET
                        {%- for col in source.columns if not (  col.isBusinessKey or
                                                                col.isSurrogateKey or
                                                                col.isSystemVersion or
                                                                col.isSystemCurrentFlag or
                                                                col.isSystemStartDate or
                                                                col.isSystemEndDate or
                                                                col.isSystemCreateDate) %}
                                "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor %}
                        WHEN NOT MATCHED THEN
                        INSERT (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )
                        VALUES (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )
                    {% endfor %}
                {% endif %}
                
                {% if config.postSQL %}   
                    {{ stage('Post-SQL') }}
                    {{ config.postSQL }}
                {% endif %}
            {% endif %}

            {% if config.testsEnabled %}
                {% for test in node.tests %}
                    {% if test.runOrder == 'After' %}
                        {{ test_stage(test.name, test.continueOnFailure) }}
                        {{ test.templateString }}
                    {% endif %}
                {% endfor %}

                {% for column in columns %}
                    {% for test in column.tests %}
                        {{ test_stage(column.name + ": " + test.name) }}
                        {{ test.templateString }}
                    {% endfor %}
                {% endfor %}
            {% endif %}
            ```

        </TabItem>
    </Tabs>
</details>

<details>
    <summary>PersistentStage-persistentStage</summary>
    <Tabs>
        <TabItem value="create" label="create.sql.j2" default>

            ```sql
            {% if node.materializationType == 'table' %}
                {{ stage('Create Persistent Stage Table') }}

                CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}" {{ col.dataType }}
                        {% if col.isSurrogateKey %}
                            identity
                        {% endif %}
                        {%- if not col.nullable %} NOT NULL
                            {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
                        {% endif %}
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

            {% elif node.materializationType == 'view' %}
                {{ stage('Create Persistent Stage View') }}

                CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%},{% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
                AS
                {% for source in sources %}

                    {% if not loop.last %} UNION ALL {% endif %}
                {% endfor %}

                    {% for col in source.columns %}
                        {% if col.isSurrogateKey or col.isSystemUpdateDate or col.isSystemCreateDate %}
                            NULL
                        {% else %}
                            {{ get_source_transform(col) }}
                        {% endif %}
                        AS "{{ col.name }}"
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                    {{ source.join }}

            {% endif %}
            ```
        </TabItem>
        <TabItem value="definition" label="definition">

            ```yaml
            fileVersion: 1
            id: persistentStage
            isDisabled: false
            metadata:
            defaultStorageLocation: null
            error: null
            nodeMetadataSpec: |
                capitalized: Persistent Stage
                short: PSTG
                plural: Persistent Stages
                tagColor: '#29B2DB'
                    
                config:
                - groupName: Options
                items:
                - type: materializationSelector
                    isRequired: true
                    default: table
                    options:
                    - table
                    - view
                
                - type: businessKeyColumns
                    isRequired: false

                - type: multisourceToggle
                    enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}"

                - type: overrideSQLToggle
                    enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"

                - displayName: Enable Tests
                    attributeName: testsEnabled
                    type: toggleButton
                    default: true
                    
                - displayName: Pre-SQL
                    attributeName: preSQL
                    type: textBox
                    syntax: sql
                    isRequired: false

                - displayName: Post-SQL
                    attributeName: postSQL
                    type: textBox
                    syntax: sql
                    isRequired: false

                systemColumns:

                - displayName: '{{NODE_NAME}}_KEY'
                transform: ''
                dataType: NUMBER
                placement: beginning
                attributeName: isSurrogateKey

                - displayName: SYSTEM_VERSION
                transform: ''
                dataType: NUMBER
                placement: end
                attributeName: isSystemVersion

                - displayName: SYSTEM_CURRENT_FLAG
                transform: ''
                dataType: VARCHAR
                placement: end
                attributeName: isSystemCurrentFlag

                - displayName: SYSTEM_START_DATE
                transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
                dataType: TIMESTAMP
                placement: end
                attributeName: isSystemStartDate

                - displayName: SYSTEM_END_DATE
                transform: CAST('2999-12-31 00:00:00' AS TIMESTAMP)
                dataType: TIMESTAMP
                placement: end
                attributeName: isSystemEndDate

                - displayName: SYSTEM_CREATE_DATE
                transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
                dataType: TIMESTAMP
                placement: end
                attributeName: isSystemCreateDate

                - displayName: SYSTEM_UPDATE_DATE
                transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
                dataType: TIMESTAMP
                placement: end
                attributeName: isSystemUpdateDate
            name: Persistent Stage
            type: NodeType
            ```
        </TabItem>
        <TabItem value="run" label="run.sql.j2">

            ```sql
                {% set has_business_key = columns | selectattr("isBusinessKey") | list | length > 0 %}
                {% set is_type_2 = columns | selectattr("isChangeTracking") | list | length > 0 %}

                    {% for test in node.tests if config.testsEnabled %}
                        {% if test.runOrder == 'Before' %}
                            {{ test_stage(test.name, test.continueOnFailure) }}
                            {{ test.templateString }}
                        {% endif %}
                    {% endfor %}

                {% if node.materializationType == 'table' %}
                    {% if config.preSQL %}
                        {{ stage('Pre-SQL') }}
                        {{ config.preSQL }}
                    {% endif %}
                    
                    {% if has_business_key and is_type_2 %}

                        {% for source in sources %}
                            {{ stage('MERGE ' + source.name | string) }}
                            MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
                            USING (
                            /* New Rows That Don't Exist */
                            SELECT
                            {% for col in source.columns if not col.isSurrogateKey %}
                                {% if col.isSystemVersion %}
                                    1
                                {% elif col.isSystemCurrentFlag %}
                                    'Y'
                                {% else %}
                                {{ get_source_transform(col) }}
                                {% endif %}
                                AS "{{ col.name }}",
                            {% endfor %}
                                'INSERT_INITAL_VERSION_ROWS' AS "DML_OPERATION"
                            {{ source.join }}
                            LEFT JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                    {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                            {% endfor %}
                            WHERE
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                "DIM"."{{ col.name }}" IS NULL
                            {% endfor %}
                            UNION ALL
                            /* New Row Needing To Be Inserted Due To Type-2 Column Changes */
                            SELECT
                            {% for col in source.columns if not col.isSurrogateKey %}
                                {% if col.isSystemVersion %}
                                    "DIM"."{{ col.name }}" + 1
                                {% elif col.isSystemCurrentFlag %}
                                    'Y'
                                {% else %}
                                {{ get_source_transform(col) }}
                                {% endif %}
                                AS "{{ col.name }}",
                            {% endfor %}
                                'INSERT_NEW_VERSION_ROWS' AS "DML_OPERATION"
                            {{ source.join }}
                            INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                            {% endfor %}
                            WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                            {% for col in source.columns if (col.isChangeTracking == true) %}
                                {% if loop.first %}
                                    AND (
                                {% else %}
                                    OR
                                {% endif %}
                                ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                                {% if loop.last %}
                                    )
                                {% endif %}
                            {% endfor %}
                            UNION ALL
                            /* Rows Needing To Be Expired Due To Type-2 Column Changes
                            In this case, only two columns are merged (version and end date) */
                            SELECT
                            {%- for col in source.columns if not col.isSurrogateKey %}
                                {% if col.isSystemEndDate %}
                                    DATEADD(MILLISECONDS, -1, CAST(CURRENT_TIMESTAMP AS TIMESTAMP))
                                {% elif col.isSystemCurrentFlag %}
                                    'N'
                                {% else %}
                                    "DIM"."{{ col.name }}"
                                {% endif %}
                                AS "{{ col.name }}",
                            {% endfor -%}
                                'update_expired_version_rows' AS "DML_OPERATION"
                            {{ source.join }}
                            INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                            {% endfor %}
                            WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                            {% for col in source.columns if (col.isChangeTracking == true) %}
                                {% if loop.first %}
                                    AND (
                                {% else %}
                                    OR
                                {% endif %}
                                ( NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**') )
                                {% if loop.last %}
                                    )
                                {% endif %}
                            {% endfor %}
                            UNION ALL
                            /* Rows Needing To Be Updated Due To Changes To Non-Type-2 source.columns
                            This case merges only when there are changes in non-type-2 column updates, but no changes in type-2 columns*/
                            SELECT
                            {%- for col in source.columns if not col.isSurrogateKey %}
                                {% if col.isSystemVersion or col.isSystemCreateDate or col.isSystemStartDate or col.isSystemEndDate %}
                                    "DIM"."{{ col.name }}"
                                {% elif col.isSystemCurrentFlag %}
                                    'Y'
                                {% else %}
                                    {{ get_source_transform(col) }}
                                {% endif %}
                                AS "{{ col.name }}",
                            {% endfor -%}
                                'UPDATE_NON_TYPE2_ROWS' AS "DML_OPERATION"
                            {{ source.join }}
                            INNER JOIN {{ ref_no_link(node.location.name, node.name) }} "DIM" ON
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                            {% endfor %}
                            WHERE "DIM"."{{ get_value_by_column_attribute("isSystemCurrentFlag") }}" = 'Y'
                            AND (
                            {% for col in source.columns if (col.isChangeTracking) -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                {{ get_source_transform(col) }} = "DIM"."{{ col.name }}"
                            {% endfor %} )
                            {% for col in source.columns if not (   col.isBusinessKey or
                                                                    col.isChangeTracking or
                                                                    col.isSurrogateKey or
                                                                    col.isSystemVersion or
                                                                    col.isSystemCurrentFlag or
                                                                    col.isSystemStartDate or
                                                                    col.isSystemEndDate or
                                                                    col.isSystemUpdateDate or
                                                                    col.isSystemCreateDate) -%}
                                {% if loop.first %}
                                    AND (
                                {% endif %}
                                {% if not loop.first %}
                                    OR
                                {% endif %}
                                NVL( CAST({{ get_source_transform(col) }} as STRING), '**NULL**') <> NVL( CAST("DIM"."{{ col.name }}" as STRING), '**NULL**')
                                {% if loop.last %}
                                    )
                                {% endif %}
                            {% endfor %}
                        ) AS "SRC"
                        ON
                        {% for col in source.columns if col.isBusinessKey -%}
                            {% if not loop.first %}
                                AND
                            {% endif %}
                            "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                        {% endfor %}
                        AND "TGT"."{{ get_value_by_column_attribute("isSystemVersion") }}" = "SRC"."{{ get_value_by_column_attribute("isSystemVersion") }}"
                        WHEN MATCHED THEN UPDATE SET
                        {%- for col in source.columns if not (col.isBusinessKey or col.isSurrogateKey or col.isSystemCreateDate) %}
                            "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        WHEN NOT MATCHED THEN INSERT (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )
                        VALUES (
                        {%- for col in source.columns if not col.isSurrogateKey %}
                            "SRC"."{{ col.name }}"
                            {% if not loop.last %}, {% endif %}
                        {% endfor -%}
                        )

                    {% endfor %}

                    {% elif has_business_key and not is_type_2 %}
                        {% for source in sources %}
                            {{ stage('MERGE ' + source.name | string ) }}
                            MERGE INTO {{ ref_no_link(node.location.name, node.name) }} "TGT"
                            USING (
                                SELECT
                                {% for col in source.columns if not col.isSurrogateKey %}
                                    {% if col.isSystemVersion %}
                                        1
                                    {% elif col.isSystemCurrentFlag %}
                                        'Y'
                                    {% else %}
                                        {{ get_source_transform(col) }}
                                    {% endif %}
                                    AS "{{ col.name }}"
                                    {%- if not loop.last %}, {% endif %}
                                {% endfor %}
                                {{ source.join }})
                                AS "SRC"
                            ON
                            {% for col in source.columns if col.isBusinessKey -%}
                                {% if not loop.first %}
                                    AND
                                {% endif %}
                                "SRC"."{{ col.name }}" = "TGT"."{{ col.name }}"
                            {% endfor %}
                            WHEN MATCHED
                            {% for col in source.columns if not (   col.isBusinessKey or
                                                                    col.isSurrogateKey or
                                                                    col.isSystemVersion or
                                                                    col.isSystemCurrentFlag or
                                                                    col.isSystemStartDate or
                                                                    col.isSystemEndDate or
                                                                    col.isSystemUpdateDate or
                                                                    col.isSystemCreateDate) %}
                                {% if loop.first %}
                                    AND (
                                {% else %}
                                    OR
                                {% endif %}
                                NVL( CAST("SRC"."{{ col.name }}" as STRING), '**NULL**') <> NVL( CAST("TGT"."{{ col.name }}" as STRING), '**NULL**')
                                {% if loop.last %}
                                    )
                                {% endif %}
                            {% endfor %}
                            THEN UPDATE SET
                            {%- for col in source.columns if not (  col.isBusinessKey or
                                                                    col.isSurrogateKey or
                                                                    col.isSystemVersion or
                                                                    col.isSystemCurrentFlag or
                                                                    col.isSystemStartDate or
                                                                    col.isSystemEndDate or
                                                                    col.isSystemCreateDate) %}
                                    "TGT"."{{ col.name }}" = "SRC"."{{ col.name }}"
                                {% if not loop.last %}, {% endif %}
                            {% endfor %}
                            WHEN NOT MATCHED THEN
                            INSERT (
                            {%- for col in source.columns if not col.isSurrogateKey %}
                                "{{ col.name }}"
                                {% if not loop.last %}, {% endif %}
                            {% endfor -%}
                            )
                            VALUES (
                            {%- for col in source.columns if not col.isSurrogateKey %}
                                "SRC"."{{ col.name }}"
                                {% if not loop.last %}, {% endif %}
                            {% endfor -%}
                            )
                        {% endfor %}
                    {% else %}
                        {% for source in sources %}
                            {{ stage('Insert ' + source.name | string ) }}
                            INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                            (
                                {% for col in source.columns if not col.isSurrogateKey %}
                                    "{{ col.name }}"
                                    {%- if not loop.last -%},{% endif %}
                                {% endfor %}
                            )

                            SELECT
                            {% for col in source.columns if not col.isSurrogateKey %}

                                {% if col.isSystemVersion %}
                                    1
                                {% elif col.isSystemCurrentFlag %}
                                    'Y'
                                {% else %}
                                    {{ get_source_transform(col) }}
                                {% endif %}
                                AS "{{ col.name }}"
                                {%- if not loop.last -%}, {% endif %}
                                
                            {% endfor %}
                            {{ source.join }}
                        {% endfor %}            
                    {% endif %}
                    
                    {% if config.postSQL %}
                        {{ stage('Post-SQL') }}
                        {{ config.postSQL }}
                    {% endif %}
                    
                {% endif %}

                {% if config.testsEnabled %}
                    {% for test in node.tests %}
                        {% if test.runOrder == 'After' %}
                            {{ test_stage(test.name, test.continueOnFailure) }}
                            {{ test.templateString }}
                        {% endif %}
                    {% endfor %}

                    {% for column in columns %}
                        {% for test in column.tests %}
                            {{ test_stage(column.name + ": " + test.name) }}
                            {{ test.templateString }}
                        {% endfor %}
                    {% endfor %}
                {% endif %}
                ```
        </TabItem>
    </Tabs>
</details>

<details>
    <summary>Stage-Stage</summary>
    <Tabs>
        <TabItem value="create" label="create.sql.j2" default>
<!-- markdownlint-disable-next-line -->
            ```sql
            {% if node.override.create.enabled %}

                {{ node.override.create.script }}

            {% elif node.materializationType == 'table' %}
                {{ stage('Create Stage Table') }}

                CREATE OR REPLACE TABLE {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}" {{ col.dataType }}
                        {%- if not col.nullable %} NOT NULL
                            {%- if col.defaultValue | length > 0 %} DEFAULT {{ col.defaultValue }}{% endif %}
                        {% endif %}
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}

            {% elif node.materializationType == 'view' %}
                {{ stage('Create Stage View') }}

                CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
                AS
                {% for source in sources %}
                    SELECT
                    {% for col in source.columns %}
                        {{ get_source_transform(col) }} AS "{{ col.name }}"
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}

                    {{ source.join }}

                    {% if not loop.last %}
                        {% if config.insertStrategy in ['UNION', 'UNION ALL'] %}
                            {{ config.insertStrategy }}
                        {% else %}
                            UNION
                        {% endif %}
                    {% endif %}
                {% endfor %}

            {% endif %}
            ```
        </TabItem>
        <TabItem value="definition" label="definition">
            ```yaml
            fileVersion: 1
            id: Stage
            isDisabled: false
            metadata:
            defaultStorageLocation: null
            error: null
            nodeMetadataSpec: |
                capitalized: Stage
                short: STG
                plural: Stages
                tagColor: '#2EB67D'

                config:
                - groupName: Options
                items:
                - type: materializationSelector
                    default: table
                    options:
                    - table
                    - view
                    isRequired: true

                - type: multisourceToggle
                    enableIf: "{% if node.materializationType == 'table' %} true {% else %} false {% endif %}"

                - type: overrideSQLToggle
                    enableIf: "{% if node.materializationType == 'view' %} true {% else %} false {% endif %}"

                - displayName: Multi Source Strategy
                    attributeName: insertStrategy
                    type: dropdownSelector
                    default: INSERT
                    options:
                    - "INSERT"
                    - "UNION"
                    - "UNION ALL"
                    isRequired: true
                    enableIf: "{% if node.isMultisource %} true {% else %} false {% endif %}"

                - displayName: Truncate Before
                    attributeName: truncateBefore
                    type: toggleButton
                    default: true

                - displayName: Enable Tests
                    attributeName: testsEnabled
                    type: toggleButton
                    default: true

                - displayName: Pre-SQL
                    attributeName: preSQL
                    type: textBox
                    syntax: sql
                    isRequired: false

                - displayName: Post-SQL
                    attributeName: postSQL
                    type: textBox
                    syntax: sql
                    isRequired: false
            name: Stage
            type: NodeType
            ```

        </TabItem>
        <TabItem value="run" label="run.sql.j2">
            ```sql
            {% for test in node.tests if config.testsEnabled %}
                {% if test.runOrder == 'Before' %}
                    {{ test_stage(test.name, test.continueOnFailure) }}
                    {{ test.templateString }}
                {% endif %}
            {% endfor %}

            {% if node.materializationType == 'table' %}
                {% if config.preSQL %}
                    {{ stage('Pre-SQL') }}
                    {{ config.preSQL }}
                {% endif %}
                
                
                
                    {% if config.truncateBefore %}
                
                        {{ stage('Truncate Stage Table') }}
                        TRUNCATE IF EXISTS {{ ref_no_link(node.location.name, node.name) }}
                
                    {% endif %}
                
                
                    {% if config.insertStrategy in ['UNION', 'UNION ALL'] %}
                        {{ stage( config.insertStrategy + ' Sources' | string ) }}
                        INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                            (
                                {% for col in columns %}
                                    "{{ col.name }}"
                                    {%- if not loop.last -%},{% endif %}
                                {% endfor %}
                            )
                    {% endif %}
                
                
                    {% for source in sources %}
                
                        {% if config.insertStrategy == 'INSERT' %}
                            {{ stage('Insert ' + source.name | string ) }}
                
                            INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
                            (
                                {% for col in source.columns %}
                                    "{{ col.name }}"
                                    {%- if not loop.last -%},{% endif %}
                                {% endfor %}
                            )
                        {% endif %}
                
                        SELECT
                        {% for col in source.columns %}
                            {{ get_source_transform(col) }} AS "{{ col.name }}"
                            {%- if not loop.last -%}, {% endif %}
                        {% endfor %}
                
                        {{ source.join }}
                
                        {% if config.insertStrategy in ['UNION', 'UNION ALL'] and not loop.last %}
                            {{config.insertStrategy}}
                        {% endif %}
                
                    {% endfor %}
                
                {% if config.postSQL %}
                    {{ stage('Post-SQL') }}
                    {{ config.postSQL }}
                {% endif %}
            {% endif %}

            {% if config.testsEnabled %}
                {% for test in node.tests %}
                    {% if test.runOrder == 'After' %}
                        {{ test_stage(test.name, test.continueOnFailure) }}
                        {{ test.templateString }}
                    {% endif %}
                {% endfor %}

                {% for column in columns %}
                    {% for test in column.tests %}
                        {{ test_stage(column.name + ": " + test.name) }}
                        {{ test.templateString }}
                    {% endfor %}
                {% endfor %}
            {% endif %}
            ```
        </TabItem>
    </Tabs>
</details>

<details>
    <summary>View-View</summary>
    <Tabs>
        <TabItem value="create" label="create.sql.j2" default>

            ```sql
            {% if node.override.create.enabled %}

                {{ node.override.create.script }}

            {% else %}
                {{ stage('Create View') }}
                CREATE OR REPLACE VIEW {{ ref_no_link(node.location.name, node.name) }}
                (
                    {% for col in columns %}
                        "{{ col.name }}"
                        {%- if col.description | length > 0 %} COMMENT '{{ col.description | escape }}'{% endif %}
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}
                )
                {%- if node.description | length > 0 %} COMMENT = '{{ node.description | escape }}'{% endif %}
                AS
                {% for source in sources %}
                    SELECT {% if config.selectDistinct %} DISTINCT {% endif %}
                    {% for col in source.columns %}
                        {{ get_source_transform(col) }} AS "{{ col.name }}"
                        {%- if not loop.last -%}, {% endif %}
                    {% endfor %}

                    {{ source.join }}

                    {% if not loop.last %}
                        {% if config.insertStrategy in ['UNION', 'UNION ALL'] %}
                            {{ config.insertStrategy }}
                        {% else %}
                            UNION
                        {% endif %}
                    {% endif %}
                {% endfor %}
            {% endif %}
            ```

        </TabItem>
        <TabItem value="definition" label="definition">

            ```yaml
            fileVersion: 1
            id: View
            isDisabled: true
            metadata:
            defaultStorageLocation: null
            error: null
            nodeMetadataSpec: |
                capitalized: View
                short: V
                tagColor: '#C4C4C4'
                isDisabled: true
                plural: Views

                config:
                - groupName: Options
                    items: 
                    - type: materializationSelector
                    options:
                        - view
                    default: view
                    isRequired: true

                    - type: toggleButton
                    attributeName: selectDistinct
                    displayName: Distinct

                    - type: multisourceToggle

                    - type: overrideSQLToggle

                    - displayName: Multi Source Strategy
                    attributeName: insertStrategy
                    type: dropdownSelector
                    default: UNION
                    options:
                    - "UNION"
                    - "UNION ALL" 
                    isRequired: true
                    enableIf: "{% if node.isMultisource %} true {% else %} false {% endif %}"
            name: View
            type: NodeType
            ```

        </TabItem>
        <TabItem value="run" label="run.sql.j2">
            empty

        </TabItem>
    </Tabs>
</details>

## Packages

Package contents can vary. Learn more about [Coalesce Packages][].

<details>
    <summary>Packages Example</summary>

        ```yaml
            config:
            entities:
                nodeTypes:
                "25":
                    defaultStorageLocation: null
                    isDisabled: false
                "35":
                    defaultStorageLocation: null
                    isDisabled: false
                "36":
                    defaultStorageLocation: null
                    isDisabled: false
                "37":
                    defaultStorageLocation: null
                    isDisabled: false
                "38":
                    defaultStorageLocation: null
                    isDisabled: false
            packageVariables: |-
                {%- set demopkg4coalesce = namespace(
                        config = {
                        "ldts_alias": "LDTS",
                        "rsrc_alias": "RSRC", 
                        "ledts_alias": "LEDTS",
                        "stg_alias": "STG", 
                        "snapshot_trigger_column": "IS_ACTIVE",
                        "use_object_name_prefix": TRUE,
                        "sdts_alias": "SDTS",
                        "is_current_col_alias": "IS_CURRENT", 
                        "hash": "MD5", 
                        "hash_datatype": "STRING", 
                        "hash_input_case_sensitive": "TRUE",
                        "hash_passthrough_input_transformations": "TRUE",
                        "beginning_of_all_times": "0001-01-01T00:00:01",
                        "end_of_all_times": "8888-12-31T23:59:59",
                        "timestamp_format": "YYYY-MM-DDTHH24:MI:SS", 
                        "default_unknown_rsrc": "SYSTEM", 
                        "default_error_rsrc": "ERROR", 
                        "rsrc_default_dtype": "STRING", 
                        "stg_default_dtype": "STRING", 
                        "error_value__STRING": "'(error)'", 
                        "error_value_alt__STRING": "'e'", 
                        "unknown_value__STRING": "'(unknown)'",
                        "unknown_value_alt__STRING": "'u'"
                        }
                    ) -%}
            fileVersion: 1
            id: "@coalesce/support-demo-pkg"
            name: secondPackage
            packageID: "@coalesce/support-demo-pkg"
            releaseID: d1ea23de-1b86-478d-916a-d8bc9f06ba41
            type: Package
            ```

</details>

<!-- vale Coalesce.Headings = NO-->

## data.yml

    ```yaml
    defaultStorageMapping: SAMPLE
    fileVersion: 3
    ```

## locations.yml

A list of storage locations and the default location.

    ```yaml
    defaultStorageMapping: SAMPLE
    fileVersion: 1
    locations:
    - HELLO
    - SAMPLE
    - WORK
    ```

[Coalesce Packages]: /docs/marketplace/manage-packages
[Node Type V2]: /docs/build-your-pipeline/v2-node-types
[Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs
[custom node]: /docs/build-your-pipeline/user-defined-nodes/

---

## Build Dynamic Tables in Snowflake with Coalesce

## What are Dynamic Tables?

[Snowflake's Dynamic Tables][] allows you to automatically keep your tables up to date by defining a query that continuously processes data from one or more base tables. The Dynamic Table updates regularly based on a query through [refreshes][]. This automation simplifies your data pipelines and removes the need for manual refreshes.

### Dynamic Tables Package

Coalesce offers a [Dynamic Table Package][] through the Coalesce Marketplace. The package contains three node types:

- Dynamic Table Work
- Dynamic Table Dimension
- Dynamic Table Latest Record Version

#### Dynamic Table Work

Dynamic Table Work is used to deploy single dynamic tables or a DAG (Direct Acyclic Graph).

A few features of the Dynamic Work Table:

- **Downstream**: You can set the table to refresh when other dynamic tables need to refresh or you can set the [Lag specification][].
- **Refresh Mode**: Set the mode to auto, incremental, or full. Decide if you want a full refresh or incremental.
- **Initialize**: Run at creation or at the next scheduled [Refresh][].

#### Dynamic Table Dimension

Dimension tables are a type of dimension in a data warehouse where the data changes slowly over time. They can be used to track things like product details and financial data. This is a Type 2 Slowly Changing Dimension table.

A few features of the Dynamic Table Dimension:

- **Downstream**: You can set the table to refresh when other dynamic tables need to refresh or you can set the [Lag specification][].
- **Refresh Mode**: Set the mode to auto, incremental, or full. Decide if you want a full refresh or incremental.
- **Initialize**: Run at creation or at the next scheduled [Refresh][].
- **Record Versioning**: You can track changes over time using the date and time based on a column. The history of the records are kept.

#### Dynamic Table Latest Record Version

The Dynamic Table Latest Record Version keeps the most recent version of a record. Use this table when you only want the most recent record. Latest record version does this by identifying the most recent record based on a specified date time or timestamp column.

A few features of the Dynamic Table Dimension:

- **Downstream**: You can set the table to refresh when other dynamic tables need to refresh or you can set the [Lag specification][].
- **Refresh Mode**: Set the mode to auto, incremental, or full. Decide if you want a full refresh or incremental.
- **Initialize**: Run at creation or at the next scheduled [Refresh][].
- **Record Versioning**: Unlike the Dynamic Table Dimension, this only keeps the most recent version of the record.

## How to use Dynamic Tables in Coalesce

You’ll learn how to use the Dynamic Tables package, specifically the Dynamic Work Table.

### Before You Begin

- You need a Coalesce account. You can start a free trial from our [Snowflake Marketplace listing][].
- Snowflake account with `ACCOUNTADMIN` privileges. You’ll need the following privileges on the table:
  - USAGE on the table you will use.
  - [CREATE DYNAMIC TABLE][] on the schema
  - SELECT on the existing tables and views that you plan to query
  - USAGE on the warehouse that you plan to use to refresh the table.
- We’ll be using the Snowflake [trial account data][].

## Update Your Parameters

For the Dynamic Tables node to work, you need to update the workspace parameters.

1. Click the **Build Settings > Workspace**. Then click the pencil to edit Workspace.

    

2. Select Parameters and update it to match the warehouse you’re using. In this example, it’s `COMPUTE_WH`.

    ```json title='Dynamic Tables Package Paramter'
    { 
        "targetDynamicTableWarehouse": "COMPUTE_WH" 
    }
    ```

    

## Add Your Data Sources

1. From the Build screen, make sure you are on [Node Graph][]. Select the `+` plus sign. Then **Add Sources**.

    

2. Select the sources you want to add, then **Add Sources**. You can get a preview of each source by selecting them.

    

3. Add the all the Snowflake sample datasets.

## Add the Dynamic Tables Package

1. Go to the [Dynamic Tables package page][] and make note of the package ID. `@coalesce/dynamic-tables`.

    

2. On the Build page, click **Build Settings**.
3. Select **Packages**, then click **Install**.
4. Enter the package ID you copied from the package page.
5. Give the package a unique name to make it easy to identify later.
6. The package will automatically default to the most recent version.

    

7. Go to **Build Settings > Node Types.** To see the three Dynamic Table node types installed. It shows the node type name, storage location, and the unique name with the installed version.

    

## Create Your Dynamic Tables

In this section you’ll create two dynamic tables from your source data.

1. Select the `Region`, `Nation` , and `Customer` source nodes using the **Shift key**.
2. Then **Right-Click > Join Nodes > Dynamic Table > Dynamic Table Work**.

    

3. Update the Dynamic Table Options to:
    1. Warehouse to the one you are using. In this example it’s `COMPUTE_WH`.
    2. Downstream - Toggled off
    3. Lag Specification - `1 Minutes`
    4. Refresh Mode - `Auto`
    5. Initialize `ON_SCHEDULE`

    

4. Because there are three tables, we need to Join them. In the **Join Tab** of the node, enter your Join statement. If your table names are different from the example, make sure to update the names.

    ```sql
    FROM {{ ref('SAMPLE', 'NATION') }} "NATION"
    INNER JOIN {{ ref('SAMPLE', 'REGION') }} "REGION"
    ON "NATION"."N_REGIONKEY" = "REGION"."R_REGIONKEY"
    INNER JOIN {{ ref('SAMPLE', 'CUSTOMER') }} "CUSTOMER"
    ON "NATION"."N_NATIONKEY" = "CUSTOMER"."C_NATIONKEY"
    ```

    

5. You need to delete columns that aren’t needed or don’t need to be updated.
6. Press the Command key to multi-select the columns `N_NATIONKEY`, `N_REGIONKEY`, `N_COMMENT`, `R_REGIONKEY` and `R_COMMENT` then right-click to delete them from your node.
7. Click **Create** to create your Dynamic Table.
8. For the next steps, you’ll create your second Dynamic Table.
9. Go back to the **Browser Tab**, and select `LINEITEM` and `ORDERS`, then**Right-Click > Join Nodes > Dynamic Table > Dynamic Table Work**.
10. Update the Dynamic Table Options to:
    1. Warehouse to the one you are using. In this example it’s `COMPUTE_WH`.
    2. Downstream - Toggled off.
    3. Lag Specification -`1 Minutes`.
    4. Refresh Mode - `Auto`
    5. Initialize `ON_SCHEDULE`
11. In the Join Tab, update your join statement. If your table names are different from the example, make sure to update the names.

    ```sql
    FROM {{ ref('SAMPLE', 'ORDERS') }} "ORDERS"
    INNER JOIN {{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    ON "ORDERS"."O_ORDERKEY" = "LINEITEM"."L_ORDERKEY"
    ```

12. In the Mapping Tab, delete the columns you don’t need. `O_ORDERSTATUS`, `O_TOTALPRICE`, `O_ORDERDATE`, `O_CLERK`, `O_SHIPRIORITY`, `L_PARTKEY` and `L_SUPPKEY`.
13. Click **Create** to create your Dynamic Table.

## Join Dynamic Tables

In this section you’ll Join the Dynamic Tables you created to aggregate the quantity of goods by customer.

1. On the **Browse Tab**, select the two tables you created. Then **Right-Click > Join Nodes > Dynamic Table > Dynamic Table Work**.

    

2. Update the Dynamic Table Options to:
    1. Warehouse to the one you are using. In this example it’s `COMPUTE_WH`.
    2. Downstream - Toggled off
    3. Lag Specification - `15 Minutes.` This is so it refreshes less frequently than the other nodes.
    4. Refresh Mode - `Auto`
    5. Initialize `ON_SCHEDULE`
3. Update the General Options to:
    1. Toggle `Group By All` to on. This is different from the other tables you created.
4. Click on the **Join Tab** and update the statement. If your table names are different from the example, make sure to update the names.

    ```sql
    FROM {{ ref('WORK', 'DT_ORDERS_LINEITEM') }} "DT_ORDERS_LINEITEM"
    INNER JOIN {{ ref('WORK', 'DT_CUSTOMER_NATION_REGION') }} "DT_CUSTOMER_NATION_REGION"
    ON "DT_ORDERS_LINEITEM"."O_CUSTKEY" = "DT_CUSTOMER_NATION_REGION"."C_CUSTKEY"
    ```

5. In the Mapping Tab, delete all columns **except** `C_NAME` and `L_QUANTITY`
6. Make sure `C_NAME` appears first by dragging it to the top.
7. Double click in the [Transform][] column for `L_QUANTITY` and enter:

    ```sql
    sum({{SRC}})
    ```

8. You’ll see the column resolve to the sum of values of SRC. SRC is a shorthand you can use in Coalesce called a [Helper Token][].

    

9. Click **Create**.

## Run Your Dynamic Tables

Currently, your Dynamic tables are set up to run based on a lag specification. You can also change it so that the dynamic table is refreshed on demand when the other tables need to refresh.

In this example, we can change our two tables (`DT_ORDERS_LINEITEM` and `DT_CUSTOMER_NATION_REGION`) to use Downstream and so they start updating based on the `DT_WRK_DT_CUSTOMER_NATION_REGION_DT_ORDERS_LINEITEM` table lag time.

1. Open the `DT_ORDERS_LINEITEM` and update the Dynamic Tables Options to **Downstream**.
2. Open the `DT_CUSTOMER_NATION_REGION` and update the Dynamic Tables Options to **Downstream**.

    

3. Go back to the **Browser** Tab and review your DAG. Your DAG might look different depending on the data you used for this example. Click **Run All**. You’ll be able to see the Nodes run.

## Review the DAG in Snowflake

1. In Snowflake go to **Data > Databases** and select the one containing your data.
2. Open the schema, in this example it’s `DEV`, open the `DT_WRK_DT_CUSTOMER_NATION_REGION_DT_ORDERS_LINEITEM` dynamic table.
3. Notice that the other tables now have a lag of 15 minutes instead of the 1 minute they were created with.

## Conclusion

Thanks to the Dynamic Tables package, you can get a pipeline up and running quickly.

### What's Next?

- [Dynamic Tables Package][]
- [Deploy Your Pipeline][]

[Snowflake's Dynamic Tables]: https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[refreshes]: https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
[Dynamic Table Package]: /docs/marketplace/package/coalesce_snowflake_dynamic-tables
[Lag specification]: https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#label-dynamic-tables-understand-dt-lag
[Refresh]: /docs/deploy-and-refresh/refresh/
[CREATE DYNAMIC TABLE]: https://docs.snowflake.com/en/user-guide/dynamic-tables-privileges
[trial account data]: https://docs.snowflake.com/en/user-guide/sample-data
[Node Graph]: /docs/build-your-pipeline/the-build-interface#dag-or-graph-view
[Dynamic Tables package page]: /docs/marketplace/package/coalesce_snowflake_dynamic-tables
[Transform]: /docs/build-your-pipeline/transforms/
[Helper Token]: /docs/reference/helper-tokens/
[Dynamic Tables Package]: /docs/marketplace/package/coalesce_snowflake_dynamic-tables
[Deploy Your Pipeline]: /docs/deploy-and-refresh/deploy/
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce

---

## Build Incremental Pipelines with Run Views and Sequences

## Introduction

When you load change data in batches across multiple sources, a timestamp watermark alone often is not enough. You need a durable record of which source slice ran, in what order, and which rows belong to the current load window. This guide walks you through building that pipeline in Coalesce with Run Views, a sequence control table, and Job parameters.

This guide assumes you have knowledge of Coalesce including:

- [Environment][]
- [Version Control][]
- [Deployment][]

### What You Will Build

When you finish this guide, you will have:

- Installed the [Incremental Loading Package][] and used **Run View** and **Incremental Load** Node Types.
- A multi-suffix pipeline where Jobs pick branch **A**, **B**, both, or **ALL** through the **`sourcesuffix`** parameter.
- Deploy-safe guards so **Deploy** creates empty views and does not advance **`LOAD_SEQUENCE`**.

### How the Pipeline Fits Together

| Node | Node Type | What it does |
| ---- | --------- | ------------ |
| `LINEITEM` | Source | Snowflake sample table that supplies row data for every branch. |
| `LOAD_SEQUENCE` | Persistent Stage | Empty control table you create first. Each suffix Run View appends a row with the next sequence number, suffix letter, and load timestamp when a Job runs. |
| `STG_LINEA` | Run View | Staging view for branch **A**. Reads `LINEITEM`, creates the view, and inserts `'A'` into **`LOAD_SEQUENCE`**. Deploy guards skip the insert. |
| `STG_LINEB` | Run View | Same pattern as **`STG_LINEA`** for branch **B**. The trailing **`B`** in the Node name is the suffix the package matches to Job parameters. |
| `RV_LINEITEM` | Run View | Multi-source union over **`STG_LINEA`** and **`STG_LINEB`**. **`sourcesuffix`** selects which branches contribute rows to the union for the current Job. |
| `INC_LINEITEM` | Incremental Load | View over **`RV_LINEITEM`** that returns only rows newer than the watermark in **`PSTG_LINEITEM`** once incremental filtering is on. |
| `PSTG_LINEITEM` | Persistent Stage | Physical table that stores merged rows after each successful load. |

### Source Suffixes and Node Names

The **suffix** is the trailing character on each branch Run View name (**`A`** on **`STG_LINEA`**, **`B`** on **`STG_LINEB`**). The Incremental Loading Package compares that character to the Job parameter **`sourcesuffix`** when **Filter data based on Source** is on.

Name branches with a uniform suffix length. This guide uses one character, which matches **Length of suffix of source table** `1` on **`RV_LINEITEM`**. See [Incremental Loading Package Run View Parameters][] for longer suffix patterns.

### When to Use Run Views

Use this table to decide whether the multi-suffix Run View pattern fits your pipeline.

| Requirement | How this pattern helps |
| ----------- | ---------------------- |
| Multiple source branches into one incremental target | Per-suffix **Run View** Nodes feed a multi-source union (**`RV_LINEITEM`**). |
| Parameterized source selection per Job | **`sourcesuffix`** selects suffix `A`, `B`, a list, or `ALL`. |
| Orchestration metadata across runs | **`LOAD_SEQUENCE`** stores batch id, suffix, and load timestamp. |
| Deploy without re-running load logic | Environment **`sourcesuffix`** `('DEPLOY')` plus **Override Create SQL** guards. |

If you only have one source and a single timestamp watermark, start with [Incremental Loading with Coalesce][] or [Incremental Loading Strategies][].

## Before You Begin

1. A Coalesce account with Snowflake access. Start a free Coalesce trial from our [Snowflake Marketplace listing][] if needed.
2. A Project and Workspace with [Storage Locations and Storage Mappings][] configured. This guide uses `SAMPLE` and `WORK`.
3. Have your sources added to the Workspace.
4. Have [Version Control][] configured.
5. Have an [Environment][] configured.
6. Snowflake roles that can create views and tables in those locations.
7. Optional: Snowflake [sample data sets][] available in your account. Map `TPCH_SF1.LINEITEM` or equivalent to location `SAMPLE`.

## Step 1: Install the Incremental Loading Package

1. Go to the [Coalesce Marketplace][] and search for [Incremental Loading][].
2. Copy the Package ID `@coalesce/snowflake/incremental-loading`.
3. Open **Build Settings** from the left sidebar.
4. In the Coalesce App, go to **Build Settings > Packages**. Click **Install**.
5. Enter the Package ID you previously copied.
6. Select the version you want to install.
7. Enter a **Package Alias**. The alias is a unique and descriptive name for the Package so you can identify it later.
   1. A Package alias can only contain letters, numbers, and dashes. It can't contain consecutive dashes.
   2. Package aliases can't start or end with a dash.
8. Click **Install** to add the Package.

    

## Step 2: Create the LOAD_SEQUENCE Control Table

**`LOAD_SEQUENCE`** is the orchestration control table for this pattern. Create it first, while it is still empty, before you add suffix Run Views that append rows to it. When a Job runs **`STG_LINEA`** or **`STG_LINEB`**, each node inserts the next sequence number, its suffix letter **`A`** or **`B`**, and a load timestamp.

1. In the **Nodes** panel, click the plus sign **+** and select **Create New Node > Persistent Stage**.
2. Rename the Node to **`LOAD_SEQUENCE`**.
3. In **Config > Node Properties**, set the **Storage Location**.
4. You can delete any existing columns.
5. Open the **Mapping** tab and click the plus sign **+** to add a column:

   | Column | Data type |
   | ------ | --------- |
   | `LOAD_SEQUENCE` | NUMBER |
   | `SOURCE_SUFFIX` | VARCHAR |
   | `LOAD_TS` | TIMESTAMP |

6. Click **Create** to create the empty table.

## Step 3: Add STG_LINEA, Suffix A Run View

This Run View stages branch **A** data and records suffix **A** in **`LOAD_SEQUENCE`** when a Job runs.

1. On the graph, right-click **`LINEITEM`** and select **Add Node** > **Incremental Loading** > **Run View**.
2. Rename the Node to **`STG_LINEA`**.
3. Open **`STG_LINEA`** in the Node editor. Go to **Config > Options** and turn **Override Create SQL** on. A **Create SQL** tab appears in the editor alongside **Join** and **Mapping**.
4. Open the **Create SQL** tab and paste the full script below. Coalesce allows only **one SQL statement per stage** in Override Create SQL, so the view `CREATE` and the `INSERT` use separate `{{ stage(...) }}` blocks. Learn more in [Run Multiple SQL Statements][].

    ```sql
    {{ stage('Override Create SQL', deploy_phase_override="create") }}
    CREATE OR REPLACE VIEW {{ ref('WORK', 'STG_LINEA') }} AS (
        SELECT
            "L_ORDERKEY" AS "L_ORDERKEY",
            "L_PARTKEY" AS "L_PARTKEY",
            "L_SUPPKEY" AS "L_SUPPKEY",
            "L_LINENUMBER" AS "L_LINENUMBER",
            "L_QUANTITY" AS "L_QUANTITY",
            "L_EXTENDEDPRICE" AS "L_EXTENDEDPRICE",
            "L_DISCOUNT" AS "L_DISCOUNT",
            "L_TAX" AS "L_TAX",
            "L_RETURNFLAG" AS "L_RETURNFLAG",
            "L_LINESTATUS" AS "L_LINESTATUS",
            "L_SHIPDATE" AS "L_SHIPDATE",
            "L_COMMITDATE" AS "L_COMMITDATE",
            "L_RECEIPTDATE" AS "L_RECEIPTDATE",
            "L_SHIPINSTRUCT" AS "L_SHIPINSTRUCT",
            "L_SHIPMODE" AS "L_SHIPMODE",
            "L_COMMENT" AS "L_COMMENT"
        FROM {{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    );

    {{ stage('Insert Load Sequence') }}
    {% raw %}{% if 'DEPLOY' not in parameters.sourcesuffix %}{% endraw %}
    INSERT INTO {{ ref_no_link('WORK', 'LOAD_SEQUENCE') }}
    SELECT
      COALESCE(MAX("LOAD_SEQUENCE"), 0) + 1,
      'A',
      CURRENT_TIMESTAMP()
    FROM {{ ref_no_link('WORK', 'LOAD_SEQUENCE') }};
    {% raw %}{% endif %}{% endraw %}
    ```

    

5. Open the **Join** tab and paste the block below. `LINEITEM` is the data source. `ref_link()` adds a graph edge so Jobs run **`LOAD_SEQUENCE`** before **`STG_LINEA`**. The `INSERT` uses `ref_no_link()` so it resolves the table name without drawing that edge.

    ```sql
    FROM {{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    {{ ref_link('WORK', 'LOAD_SEQUENCE') }}
    ```

6. Click **Create**.

### Insert Load Sequence

The **Insert Load Sequence** SQL stage appends one row to **`LOAD_SEQUENCE`** each time **`STG_LINEA`** runs on a Job. **`STG_LINEB`** uses the same pattern with suffix `'B'`.

- **`{% raw %}{{ stage('Insert Load Sequence') }}{% endraw %}`** - Separate SQL statement from the view `CREATE` in the first stage.
- **`{% raw %}{% if 'DEPLOY' not in parameters.sourcesuffix %}{% endraw %}`** - Skips the insert during Deploy when **`sourcesuffix`** is `('DEPLOY')`. Sequence rows are written on Jobs only.
- **`INSERT`** - Stores the next sequence number (`COALESCE(MAX("LOAD_SEQUENCE"), 0) + 1`, starting at 1), suffix `'A'`, and load timestamp.
- **`ref_no_link()`** - Resolves **`LOAD_SEQUENCE`** without adding a graph edge. Run order comes from `ref_link()` on the **Join** tab.

Each Job run records the batch number, suffix **A**, and load time in the control table.

## Step 4: Add STG_LINEB, Suffix B Run View

Repeat the **`STG_LINEA`** pattern for branch **B**. The only differences are the Node name, view name, and `'B'` in the sequence insert.

1. Right-click **`LINEITEM`** and select **Add Node** > **Incremental Loading** > **Run View**.
2. Rename the Node to **`STG_LINEB`**.
3. Set **Storage Location**.
4. In **Config > Options**, turn **Override Create SQL** on, then open the **Create SQL** tab. Paste the full script below.

    ```sql
    {{ stage('Override Create SQL', deploy_phase_override="create") }}
    CREATE OR REPLACE VIEW {{ ref('WORK', 'STG_LINEB') }} AS (
        SELECT
            "L_ORDERKEY" AS "L_ORDERKEY",
            "L_PARTKEY" AS "L_PARTKEY",
            "L_SUPPKEY" AS "L_SUPPKEY",
            "L_LINENUMBER" AS "L_LINENUMBER",
            "L_QUANTITY" AS "L_QUANTITY",
            "L_EXTENDEDPRICE" AS "L_EXTENDEDPRICE",
            "L_DISCOUNT" AS "L_DISCOUNT",
            "L_TAX" AS "L_TAX",
            "L_RETURNFLAG" AS "L_RETURNFLAG",
            "L_LINESTATUS" AS "L_LINESTATUS",
            "L_SHIPDATE" AS "L_SHIPDATE",
            "L_COMMITDATE" AS "L_COMMITDATE",
            "L_RECEIPTDATE" AS "L_RECEIPTDATE",
            "L_SHIPINSTRUCT" AS "L_SHIPINSTRUCT",
            "L_SHIPMODE" AS "L_SHIPMODE",
            "L_COMMENT" AS "L_COMMENT"
        FROM {{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    );

    {{ stage('Insert Load Sequence') }}
    {% raw %}{% if 'DEPLOY' not in parameters.sourcesuffix %}{% endraw %}
    INSERT INTO {{ ref_no_link('WORK', 'LOAD_SEQUENCE') }}
    SELECT
      COALESCE(MAX("LOAD_SEQUENCE"), 0) + 1,
      'B',
      CURRENT_TIMESTAMP()
    FROM {{ ref_no_link('WORK', 'LOAD_SEQUENCE') }};
    {% raw %}{% endif %}{% endraw %}
    ```

5. On the **Join** tab, add `LINEITEM` as the source plus `ref_link()` to **`LOAD_SEQUENCE`**.

    ```sql
    FROM {{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    {{ ref_link('WORK', 'LOAD_SEQUENCE') }}
    ```

6. Click **Create**.

## Step 5: Add RV_LINEITEM

**`RV_LINEITEM`** unions the suffix staging views and applies **`sourcesuffix`** so each Job reads only the branches you select.

1. Right-click **`STG_LINEA`** and select **Add Node** > **Incremental Loading** > **Run View**.
2. Rename the Node to **`RV_LINEITEM`**.
3. In **Config > Options**, set:

   | Option | Value |
   | ------ | ----- |
   | **Multi Source** | On |
   | **Multi Source Strategy** | `UNION ALL` |
   | **Filter data based on Source** | On |
   | **Length of suffix of source table** | `1` |

    <figure>
      
      <figcaption>RV_LINEITEM Config and Options with multi-source union and `sourcesuffix` filtering enabled</figcaption>
    </figure>

4. On the **Join** tab for `STG_LINEA`, add the following. Make sure to update to match your Nodes and Storage Locations:

    ```sql
    FROM {{ ref('WORK', 'STG_LINEA') }} "STG_LINEA"
    {% if 'DEPLOY' in parameters.sourcesuffix %}
      WHERE 1=0
    {%- elif 'ALL' not in parameters.sourcesuffix -%}
    WHERE 'A' in {{ parameters.sourcesuffix }}
    {% endif %}
    ```

5. Click the plus sign **+** to add a new source. Rename it `STG_LINEB`. Then drag `STG_LINEB` from the Nodes list into **Map to Existing Columns**. You should see the source update for `STG_LINEB`.

    

6. On the **Join** tab for `STG_LINEB`, add the following. Make sure to update to match your Nodes and Storage Locations:

    ```sql
    FROM {{ ref('WORK', 'STG_LINEB') }} "STG_LINEB"
    {%- if 'DEPLOY' in parameters.sourcesuffix -%}
    WHERE 1=0
    {%- elif 'ALL' not in parameters.sourcesuffix -%}
    WHERE 'B' in {{ parameters.sourcesuffix }}
    {% endif %}
    ```

Each source pill on the **Join** tab uses the same `sourcesuffix` pattern with a different suffix letter:

- **`FROM STG_LINEA` or `FROM STG_LINEB`** - Reads from that branch Run View.
- **`DEPLOY` in `sourcesuffix`** - `WHERE 1=0` creates the view empty so no data loads during Deploy.
- **`'ALL' not in sourcesuffix`** - `WHERE 'A' in sourcesuffix` or `WHERE 'B' in sourcesuffix` includes that branch only when its letter is in the Job parameter list, for example `('A')`, `('B')`, or `('A','B')`.
- **`('ALL')`** - No `WHERE` clause, so both branches contribute rows.

`sourcesuffix` controls which branches contribute rows: A, B, both, or all.

:::note[Checkpoint]

At this point you should have a multi-source Node named `RV_LINEITEM` with `STG_LINEA` and `STG_LINEB` mapped. Do not click **Create** until you set Workspace parameters in **Step 6**.

:::

## Step 6: Set Workspace Parameters and Create RV_LINEITEM

Set **Workspace** parameters before you **Create** **`RV_LINEITEM`**. With **Filter data based on Source** on, the join template reads `parameters.sourcesuffix`. Workspace defaults apply until an Environment or Job Schedule overrides them.

1. Open **Build Settings** > **Workspace** > **Settings** > **Parameters**.
2. Add a default for the first **Create**, for example:

    ```json
    {
      "sourcesuffix": "('ALL')"
    }
    ```

    You can also set it to `('A')` or `('B')`. **`RV_LINEITEM`** then returns rows from that branch only. With `('ALL')`, it unions both branches.

    

3. Open **`RV_LINEITEM`** and click **Create** so the union view exists in Snowflake with your default `sourcesuffix`.

## Step 7: Add INC_LINEITEM and PSTG_LINEITEM and Run the Initial Load

Create the incremental view and its persistent target before you turn on watermark filtering.

1. Right-click **`RV_LINEITEM`** and select **Add Node** > **Incremental Load**.
2. Rename the Node to **`INC_LINEITEM`** and set the **Storage Location** . Leave **Filter data based on Persistent table** **off** for now.

    <figure>
      
      <figcaption>INC_LINEITEM initial setup with Storage Location WORK and persistent-table filtering off</figcaption>
    </figure>

3. Right-click **`INC_LINEITEM`** and select **Add Node** > **Base Node Types** > **Persistent Stage**.
4. Rename the Node to **`PSTG_LINEITEM`**.
5. Open **`INC_LINEITEM`**. On the **Join** tab, confirm the source is **`RV_LINEITEM`**, then click **Create** so the view exists before you load the persistent table.
6. Open **`PSTG_LINEITEM`** and click **Create**, then **Run** to load the initial data set.

## Step 8: Configure INC_LINEITEM for Incremental Loads

Turn on incremental filtering so **`INC_LINEITEM`** returns only rows newer than the watermark in **`PSTG_LINEITEM`**.

1. Open **`INC_LINEITEM`** Node.
2. In **Config > Options**:
   1. **Filter data based on Persistent table** **on**. Set **Persistent table location** to your Storage Location.
   2. **Persistent table name** to `PSTG_LINEITEM`
   3. **Incremental load column (date)** to `L_SHIPDATE`.
3. On the **Join** tab, remove any existing Join text, enter the following SQL. Make sure to update to match your Nodes and sources.

    ```sql
    FROM {{ ref('WORK', 'RV_LINEITEM') }} "RV_LINEITEM"
    WHERE "RV_LINEITEM"."L_SHIPDATE" >
    (SELECT COALESCE(MAX("L_SHIPDATE"), '1900-01-01')
                FROM {{ ref_no_link('WORK', 'PSTG_LINEITEM') }} )
    ```

   This keeps rows from **`RV_LINEITEM`** whose **`L_SHIPDATE`** is newer than the maximum already in **`PSTG_LINEITEM`**, or after `1900-01-01` when the table is empty.

4. Click **Create**, then **Run**.

<figure>
  
  <figcaption>INC_LINEITEM incremental configuration with persistent-table filtering and Join watermark on L_SHIPDATE</figcaption>
</figure>

:::note[Checkpoint]

**`INC_LINEITEM`** returns only rows newer than the watermark in **`PSTG_LINEITEM`**.

:::

## Step 9: Create a Job

You'll create a Job that contains the Nodes to refresh.

1. In the left sidebar, click **Jobs**, then click **+** and select **Create Job**.

    

2. Create a new Job called `RV_REFRESH_JOB`.
3. While on the Jobs screen, click **Nodes**.
4. Drag all your Nodes into the Job except for the Source Node.

## Step 10: Commit and Deploy

Commit and deploy your changes using the version control provider and Environment created in Before You Begin.

Set **`sourcesuffix`** on the [target environment][] so Deploy uses empty views and skips sequence inserts.

    ```json
    {
      "sourcesuffix": "('DEPLOY')"
    }
    ```

## Step 11: Schedule the Job with Parameters

Use the Coalesce Scheduler to refresh the pipeline on a schedule. You can set different refresh times and a different `sourcesuffix` for each schedule if needed.

1. In the Coalesce App, open **Deploy**, click the dropdown next to the Environment and click **View Scheduled Jobs**.
2. Click **Create Job Schedule**.
3. Fill out the **Configuration** information.
4. Then set the **Notifications**.
5. On Parameters, enable **Override Environment**. Then set the parameters based on how you want this refresh Job to run. You can create multiple Job Schedules if you want to run this job multiple times with different parameters.

    - **Branch A only**

        ```json
        {
          "sourcesuffix": "('A')"
        }
        ```

    - **Branches A and B**

        ```json
        {
          "sourcesuffix": "('A','B')"
        }
        ```

    - **All branches**

        ```json
        {
          "sourcesuffix": "('ALL')"
        }
        ```

    

6. Save the **Job Schedule**. Each scheduled run re-creates the Run Views with your **`sourcesuffix`**, advances **`LOAD_SEQUENCE`**, and drives **`INC_LINEITEM`** for the selected branches.

## Common Issues

- **Sequence rows on every Deploy:** Confirm **Step 10** sets environment **`sourcesuffix`** to `('DEPLOY')`, and **Steps 3 and 4** include `deploy_phase_override="create"` and the `{% if 'DEPLOY' not in parameters.sourcesuffix %}` guard on sequence inserts.
- **`Actual statement count 2 did not match the desired statement count 1`:** Put the view `CREATE` and the `INSERT` into separate `{{ stage(...) }}` blocks. See [Run Multiple SQL Statements][].
- **Wrong branch or no rows from `RV_LINEITEM`:** Keep suffix length uniform on **`STG_LINEA`** and **`STG_LINEB`** with length **1**, and regenerate Join on each source pill after you change suffix length.
- **Empty incremental load on first run:** Leave **Filter data based on Persistent table** off until **`PSTG_LINEITEM`** has data from **Step 7**.
- **`LOAD_SEQUENCE` dependency errors:** Create and **Create** **`LOAD_SEQUENCE`** before you add `ref_link()` on the suffix Run Views.

## What's Next?

- [Incremental Loading Strategies][] for chooser context across warehouses
- [Incremental Loading with Coalesce][] for watermark basics on a single source
- [Incremental Loading Package][] listing for Run View, Looped Load, and Grouped Incremental Load reference
- [Parameters][] for inheritance between environments and Jobs
- [Refreshing Your Pipeline Using the Coalesce Scheduler][] for cron, notifications, and schedule management

[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[sample data sets]: /docs/setup-your-project/connection-guides/snowflake/
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[Incremental Loading with Coalesce]: /docs/guides/using-incremental-nodes
[Incremental Loading Strategies]: /docs/build-your-pipeline/incremental-loading-strategies
[Incremental Loading Package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading/
[Incremental Loading Package Run View Parameters]: /docs/marketplace/package/coalesce_snowflake_incremental-loading#run-view-parameters
[Parameters]: /docs/deploy-and-refresh/parameters/
[Coalesce Marketplace]: /docs/marketplace
[Run Multiple SQL Statements]: /docs/build-your-pipeline/pre-post-sql#run-multiple-sql-statements
[Version Control]: /docs/git-integration
[Environment]: /docs/get-started/coalesce-fundamentals/environments
[Deployment]: /docs/deploy-and-refresh/deploy
[Incremental Loading]: /docs/marketplace/package/coalesce_snowflake_incremental-loading
[Refreshing Your Pipeline Using the Coalesce Scheduler]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[target environment]: /docs/marketplace/package/coalesce_snowflake_incremental-loading#run-view-deployment

---

## Building Data Pipelines with Iceberg and Snowflake LLM-Based Functions

## Overview

This Hands-On Lab exercise is designed to help you learn how to build and manage Iceberg tables with Snowflake Cortex LLM nodes within Coalesce. In this lab, you’ll explore the Coalesce interface, load Iceberg tables into your project, learn how to easily transform and model your data with our core capabilities, and use the Cortex LLM functions node that Coalesce provides users.

### **What You’ll Need**

- A Snowflake account (either a [trial account][] or access to an existing account)
- A Coalesce account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account)
- Basic knowledge of SQL, database concepts, and objects
- The [Google Chrome browser][]

### **What You’ll Build**

- A Directed Acyclic Graph (DAG) that builds an Cortex LLM Pipeline which leverages Iceberg tables

### **What You’ll Learn**

- How to navigate the Coalesce interface
- Load in iceberg tables
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to union tables
- How to set up and configure Cortex LLM Nodes
- How to analyze the output of your results in Snowflake

By completing the steps we’ve outlined in this guide, you’ll have mastered the basics of Coalesce and can venture into our more advanced features.

## About Coalesce

Coalesce is the first cloud-native, visual data transformation platform. Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

With Coalesce, you can:

- Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
- Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Snowflake functionality
- Analyze the impact of changes to pipelines with built-in data lineage down to the column level
- Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
- Ensure consistent data standards and governance across pipelines, with data never leaving your Snowflake instance

### How Is Coalesce Different?

Coalesce’s unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### Core Concepts in Coalesce

#### **Snowflake**

Coalesce currently only supports Snowflake as its target database, As you will be using a trial Coalesce account created with [Snowflake Marketplace][], your basic database settings will be configured automatically and you can instantly build code.

### **Organization**

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

### **Projects**

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Gi repository and Snowflake account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

### **Workspaces vs. Environments**

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Snowflake schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, PreProd, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

## Lab Use Case

You will be stepping into the shoes of the lead Data & Analytics manager for the fictional Snowflake Ski Store brand. You are responsible for building and managing data pipelines that deliver insights to the rest of the company in a timely and actionable manner. Management has asked that they receive insight into how customers feel about product defects, and which customers are the most unhappy when defects are received. The data you will be analyzing is from the Snowflake Ski Store call center, where customers call in with questions or complaints. Each conversation is recorded and stored as a call transcript within a table of your database. This is what you will use to uncover the insights that management is seeking.

Management is hoping you can create and build a pipeline that uses Large Language Model functions that allow them to have consistent insight into customer sentiment and around the brand and product.

## Before You Start

To complete this lab, please create free trial accounts with Snowflake and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

We recommend using Google Chrome as your browser for the best experience.

:::warning[Complete All Steps]

Not following these steps will cause delays and reduce your time spent in the Coalesce environment.

:::

### Step 1: Create a Snowflake Trial Account

1. Fill out the [Snowflake trial account form][]. Use an email address that is not associated with an existing Snowflake account.

   ![image1](./building-data-pipelines-with-iceberg-llm-functions/assets/image1.png)

2. After registering, you will receive an email from Snowflake with an activation link and URL for accessing your trial account. Finish setting up your account following the instructions in the email.

### Step 2: Create a Coalesce Trial Account from Snowflake Marketplace

1. Sign in to Snowsight. Make sure you are using the ACCOUNTADMIN role (or a role with privileges to create a database, warehouse, user, and role) and that your Snowflake email address is verified.
2. In the navigation menu, select **Marketplace > Snowflake Marketplace** and search for Coalesce.

      

3. Open the Coalesce listing and select Start free trial in the upper right. To see what the trial includes, review the Your free trial tab.
4. In the Connect to Coalesce dialog, review the information to be shared and the Snowflake objects that will be created, then select Connect to Coalesce.
5. You will be redirected to Coalesce to finish activating your trial account. Fill in your information to complete activation.

Congratulations. You’ve successfully created your Coalesce trial account.

### Step 3: Set Up External Volume and Catalog Integration

We will be using a dataset from a public S3 bucket for this lab. The bucket has been provisioned so that any Snowflake account can access the dataset. In order to use Iceberg tables, you are required to set up an external integration as well as a catalog integration. This step will walk you through how to do this.

1. Within Worksheets, click the "+" button in the top-right corner of Snowsight and choose "SQL Worksheet.”

   ![image6](building-data-pipelines-with-iceberg-llm-functions/assets/image6.png)

2. Now you can copy and paste the following code into your worksheet. Ensure you are using the ***ACCOUNTADMIN*** role in Snowflake for this process.

   ```SQL
      CREATE or REPLACE database coalesce_hol;
      CREATE or REPLACE schema coalesce_hol.calls;
      CREATE or REPLACE file format jsonformat
      type = 'JSON';

      create or replace stage coalesce_hol.calls.cortex_iceberg_hol
      url = 's3://iceberg-hol/';

      CREATE OR REPLACE EXTERNAL VOLUME iceberg_external_volume
      STORAGE_LOCATIONS =
   (
            (
               NAME = 'iceberg-hol'
               STORAGE_PROVIDER = 'S3'
               STORAGE_BASE_URL = 's3://iceberg-hol/'
               STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::034362027654:role/iceberg-hol-role'
               STORAGE_AWS_EXTERNAL_ID = 'iceberg-hol'
            )
         );

      CREATE OR REPLACE CATALOG INTEGRATION iceberg_catalog_integration
      CATALOG_SOURCE = OBJECT_STORE
      TABLE_FORMAT = ICEBERG
      ENABLED = TRUE;

      GRANT USAGE ON EXTERNAL VOLUME iceberg_external_volume TO ROLE pc_coalesce_role;
      GRANT USAGE ON INTEGRATION iceberg_catalog_integration TO ROLE pc_coalesce_role;
   ```

3. You have now successfully set up an external volume and catalog integration.

## Navigating the Coalesce User Interface

> **📌 Note:** About this lab.
> Screenshots (product images, sample code, environments) depict examples and results that may vary slightly from what you see when you complete the exercises. This lab exercise does not include Git (version control). Please note that if you continue developing in your Coalesce account after this lab, none of your work will be saved or committed to a repository unless you set up before developing.

Let's get familiar with Coalesce by walking through the basic components of the user interface.

​​Once you've activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started.

   ![image7](building-data-pipelines-with-iceberg-llm-functions/assets/image7.png)

1. Launch your Development Workspace by clicking the Launch button and navigate to the Build Interface of the Coalesce UI. This is where you can create, modify, and publish nodes that will transform your data.

   Nodes are visual representations of objects in Snowflake that can be arranged to create a Directed Acyclic Graph (DAG or Graph). A DAG is a conceptual representation of the nodes within your data pipeline and their relationships to and dependencies on each other.

2. In the Browser tab of the Build interface, you can visualize your node graph using the Graph, Node Grid, and Column Grid views. In the upper right hand corner, there is a person-shaped icon where you can manage your account and user settings.

   ![image8](building-data-pipelines-with-iceberg-llm-functions/assets/image8.png)

3. By clicking the question mark icon, you can access the Resource Center to find product guides, view announcements, request support, and share feedback.

   ![image9](building-data-pipelines-with-iceberg-llm-functions/assets/image9.png)

4. The Browser tab of the Build interface is where you’ll build your pipelines using nodes. You can visualize your graph of these nodes using the Graph, Node Grid, and Column Grid views. While this area is currently empty, we will build out nodes and our Graph in subsequent steps of this lab.

   ![image10](building-data-pipelines-with-iceberg-llm-functions/assets/image10.png)

5. Next to the Build interface is the Deploy page. This is where you will push your pipeline to other environments such as Testing or Production.

   ![image11](building-data-pipelines-with-iceberg-llm-functions/assets/image11.png)

6. Next to the Deploy page is the Docs interfaces. Documentation is an essential step in building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. To help with this, Coalesce automatically generates and updates documentation as you work.

   ![image12](building-data-pipelines-with-iceberg-llm-functions/assets/image12.png)

## Install Iceberg and Cortex Packages from Coalesce Marketplace

In order to leverage Iceberg table functionality, we need to add Iceberg table nodes to our workspace. Using Coalesce Marketplace, we can easily install Iceberg nodes that will be immediately available to use, which will allow us to read in and manage data in Iceberg table format. Additionally, we will be working with a dataset that contains multiple languages in the form of call transcripts. We want to explore these transcripts further using Cortex LLM functions, so we’ll need to use the Cortex package from Coalesce Marketplace.

1. In the build interface, navigate to the build settings in the lower left corner of the screen.

   ![image13](building-data-pipelines-with-iceberg-llm-functions/assets/image13.png)

2. Select Packages from the menu list presented.

   ![image14](building-data-pipelines-with-iceberg-llm-functions/assets/image14.png)

3. Select the Browse button in the upper right corner of the screen to go to Coalesce Marketplace. Scroll down to find the Iceberg Tables Package and click "Find out more." In the package details, find the Package ID and copy it.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image6.3.png)

4. Navigate back to Coalesce and select the Install button in the upper right corner of the screen. Paste in the Package ID into the Install Package modal. Coalesce will automatically select the most recent version of the package.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image6.4.png)

5. Provide a Package Alias. The Package Alias will be the name of the package as it is displayed within the Build Interface of Coalesce. In this case, we’ll call the package Iceberg. Select Install once you have filled out all of the information in the modal.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image6.5.png)

6. Repeat these steps to install the Cortex Package and call the Package Alias Cortex.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image6.6.png)

7. Finally, within the Build settings of the workspace, navigate to Node Types and toggle on View.

## Adding an Iceberg Table to a Pipeline

Let’s start to build the foundation of your LLM data pipeline by creating a Graph (DAG) and adding data in the form of Iceberg Tables.

1. Start in the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and navigate to Create New Node → Iceberg → Copy into Iceberg Table.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.1.png)

2. Coalesce will automatically open the Iceberg node which we can configure using the information from the SQL we ran in section 4 of the lab guide. Select the Iceberg Options group from within the node.

   ![image25](building-data-pipelines-with-iceberg-llm-functions/assets/image25.png)

   For the Type of catalog dropdown, we will be creating a Snowflake Iceberg table so we can leave that to the default of Snowflake.

3. For the External Volume parameter, we will pass through the external volume using S3 that was created in section 4 of this lab guide. The external volume is called iceberg_external_volume.

   ![image27](building-data-pipelines-with-iceberg-llm-functions/assets/image27.png)

4. Next, provide a base location name to the base location parameter. This will be the folder location within S3 that the table will be created. For the sake of this lab, use your first name and iceberg_hol as the location name so everyone has their own separate folder i.e. firstname_iceberg_hol.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.5.png)

5. Close the Iceberg Options group and open the Source Data group. Enable the toggle button for External location. In the External URI textbox that appears, enter s3://iceberg-hol/call_transcripts.json

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.6.png)

6. Close the Source Data group and open the File Format group. From the File Format Definition dropdown, select File Format Values. In the File Type dropdown that appears, select JSON.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.7.png)

7. Rename the node to CALL_TRANSCRIPTS.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.8.png)

8. Select Create and then Run to create the object in Snowflake and populate it with the data in the Iceberg table format from S3.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image7.9.png)

9. With the data read from the S3 bucket, we can now continue to transform this data.

## (Optional) Processing JSON As The Data Source

Instead of using Iceberg tables, you can easily ingest JSON files using Coalesce with just a few clicks. In this optional section, you can ingest the same call transcript data as a JSON file rather than iceberg.

1. Install the External Data Package like you would have with the Cortex or Iceberg Packages.

   ![image96](building-data-pipelines-with-iceberg-llm-functions/assets/image96.png)

2. Let’s start to build the foundation of your LLM data pipeline by creating a Graph (DAG) and adding data from a Snowflake stage in JSON format.

3. In the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and navigate to Create New Node → EDP → Copy Into.

   ![image87](building-data-pipelines-with-iceberg-llm-functions/assets/image87.png)

4. Coalesce will automatically open the Copy Into node which we can configure using the information from the SQL we ran in section 4 of the lab guide. Select the Source Data dropdown from within the node and toggle on **Enternal or External Stage**

   ![image88](building-data-pipelines-with-iceberg-llm-functions/assets/image88.png)

5. We will fill out the parameters for this configuration. First, we need to supply the storage location of the stage. Which is the database and schema where the stage was created. In this case the storgae location name is **CALLS**.

   ![image89](building-data-pipelines-with-iceberg-llm-functions/assets/image89.png)

6. For the Stage Name, we will input the name of the stage what we created in section 4 of the lab guide. In this case, **cortex_iceberg_hol**.

   ![image90](building-data-pipelines-with-iceberg-llm-functions/assets/image90.png)

7. We won't be including a path of subfolder, so delete the text from this input box. For the File Name, input **call_transcripts.json**

   ![image91](building-data-pipelines-with-iceberg-llm-functions/assets/image91.png)

   > **📌 Note:** Older versions of the External Data Package (EDP) may require single quotes ('') to be wrapped around the file name, such as 'call_transcripts.json'. If you are using an older version of the package and having issues loading the file, put the file name in single quotes.

8. From here, the last thing we need to do is open the File Format dropdown. In the parameters of the dropdown, we first need to supply the storage location of our file format, which was created in section 4 of the lab guide. In this case, **CALLS**.

   ![image92](building-data-pipelines-with-iceberg-llm-functions/assets/image92.png)

9. Next, supply the file format name which was also created in section 4 of the lab guide. In this case **JSONFORMAT**.

   ![image93](building-data-pipelines-with-iceberg-llm-functions/assets/image93.png)

10. Change the File Type from CSV to JSON to match the file type we are loading from the Snowflake stage.

      ![image94](building-data-pipelines-with-iceberg-llm-functions/assets/image94.png)

11. Finally, change the name of the node to **CALL_TRANSCRIPTS** and select Create to create an object in Snowflake which will contain the JSON for you to work with. Then click Run to populate the table with the JSON string.

   ![image95](building-data-pipelines-with-iceberg-llm-functions/assets/image95.png)

## Creating Stage Nodes

Now that you’ve added your Source node, let’s prepare the data by adding business logic with Stage nodes. Stage nodes represent staging tables within Snowflake where transformations can be previewed and performed. Let's start by adding a standardized "stage layer" for the data source.

First of all, we'll need to turn the JSON array with attributes into rows and columns in a table.

1. In the mapping grid of the CALL_TRANSCRIPT node, right click on the SRC column, select Add Node > Stage. This will create a new stage node and Coalesce will automatically open the mapping grid of the node for you to begin transforming your data.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image8.1.png)

2. Within the new node, right click on the SRC column and select Derive Mappings > From JSON.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image8.2.1.png)

After a few seconds, the mapping grid will be populated with new columns based on the attributes found inside the flattened JSON structure, including derived data types.

![image36](building-data-pipelines-with-iceberg-llm-functions/assets/image8.2.2.png)

1. Delete the SRC column by right clicking on it and selecting Delete Column.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image8.3.png)

2. Rename the node to STG_CALL_TRANSCRIPTS_GERMAN

   ![image39](building-data-pipelines-with-iceberg-llm-functions/assets/image39.png)

3. Coalesce also allows you to write as much custom SQL as needed for your business use case. In this case, we need to process French and German language records in separate tables so that we can pass each language to its own Cortex translation function and union the tables together.

   To do this, navigate to the Join tab, where you will see the node dependency listed within the Ref function and the lateral flatten logic that Coalesce added automatically to flatten the JSON structure into separate records. However, since the source data is a string we'll need to add explicit JSON parsing to this code. Also, we will add a filter to limit our dataset to only records with German data. Use the following code snippet, the bits in bold are what changed compared to the generated code:

   ```sql
   FROM {{ ref('STAGING', 'CALL_TRANSCRIPTS') }} "CALL_TRANSCRIPTS",
   lateral flatten(input => parse_json("SRC"), OUTER => TRUE) "SRC"
   WHERE "LANGUAGE" = 'German'
   ```

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/9.3.jpg)

4. Now that we have transformed and filtered the data for our German node, we need to process our French and English data. Coalesce leverages metadata to allow users to quickly and easily build objects in Snowflake. Because of this metadata, users can duplicate existing objects, allowing everything contained in one node to be duplicated in another, including SQL transformations.

   Navigate back to the Build Interface and right click on the STG_CALL_TRANSCRIPTS_GERMAN node and select duplicate node.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image8.6.png)

5. Coalesce automatically opens the new node. Rename the node to STG_CALL_TRANSCRIPTS_FRENCH.

   ![image43](building-data-pipelines-with-iceberg-llm-functions/assets/image43.png)

6. Navigate to the Join tab, where we will update the where condition to exclude any German data, so we can process the rest of the languages.

   ```SQL
   WHERE "LANGUAGE" <> 'German'
   ```

   ![image44](building-data-pipelines-with-iceberg-llm-functions/assets/image44.png)

7. All the changes made from the STG_CALL_TRANSCRIPTS_GERMAN carry over into this node, so we don’t need to rewrite any of our transformations. Let’s change the view back to Graph and select Create All and then Run All

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image.8.9.png)

## Translating Text Data with Cortex LLM Functions

Now that we have prepared the call transcript data by creating nodes for each language, we can now process the language of the transcripts and translate them into a singular language. In this case, English.

1. Select the STG_CALL_TRANSCRIPTS_GERMAN node and hold the Shift key and select the STG_CALL_TRANSCRIPTS_FRENCH node. Right click on either node and navigate to Add Node. You should see the Cortex package that you installed from the Coalesce Marketplace. By hovering over the Cortex package, you should see the available nodes. Select Cortex Functions. This will add the Cortex Function node to each STG node.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image9.1.png)

2. You now need to configure the Cortex Function nodes to translate the language of the transcripts. Double click on the LLM_CALL_TRANSCRIPTS_GERMAN node to open it.

   ![image47](building-data-pipelines-with-iceberg-llm-functions/assets/image47.png)

3. In the configuration options on the right hand side of the node, open the Cortex Package dropdown and toggle on TRANSLATE.

   ![image48](building-data-pipelines-with-iceberg-llm-functions/assets/image48.png)

4. With the TRANSLATE toggle on, In the column name selector dropdown, you need to select the column to translate. In this case, the TRANSCRIPT column.

   ![image49](building-data-pipelines-with-iceberg-llm-functions/assets/image49.png)

5. Now that you have selected the column that will be translated, you will pass through the language you wish to translate from and the language you wish to translate to, into the translation text box. In this case, you want to translate from German to English. The language code for this translation is as follows:

   'de', 'en'

   ![image50](building-data-pipelines-with-iceberg-llm-functions/assets/image50.png)

6. Now that you have configured the LLM node to translate German data to English, you can click Create and Run to build the table in Snowflake and populate it with data.

   ![image51](building-data-pipelines-with-iceberg-llm-functions/assets/image51.png)

7. While the LLM_CALL_TRANSCRIPT_GERMAN node is running, you can configure the LLM_CALL_TRANSCRIPT_FRENCH. Double click on the LLM_CALL_TRANSCRIPT_FRENCH node to open it.

   ![image52](building-data-pipelines-with-iceberg-llm-functions/assets/image52.png)

8. Open the Cortex Package dropdown on the right hand side of the node and toggle on TRANSLATE.

   ![image53](building-data-pipelines-with-iceberg-llm-functions/assets/image53.png)

9. Just like the German node translation, you will pass the TRANSCRIPT column through as the column you want to translate.

   ![image54](building-data-pipelines-with-iceberg-llm-functions/assets/image54.png)

10. Finally, you will configure the language code for what you wish to translate the language of the transcript column from to the language you wish to translate to. In this case, the language code is as follows:

   'fr', 'en'

   Any values in the transcript field which do not match the language being translated from will be ignored. In this case, there are both French and English language values in the TRANSCRIPT field. Because the English values are not in French, they will automatically pass through as their original values. Since those values are already in English, they don’t require any additional processing.

   ![image55](building-data-pipelines-with-iceberg-llm-functions/assets/image55.png)
11. Select Create and Run to build the object in Snowflake and populate it with data

   ![image56](building-data-pipelines-with-iceberg-llm-functions/assets/image56.png)

## Unifying the Translated Data

You have now processed the transcript data by translating the German and French transcripts into English. However, this translated data exists in two different tables, and in order to build an analysis on all of our transcript data, we need to unify the two tables together into one.

1. Select the LLM_CALL_TRANSCRIPTS_GERMAN node and Add Node > Stage.

   ![image57](building-data-pipelines-with-iceberg-llm-functions/assets/image57.png)

2. Rename the node to STG_CALL_TRANSCRIPTS_ALL.

   ![image58](building-data-pipelines-with-iceberg-llm-functions/assets/image58.png)

3. In the configuration options on the right hand side of the node, open the options dropdown and toggle on Multi Source.

   ![image59](building-data-pipelines-with-iceberg-llm-functions/assets/image59.png)

4. Multi Source allows you to union together nodes with the same schema without having to write any of the code to do so. Click on the Multi Source Strategy dropdown and select UNION ALL.

   ![image60](building-data-pipelines-with-iceberg-llm-functions/assets/image60.png)

5. There will be a union pane next to the columns in the mapping grid that will list all of the nodes associated with the multi source strategy of the node. Click the + button to add a node to union to the current node. You will see a new node get added to the pane called New Source.

   ![image61](building-data-pipelines-with-iceberg-llm-functions/assets/image61.png)

6. Within this new source, there is an area to drag and drop any node from your workspace into the grid to automatically map the columns to the original node. Make sure you have the Nodes navigation menu item selected so you can view all of the nodes in your project.

   ![image62](building-data-pipelines-with-iceberg-llm-functions/assets/image62.png)

7. You will now drag and drop the LLM_CALL_TRANSCRIPTS_FRENCH node into the multi source mapping area of the node. This will automatically map the columns to the original node, in this case LLM_CALL_TRANSCRIPTS_GERMAN.

   ![image63](building-data-pipelines-with-iceberg-llm-functions/assets/image63.png)

8. Finally, select the join tab to configure the reference of the node we are mapping. Using metadata, Coalesce is automatically able to generate this reference for you. Hover over the Generate Join button and select Copy to Editor. Coalesce will automatically insert the code into the editor, and just like that, you have successfully created a union for the two datasets without writing a single line of code.

   ![image64](building-data-pipelines-with-iceberg-llm-functions/assets/image64.png)

9. Select Create and Run to build the object and populate it with data.

   ![image65](building-data-pipelines-with-iceberg-llm-functions/assets/image65.png)

## Sentiment Analysis and Finding Customers

Now that we have all of our translated transcript data in the same table, we can now begin our analysis and extract insights from the data. For the sake of our use case, we want to perform a sentiment analysis on the Transcript, to understand how each of our customers felt during their interaction with our company.

Additionally, our call center reps are trained to ask for the customer’s name when someone calls into the call center. Since we have this information, we want to extract the customer name from the transcript so we can associate the sentiment score with our customer to better understand their experience with our company.

1. Right click on the STG_CALL_TRANSCRIPTS_ALL node and we will add one last Cortex Function node. Add Node > Cortex > Cortex Functions.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image11.1.png)

2. Within the node click on the Cortex Package dropdown and toggle on SENTIMENT and EXTRACT ANSWER.

   ![image67](building-data-pipelines-with-iceberg-llm-functions/assets/image67.png)

3. When cortex functions are applied to a column, they overwrite the preexisting values of the column. Because of this, you will need two transcript columns to pass through to your two functions. One to perform the sentiment analysis, and one to extract the customer name from the transcript. Right click on the TRANSCRIPT column and select Duplicate Column.

   ![image68](building-data-pipelines-with-iceberg-llm-functions/assets/image68.png)

4. Double click on the original TRANSCRIPT column name and rename the column to TRANSCRIPT_SENTIMENT.

   ![image69](building-data-pipelines-with-iceberg-llm-functions/assets/image69.png)

5. Double click on the duplicated TRANSCRIPT column name and rename the column to TRANSCRIPT_CUSTOMER.

   ![image70](building-data-pipelines-with-iceberg-llm-functions/assets/image70.png)

6. Next, double click on the data type value for the TRANSCRIPT_CUSTOMER column. Change the data type to ARRAY. This is necessary because the output of the EXTRACT ANSWER function is an array that contains JSON values from the extraction.

   ![image71](building-data-pipelines-with-iceberg-llm-functions/assets/image71.png)

7. Now that your columns are ready to be processed, we can pass them through to each function in the Cortex Package. For the SENTIMENT ANALYSIS, select the TRANSCRIPT_SENTIMENT column as the column name.

   ![image72](building-data-pipelines-with-iceberg-llm-functions/assets/image72.png)

8. For the EXTRACT ANSWER function, select the TRANSCRIPT_CUSTOMER column as the column name.

   ![image73](building-data-pipelines-with-iceberg-llm-functions/assets/image73.png)

9. The EXTRACT ANSWER function accepts a plain text question as a parameter to use to extract the values from the text being processed. In this case, we’ll ask the question “Who is the customer?”

   ![image74](building-data-pipelines-with-iceberg-llm-functions/assets/image74.png)

10. With the node fully configured to process our sentiment analysis and answer extraction, you can Create and Run the node to build the object and populate it with the values being processed.

   ![image75](building-data-pipelines-with-iceberg-llm-functions/assets/image75.png)

## Process and Expose results with a View

You have now used Cortex LLM Functions to process all of your text data without writing any code to configure the cortex functions, which are now ready for analysis. Let’s perform some final transformations to expose for your analysis.

1. Right click on the LLM_CALL_TRANSCRIPTS_ALL node and Add Node > View.

   ![image76](building-data-pipelines-with-iceberg-llm-functions/assets/image76.png)

2. Select the Create button and then Fetch Data. You will see the output from our LLM functions is the sentiment score and an array value containing a customer value with a confidence interval. We want to be able to extract the customer name out of the array value so we can associate the sentiment score with the customer name.

   Right click on the TRANSCRIPT_CUSTOMER column and hover over Derive Mappings and select From JSON.

   ![image77](building-data-pipelines-with-iceberg-llm-functions/assets/image77.png)

3. You will see two new columns automatically created. Answer and score. The answer column contains our customer name. Double click on the answer column name and rename it to CUSTOMER.

   ![image78](building-data-pipelines-with-iceberg-llm-functions/assets/image78.png)

4. Rename the score column to CONFIDENCE_SCORE.

   ![image79](building-data-pipelines-with-iceberg-llm-functions/assets/image79.png)

5. Rerun the view by selecting Create, which will automatically rerun the query that generates the view, which will contain the updated CUSTOMER column we just created.

   ![image80](building-data-pipelines-with-iceberg-llm-functions/assets/image80.png)

## (Optional) Output the View in Iceberg Format

We now have a view that creates an output that can be used by our organization in a variety of ways. In some cases, other systems in our organization may need access to this output in order to allow our company to make decisions. In this case, we can allow everyone to operate on a single copy of data, by using Iceberg tables to output this data.

1. Select the V_CALL_TRANSCRIPTS_ALL node and right click on the node. Select Add Node → Iceberg → Snowflake Iceberg Table. This will create a Snowflake managed Iceberg table in your object storage location.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image13.1.png)

2. Within the mapping grid, delete the TRANSCRIPT_CUSTOMER column which is an array data type, as Iceberg tables do not support array data types.

   ![image82](building-data-pipelines-with-iceberg-llm-functions/assets/image82.png)

3. Within the node config options, select the Iceberg Options dropdown.

   ![image](building-data-pipelines-with-iceberg-llm-functions/assets/image13.3.png)

4. For the External Volume, pass through the external volume that was configured in step 4 of the lab guide:

   iceberg_external_volume
   ![image84](building-data-pipelines-with-iceberg-llm-functions/assets/image84.png)

5. Next, provide a base location name to the base location parameter. This will be the folder location within S3 that the table will be created. For the sake of this lab, use your first name and iceberg_hol as the location name so everyone has their own separate folder, such as firstname_iceberg_hol.

   ![image85](building-data-pipelines-with-iceberg-llm-functions/assets/image85.png)

6. Select Create and Run to create and populate the Snowflake managed table within S3.

## Conclusion and Next Steps

   ![image86](building-data-pipelines-with-iceberg-llm-functions/assets/image85.png)

Congratulations on completing your lab. You've mastered the basics of building and managing Snowflake Cortex LLM functions in Coalesce and can now continue to build on what you learned. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

### What We’ve Covered

- How to navigate the Coalesce interface
- How to load Iceberg tables into your project
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to union tables
- How to set up and configure Cortex LLM Nodes
- How to analyze the output of your results in Snowflake

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our [documentation](https://docs.coalesce.io/docs) and [resources](https://coalesce.io/resources/). For more technical guides on advanced Coalesce and Snowflake features, check out Snowflake Quickstart guides covering [Dynamic Tables](https://quickstarts.snowflake.com/guide/building_dynamic_tables_in_snowflake_with_coalesce/index.html?index=..%2F..index#1) and [Cortex ML functions](https://quickstarts.snowflake.com/guide/ml_forecasting_ad/index.html?index=..%2F..index#0).

### Additional Coalesce Resources

- [Getting Started](https://coalesce.io/get-started/)
- [Documentation](https://docs.coalesce.io/docs) & [Quickstart Guide](https://docs.coalesce.io/docs/quick-start)
- [Video Tutorials](https://fast.wistia.com/embed/channel/foemj32jtv)
- [Help Center](mailto:support@coalesce.io)

Reach out to our sales team at [coalesce.io](https://coalesce.io/contact-us/) or by emailing [sales@coalesce.io](mailto:sales@coalesce.io) to learn more.

[trial account]: https://signup.snowflake.com/
[Google Chrome browser]: https://www.google.com/chrome/
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake Marketplace]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake trial account form]: https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=Cj0KCQiAtvSdBhD0ARIsAPf8oNm6YH7UeRqFRkVeQQ-3Akuyx2Ijzy8Yi5Om-mWMjm6dY4IpR1eGvqAaAg3MEALw_wcB

---

## Building an Enterprise Compliance Pipeline with Coalesce

## Overview

This hands-on lab teaches you how to build an enterprise-grade regulatory compliance pipeline using Coalesce and Snowflake. You'll create a system that tracks customer history, efficiently processes large transaction volumes using incremental patterns, and generates audit reports for SOX and AML compliance.

### What You'll Need

- A Coalesce account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account).
- Basic knowledge of SQL and data warehousing concepts.
- A Snowflake account with access to `SNOWFLAKE_SAMPLE_DATA.TPCDS_SF100TCL` [database][].
  - This is a large database. It's recommended to use a small sampling of the data. Specifically the `STORE_SALES` table. Try using:

    ```sql
    CREATE TABLE YOUR_TABLE.TPCDS_SF10TCL.STORE_SALES_SAMPLE AS
            SELECT * FROM TPCDS_10TB.TPCDS_SF10TCL.STORE_SALES
            SAMPLE SYSTEM (0.23);  -- Adjust percentage to get ~65M rows
    ```

  - A Snowflake warehouse (SMALL or larger recommended)

### What You'll Build

- A complete compliance data pipeline with incremental processing
- Customer dimension with full audit history (SCD Type 2)
- Automated SOX and AML compliance reporting
- High-performance transaction processing system

### What You'll Learn

- How to implement incremental data loading patterns
- How to create slowly changing dimensions for audit trails
- How to build compliance risk assessments
- How to generate regulatory reports
- How to simulate and test incremental loads

### Understanding the Source Data

The TPC-DS dataset simulates a retail company's data warehouse, providing realistic volumes and relationships for testing enterprise-scale data pipelines. For our compliance pipeline, we'll focus on two critical tables that form the foundation of customer transaction tracking.

#### CUSTOMER Table

The `CUSTOMER` table contains master data for all customers in the retail system. This table serves as the source for our slowly changing dimension, allowing us to track customer information changes over time for compliance purposes.

**Key Columns:**

| Column Name | Description | Usage in Pipeline |
|:---|:---|:---|
| `C_CUSTOMER_SK` | Surrogate key uniquely identifying each customer | Business key for SCD Type 2 tracking |
| `C_FIRST_NAME` | Customer's first name | Track name changes for compliance |
| `C_LAST_NAME` | Customer's last name | Track name changes for compliance |
| `C_EMAIL_ADDRESS` | Customer's email address | Risk indicator when missing |
| `C_PREFERRED_CUST_FLAG` | Flag indicating preferred customer status (Y/N) | Used in risk assessment calculations |

:::info[Data Volume]
The CUSTOMER table contains approximately 65 million customer records in the SF100TCL dataset, providing a realistic volume for testing enterprise compliance systems.
:::

#### STORE_SALES Table

The `STORE_SALES` table captures every retail transaction, making it ideal for demonstrating high-volume incremental processing patterns. Each row represents a line item from a sales transaction.

**Key Columns:**

| Column Name | Description | Usage in Pipeline |
|:---|:---|:---|
| `SS_SOLD_DATE_SK` | Date key indicating when the sale occurred | Incremental load column (high water mark) |
| `SS_CUSTOMER_SK` | Foreign key linking to the customer | Joins to customer dimension |
| `SS_ITEM_SK` | Product identifier | Part of composite business key |
| `SS_TICKET_NUMBER` | Unique transaction identifier | Ensures transaction uniqueness |
| `SS_NET_PAID` | Net amount paid after discounts | AML threshold monitoring ($10,000+) |

:::caution[Large Dataset Warning]
The STORE_SALES table contains over 2.8 billion rows in the `SF100TCL` dataset. This massive volume is why we implement incremental loading patterns rather than processing the entire table repeatedly.
:::

#### Data Relationships

The tables are connected through a foreign key relationship:

- `STORE_SALES.SS_CUSTOMER_SK` → `CUSTOMER.C_CUSTOMER_SK`

This relationship allows us to:

- Enrich transactions with current customer information
- Track customer behavior over time
- Identify high-risk transactions based on customer profiles
- Generate comprehensive compliance reports

#### Compliance Relevance

These tables simulate common compliance scenarios:

- **Customer Due Diligence**: The CUSTOMER table provides identity information required for Know Your Customer (KYC) processes
- **Transaction Monitoring**: STORE_SALES enables detection of suspicious patterns and high-value transactions
- **Audit Trail**: The combination supports complete transaction lineage from customer to purchase

:::info[Processing Large Datasets]
This tutorial uses the TPC-DS 100 TB dataset which contains over 2.8 billion rows in the `STORE_SALES` table. We'll use a simulation pattern to demonstrate incremental loading without processing the entire dataset.
:::

## Setting Up Your Environment

Let's configure your Coalesce workspace with the necessary packages and storage locations for our compliance pipeline.

### Install Required Packages

1. Navigate to **Build Settings** > **Packages**.

2. Install the following packages:
   - [Snowflake Base Node Types][] (latest version)
   - [Snowflake Incremental Loading][] (latest version)

   <figure>
     
     <figcaption>Build Settings - Package Install</figcaption>
   </figure>

3. Give the packages a name. It's recommended to give them a name related to either the package name or use in your DAG.

### Configure Storage Locations

1. Navigate to **Build Settings** > **Storage Locations**.
2. Create three Storage Locations to organize your compliance data:

   | Storage Location | Purpose |
   |:---|:---|
   | `REGULATORY_RAW` | Source data ingestion |
   | `REGULATORY_STAGE` | Data transformation and processing |
   | `REGULATORY_MART` | Final compliance reports and analytics |

3. Go to **Workspace** and select the Workspace you're using.
4. Click **Storage Mappings**.
5. Update your Storage Mappings to match your database and schema.

<figure>
  
  <figcaption>The Storage Mappings in this example are the same as the schema.</figcaption>
</figure>

:::tip[Best Practice]
Separating data into distinct Storage Locations helps maintain clear data lineage and simplifies compliance auditing.
:::

## Connecting to Source Data

Now we'll connect to the Snowflake sample data that simulates our enterprise transaction system.

1. Click the **+** icon in the Build interface and select **Add Sources**.
2. Navigate to the `SNOWFLAKE_SAMPLE_DATA.TPCDS_SF100TCL` database or a database with a sample of data.
3. Select the following tables:

   - **CUSTOMER** - Customer data
   - **STORE_SALES** - Transaction records

4. Click **Add Sources** to import the tables into your pipeline.

   <figure>
     
     <figcaption>Add Sources</figcaption>
   </figure>

## Building the Customer Dimension

To maintain a complete audit trail of customer changes for SOX compliance, we'll implement a Type 2 Slowly Changing Dimension.

### Create the Dimension Node

1. Right-click the **CUSTOMER** source Node and select **Add Node > Base Node Type Package > Dimension**.

    <figure>
      
      <figcaption>Right-click on the Customer Node to add the Dimension Node.</figcaption>
    </figure>

2. Rename the Node to `DIM_CUSTOMER_MAIN`.
3. Configure the Node properties:
   - **Storage Location**: `REGULATORY_STAGE`

<figure>
  
  <figcaption>Set the Storage Location to REGULATORY_STAGE </figcaption>
</figure>

### Configure Change Tracking and Business Keys

1. In the **Options** section, go to Business Key.
2. Select the column:
   1. `C_CUSTOMER_SK`
3. Go to Change Tracking and select the following columns:
   1. `C_FIRST_NAME`
   2. `C_LAST_NAME`
   3. `C_EMAIL_ADDRESS`
   4. `C_PREFERRED_CUST_FLAG`

   

### Add Compliance Risk Assessment

You'll add a `compliance_risk_flag` column. This risk flag helps identify customers requiring additional review based on missing contact information or preferred status. You can optionally add an `audit_timestamp` if required for compliance.

1. In the **Mapping Grid**, scroll to the bottom and add a new column by clicking in the empty row.
2. Configure the new column:
   - **Column Name**: `compliance_risk_flag`
   - **Data Type**: `VARCHAR(10)`
   - **Transformation**:
  
   ```sql
    CASE
        WHEN "CUSTOMER"."C_PREFERRED_CUST_FLAG" = 'Y' THEN 'GREEN'
        WHEN "CUSTOMER"."C_EMAIL_ADDRESS" IS NULL THEN 'YELLOW'
        ELSE 'GREEN'
    END
   ```

3. Click **Create** to build the Dimension Node.

<figure>
  
  <figcaption>You can use the Column Editor tab to make it easy to create transformations</figcaption>
</figure>

## Implementing Incremental Processing

The `STORE_SALES` table contains billions of rows, making full table scans impractical. We'll implement an incremental loading pattern to process only new transactions.

### Understanding the Challenge

:::caution[Large Dataset Simulation]
The `STORE_SALES` table is massive (2.8+ billion rows) and static. To demonstrate incremental loading in this tutorial, we'll use a filtering pattern that simulates new data arriving over time.
:::

### Create the Filtering View

This view will control which data is visible to our incremental logic, simulating a real-world scenario where new data arrives continuously.

1. Right-click the **STORE_SALES** source Node and select **Add Node > Base Node Types Package > View**.
2. Rename the Node to `STG_SALES_FILTERED`.
3. Set the Storage Location to `REGULATORY_STAGE`.
4. In Options, enable **Override Create SQL**.
5. In the **Create SQL**, update the script to include the following WHERE clause: `WHERE "SS_SOLD_DATE_SK" < 2452000`.

   ```sql title="STG_SALES_FILTERED Create SQL Full Example"
    {{ stage('Override Create SQL') }}
        CREATE OR REPLACE VIEW {{ ref('REGULATORY_STAGE', 'STG_SALES_FILTERED')}} AS (
            SELECT 
    "SS_SOLD_DATE_SK" AS "SS_SOLD_DATE_SK",
    "SS_SOLD_TIME_SK" AS "SS_SOLD_TIME_SK",
    "SS_ITEM_SK" AS "SS_ITEM_SK",
    "SS_CUSTOMER_SK" AS "SS_CUSTOMER_SK",
    "SS_CDEMO_SK" AS "SS_CDEMO_SK",
    "SS_HDEMO_SK" AS "SS_HDEMO_SK",
    "SS_ADDR_SK" AS "SS_ADDR_SK",
    "SS_STORE_SK" AS "SS_STORE_SK",
    "SS_PROMO_SK" AS "SS_PROMO_SK",
    "SS_TICKET_NUMBER" AS "SS_TICKET_NUMBER",
    "SS_QUANTITY" AS "SS_QUANTITY",
    "SS_WHOLESALE_COST" AS "SS_WHOLESALE_COST",
    "SS_LIST_PRICE" AS "SS_LIST_PRICE",
    "SS_SALES_PRICE" AS "SS_SALES_PRICE",
    "SS_EXT_DISCOUNT_AMT" AS "SS_EXT_DISCOUNT_AMT",
    "SS_EXT_SALES_PRICE" AS "SS_EXT_SALES_PRICE",
    "SS_EXT_WHOLESALE_COST" AS "SS_EXT_WHOLESALE_COST",
    "SS_EXT_LIST_PRICE" AS "SS_EXT_LIST_PRICE",
    "SS_EXT_TAX" AS "SS_EXT_TAX",
    "SS_COUPON_AMT" AS "SS_COUPON_AMT",
    "SS_NET_PAID" AS "SS_NET_PAID",
    "SS_NET_PAID_INC_TAX" AS "SS_NET_PAID_INC_TAX",
    "SS_NET_PROFIT" AS "SS_NET_PROFIT"
    FROM {{ ref('REGULATORY_RAW', 'STORE_SALES_SAMPLE') }}
    WHERE "SS_SOLD_DATE_SK" < 2452000
    )
   ```

6. Click **Create** to build the view.

### Create the Incremental Logic

Now we'll add the high-water mark logic that identifies new records.

1. Right-click `STG_SALES_FILTERED` and select **Add Node > Incremental Loading > Incremental Load**.
2. Rename the Node to `INC_STORE_SALES_HWM`.
3. In the **Options** tab, configure:

   - **Materialization**: `View`
   - **Filter data based on Persistent Table**: `True`
   - **Persistent Table Location**: `REGULATORY_STAGE`
   - **Persistent Table Name**: `PSTG_STORE_SALES_HWM`
   - **Incremental Load Column**: `SS_SOLD_DATE_SK`

   

4. Navigate to the **Join** tab and click **Generate Join**.
5. Click **Copy to Editor** to populate the SQL.
6. Verify the generated SQL. You'll need to update `1900-01-01` to `0`.

   ```sql
   -- Find this line:
   SELECT COALESCE(MAX("SS_SOLD_DATE_SK"), '1900-01-01')
   
   -- Correct it to:
   SELECT COALESCE(MAX("SS_SOLD_DATE_SK"), '0')

   --Final SQL

    FROM{{ ref('REGULATORY_STAGE', 'STG_SALES_FILTERED') }} "STG_SALES_FILTERED"
    WHERE "STG_SALES_FILTERED"."SS_SOLD_DATE_SK" > 
    (SELECT COALESCE(MAX("SS_SOLD_DATE_SK"), '0') 
                FROM {{ ref_no_link('REGULATORY_STAGE', 'PSTG_STORE_SALES_HWM') }} )

   ```

7. **Create** the View.

:::important[Critical Fix]
The generated code may use a date string as the default value. Since `SS_SOLD_DATE_SK` is an integer, you must change this to `0` or the query will fail.
:::

### Create the Persistent Staging Table

This table stores the incrementally loaded transaction data.

1. Right-click `INC_STORE_SALES_HWM` and select **Add Node > Base Node Types Package > Persistent Stage**.
2. Rename the Node to `PSTG_STORE_SALES_HWM`.
3. Configure the Node:

   - **Storage Location**: `REGULATORY_STAGE`
   - **Business Key**:
     - `SS_SOLD_DATE_SK`
     - `SS_CUSTOMER_SK`
     - `SS_ITEM_SK`
     - `SS_TICKET_NUMBER`

#### Add Compliance Columns

1. In the **Mapping Grid**, add two custom columns:

   **Transaction Flag Column:**
   - **Column Name**: `transaction_flag`
   - **Data Type**: `VARCHAR(15)`
   - **Transformation**:

    ```sql
    CASE
        WHEN "INC_STORE_SALES_HWM"."SS_NET_PAID" > 10000 THEN 'AML_REVIEW'
        ELSE 'NORMAL'
    END
    ```

   **Audit Timestamp Column (optional):**
   - **Column Name**: `audit_processed_timestamp`
   - **Data Type**: `TIMESTAMP_NTZ`
   - **Transformation**:

    ```sql
    CURRENT_TIMESTAMP()
    ```

    :::info[System Generated Columns]

    You'll notice some columns that are automatically generated. These can be used for compliance or a custom column can be added such as `audit_processed_timestamp`. In this example, `SYSTEM_UPDATE_DATE` tracks the last modification to the record. Learn more about [System Columns][].

    :::

2. Click **Create** to build the staging table.

## Running the Initial Load

Let's perform the first run to load our partial dataset.

1. In Snowflake, ensure the target table is empty:

   ```sql
   TRUNCATE TABLE YOUR_DB.REGULATORY_STAGE.PSTG_STORE_SALES_HWM;
   ```

2. In Coalesce, click on `PSTG_STORE_SALES_HWM` and click **Run**.
3. Monitor the run progress. This will load all records where `SS_SOLD_DATE_SK < 2452000`.

:::tip[Performance Check]
Note the execution time and row count. This establishes your baseline for comparing incremental load performance.
:::

## Creating the Regulatory Fact Table

This fact table combines transaction data with current customer information for compliance analysis.

### Build the Fact Node

1. Right-click `PSTG_STORE_SALES_HWM` and select **Add Node >  Base Node Types Package > Fact**.
2. Rename the Node to `FCT_STORE_SALES_SAMPLE`.
3. Configure the Node:
   - **Storage Location**: `REGULATORY_MART`
   - **Business Key**:
     - `SS_SOLD_DATE_SK`
     - `SS_CUSTOMER_SK`
     - `SS_ITEM_SK`
     - `SS_TICKET_NUMBER`

### Join Customer Dimension

Navigate to the **Join** tab and enter:

```sql
FROM {{ ref('REGULATORY_STAGE','PSTG_STORE_SALES_HWM') }} "PSTG_STORE_SALES_HWM"
JOIN {{ ref('REGULATORY_STAGE','DIM_CUSTOMER_MAIN') }} "DIM_CUSTOMER_MAIN"
ON "PSTG_STORE_SALES_HWM"."SS_CUSTOMER_SK" = "DIM_CUSTOMER_MAIN"."C_CUSTOMER_SK"
AND "DIM_CUSTOMER_MAIN"."SYSTEM_END_DATE" = '2999-12-31 00:00:00'
```

:::tip[Current Records Only]
The JOIN condition on `SYSTEM_END_DATE` ensures we only use the current version of each customer record.
:::

### Add Risk Assessment

1. In the **Mapping Grid**, add a custom column:

   - **Column Name**: `customer_risk_assessment`
   - **Data Type**: `VARCHAR(20)`
   - **Transformation**:

   ```sql
    CASE
        WHEN "DIM_CUSTOMER_MAIN"."compliance_risk_flag" = 'YELLOW' AND "PSTG_STORE_SALES_HWM"."SS_NET_PAID" > 10000 THEN 'HIGH_RISK'
        WHEN "PSTG_STORE_SALES_HWM"."SS_NET_PAID" > 10000 THEN 'MEDIUM_RISK'
        ELSE 'LOW_RISK'
    END
   ```

2. There are duplicate `SYSTEM` columns. Delete the `SYSTEM_CREATE_DATE` and the `SYSTEM_UPDATE_DATE` automatically created in the `FCT_STORE_SALES_SAMPLE` table by right-clicking and Delete Column. You are keeping the ones from the Node `PSTG_STORE_SALES_HWM` to keep the audit history. They would only track when they were inserted into the Fact table.

    <figure>
      
      <figcaption>Delete the SYSTEM_CREATE_DATE and SYSTEM_UPDATE_DATE.</figcaption>
    </figure>

3. Click **Create** and **Run** to build the Fact table.

## Build the Compliance Reports

### Customer 360 View

Create a comprehensive customer profile with transaction history.

1. Right-click `DIM_CUSTOMER_MAIN` and select **Add Node > Base Node Types Packages > View**.
2. Rename to `CUSTOMER_360_VIEW`.
3. Change the Storage Location to `REGULATORY_MART`.
4. In the **Join** tab, add:

   ```sql
    FROM {{ ref('REGULATORY_STAGE','DIM_CUSTOMER_MAIN') }} "DIM_CUSTOMER_MAIN"
    LEFT JOIN {{ ref('REGULATORY_MART','FCT_STORE_SALES_SAMPLE') }} "FCT_STORE_SALES_SAMPLE"
    ON "DIM_CUSTOMER_MAIN"."C_CUSTOMER_SK" = "FCT_STORE_SALES_SAMPLE"."SS_CUSTOMER_SK"
    WHERE "DIM_CUSTOMER_MAIN"."SYSTEM_END_DATE" = '2999-12-31 00:00:00'
   ```

5. In **Options**, toggle **Group By All** to `True`.
6. Add aggregation columns with the transformations in the **Mapping Grid**:

    ```sql
    total_transactions (NUMBER)-  COUNT(FCT_STORE_SALES_SAMPLE.SS_TICKET_NUMBER)

    total_amount (NUMBER) - COALESCE(SUM(FCT_STORE_SALES_SAMPLE.SS_NET_PAID), 0)

    high_risk_transaction_count (VARCHAR(10)) - COUNT(CASE WHEN "FCT_STORE_SALES_SAMPLE"."customer_risk_assessment" = 'HIGH_RISK' THEN 1 END)
    ```

7. Create the Node.

### Exception Detection View

Monitor high-risk transactions requiring regulatory review.

1. Right-click `FCT_STORE_SALES_SAMPLE` and select **Add Node > Base Node Types Package > View**.
2. Rename to `EXCEPTION_DETECTION_VIEW`.
3. Change the Storage Location to `REGULATORY_MART`.
4. In the **Join** tab, add the WHERE clause:

   ```sql
    FROM {{ ref('REGULATORY_MART', 'FCT_STORE_SALES_SAMPLE') }} "FCT_STORE_SALES_SAMPLE"
    WHERE "FCT_STORE_SALES_SAMPLE"."customer_risk_assessment" = 'HIGH_RISK'
    OR "FCT_STORE_SALES_SAMPLE"."SS_NET_PAID" > 10000
   ```

5. Create the Node.

### Monthly Regulatory Report

Create an audit report snapshot for compliance reporting.

1. Right-click `FCT_STORE_SALES_SAMPLE` and select **Add Node > Base Node Types Packages > Fact**.
2. Rename to `REG_MONTHLY_REPORTS`.
3. Delete all columns, and add the following columns.

   | Column Name | Transformation | Data Type |
   |:---|:---|:---|
   | `report_date` | `DATE_TRUNC('MONTH', CURRENT_DATE())` | `DATE` |
   | `customer_risk_category` | `"FCT_STORE_SALES_SAMPLE"."customer_risk_assessment"` | `VARCHAR(20)` |
   | `customer_count` | `COUNT(DISTINCT "FCT_STORE_SALES_SAMPLE"."SS_CUSTOMER_SK")` | `NUMBER` |
   | `transaction_count` | `COUNT(*)` | `NUMBER` |
   | `total_amount` | `SUM("FCT_STORE_SALES_SAMPLE"."SS_NET_PAID")` | `NUMBER(38,2)` |
   | `report_generated_timestamp` | `CURRENT_TIMESTAMP()` | `TIMESTAMP_NTZ` |

4. Configure the **Business Key**:
   - `report_date`
   - `customer_risk_category`

5. In the **Join** tab:

   ```sql
    FROM {{ ref('REGULATORY_MART', 'FCT_STORE_SALES_SAMPLE') }} "FCT_STORE_SALES_SAMPLE"
    GROUP BY 1, 2
   ```

6. Create the Node.

:::tip[Why a Fact Node?]
Regulatory reports must be static, historical snapshots. A Fact Node materializes the data, creating a permanent record for each reporting period. Views would show live data, which isn't suitable for audit purposes.
:::

## Testing the Incremental Load

Now let's simulate new data arriving and verify our incremental processing works correctly.

### Run the Pipeline

1. Return to the `STG_SALES_FILTERED` view.
2. Update the SQL to change the filter. This is to prevent all the rows being processed, but simulating new data.

   ```sql
    WHERE "SS_SOLD_DATE_SK" < 2452100
   ```

3. Click **Create**.
4. Go to the DAG and click **Run All**.
5. Verify the new data appears in your reports by opening one of the reports and click on Fetch Data if the data isn't already loaded.

## Conclusion and Next Steps

Congratulations. You've built a production-ready compliance pipeline that demonstrates enterprise data engineering best practices.

### What You've Accomplished

- Created an efficient incremental loading system
- Implemented audit-compliant customer history tracking
- Built automated compliance reporting
- Established a scalable pipeline architecture

### Next Steps

- Explore adding [data quality checks][] using Coalesce's testing framework
- Implement [automated scheduling][] for regular report generation
- Add additional compliance rules based on your organization's needs
- Consider integrating with external compliance systems

[Snowflake Base Node Types]: https://docs.coalesce.io/docs/marketplace/package/coalesce_snowflake_base-node-types
[Snowflake Incremental Loading]: /docs/marketplace/package/coalesce_snowflake_incremental-loading
[System Columns]: https://docs.coalesce.io/docs/build-your-pipeline/user-defined-Nodes/Node-definition#system-columns
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[automated scheduling]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce
[data quality checks]: /docs/build-your-pipeline/data-quality-testing
[database]: https://app.snowflake.com/marketplace/listing/GZSTZTP0KNB/snowflake-tpc-ds-10tb?search=TPCDS_SF100TCL

---

## Building Pipelines with Iceberg and Cortex in Coalesce

## Overview

This entry-level Hands-On Lab exercise is designed to help you master the basics of Coalesce. In this lab, you’ll explore the Coalesce interface, learn how to easily transform and model your data with our core capabilities, understand how to deploy and refresh version-controlled data pipelines, and build a ML Forecast node that forecasts sales values for a fiction food truck company.

### **What You’ll Need**

* A Snowflake [trial account](https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=CjwKCAiAk--dBhABEiwAchIwkYotxMl4v5sRhNMO6LxCXUTaor9yH2mAbOwQHVO0ZNJQRuNyHrmYwRoCPucQAvD_BwE)
* A Coalesce trial account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account)
* Basic knowledge of SQL, database concepts, and objects
* The [Google Chrome browser](https://www.google.com/chrome/)

### **What You’ll Build**

* A Directed Acyclic Graph (DAG) representing a basic star schema in Snowflake

### **What You’ll Learn**

* How to navigate the Coalesce interface
* How to add data sources to your graph
* How to prepare your data for transformations with Stage nodes
* How to join tables
* How to apply transformations to individual and multiple columns at once
* How to build out Dimension and Fact nodes
* How to make changes to your data and propagate changes across pipelines
* How to configure and build an ML Forecast node

By completing the steps we’ve outlined in this guide, you’ll have mastered the basics of Coalesce and can venture into our more advanced features.

---

## About Coalesce

Coalesce is the first cloud-native, visual data transformation platform built for Snowflake. Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

With Coalesce, you can:

* Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
* Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Snowflake functionality
* Analyze the impact of changes to pipelines with built-in data lineage down to the column level
* Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
* Ensure consistent data standards and governance across pipelines, with data never leaving your Snowflake instance

### How Is Coalesce Different?

Coalesce’s unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### Core Concepts In Coalesce

Coalesce currently only supports Snowflake as its target database, As you will be using a trial Coalesce account created with [Snowflake Marketplace][], your basic database settings will be configured automatically and you can instantly build code.

### **Organization**

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

### **Projects**

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Git repository and Snowflake account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

### **Workspaces vs. Environments**

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Snowflake schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, PreProd, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

## Lab Use Case

As the lead Data & Analytics manager for TastyBytes Food Trucks, you're responsible for building and managing data pipelines that deliver insights to the rest of the company. There customer-related questions that the business needs to answer that will help with inventory planning and marketing. Included in this, is building a machine learning forecast that will allow management to determine sales volume for each item on the menu.

In order to help your extended team answer these questions, you’ll need to build a customer data pipeline first.

---

## Before You Start

To complete this lab, please create free trial accounts with Snowflake and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

We recommend using Google Chrome as your browser for the best experience.

| Note: Not following these steps will cause delays and reduce your time spent in the Coalesce environment. |
| :---- |

### Step 1: Create a Snowflake Trial Account

1. Fill out the Snowflake trial account form [here](https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=Cj0KCQiAtvSdBhD0ARIsAPf8oNm6YH7UeRqFRkVeQQ-3Akuyx2Ijzy8Yi5Om-mWMjm6dY4IpR1eGvqAaAg3MEALw_wcB). Use an email address that is not associated with an existing Snowflake account.

2. When signing up for your Snowflake account, select the region that is physically closest to you and choose Enterprise as your Snowflake edition. Please note that the Snowflake edition, cloud provider, and region used when following this guide do not matter.

   ![image1](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image1.png)

3. After registering, you will receive an email from Snowflake with an activation link and URL for accessing your trial account. Finish setting up your account following the instructions in the email.

### Step 2: Create a Coalesce Trial Account from Snowflake Marketplace

1. Sign in to Snowsight. Make sure you are using the ACCOUNTADMIN role (or a role with privileges to create a database, warehouse, user, and role) and that your Snowflake email address is verified.
2. In the navigation menu, select **Marketplace > Snowflake Marketplace** and search for Coalesce.

      

3. Open the Coalesce listing and select Start free trial in the upper right. To see what the trial includes, review the Your free trial tab.
4. In the Connect to Coalesce dialog, review the information to be shared and the Snowflake objects that will be created, then select Connect to Coalesce.
5. You will be redirected to Coalesce to finish activating your trial account. Fill in your information to complete activation.

Congratulations! You've successfully created your Coalesce trial account.

### Step 3: Set Up The Dataset

1. We will be using a M Warehouse size within Snowflake for this lab. You can upgrade this within the admin tab of your Snowflake account.

   ![image6](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image6.png)

2. In your Snowflake account, click on the Worksheets Tab in the left-hand navigation bar.

   ![image7](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image7.png)

3. Within Worksheets, click the "+" button in the top-right corner of Snowsight and choose "SQL Worksheet.”

   ![image8](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image8.png)

4. Navigate to the [Cortex HOL Data Setup File](https://github.com/coalesceio/hands-on-labs/blob/main/cortex-hol/setup_guide.sql) that is hosted on GitHub.

5. Within GitHub, navigate to the right side and click "Copy raw contents". This will copy all of the required SQL into your clipboard.

   ![image9](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image9.png)

6. Paste the setup SQL from GitHub into your Snowflake Worksheet. Then click inside the worksheet and select All (*CMD + A for Mac or CTRL + A for Windows*) and Click "► Run".

7. After clicking "► Run" you will see queries begin to execute. These queries will run one after another with the entire worksheet taking around 5 minutes. Upon completion you will see a message that states your HOL data setup is now complete.

Once you’ve activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started. Click on the Launch button next to your Development Workspace to get started.

   ![image10](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image10.png)

### Step 4: Add the Cortex and Iceberg Package from the Coalesce Marketplace

You will need to add the ML Forecast node into your Coalesce workspace in order to complete this lab.

1. Launch your workspace within your Coalesce account

    ![image11](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image11.png)

2. Navigate to the build settings in the lower left hand corner of the left sidebar

   ![image12](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image12.png)

3. Select Packages from the Build Settings options

   ![image13](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image13.png)

4. Select Browse to Launch the Coalesce Marketplace

    ![image14](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image14.png)

5. Select Find out more within the Cortex package

   ![image15](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image15.png)

6. Copy the package ID from the Cortex page

   ![image16](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image16.png)

7. Back in Coalesce, select the Install button:

   ![image17](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image17.png)

8. Paste the Package ID into the corresponding input box:

   ![image18](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image18.png)

9. Give the package an Alias, which is the name of the package that will appear within the Build Interface of Coalesce. And finish by clicking Install.

   ![image19](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image19.png)

10. Follow these same steps to install the Iceberg package.

## Navigating the Coalesce User Interface

| About this lab: Screenshots (product images, sample code, environments) depict examples and results that may vary slightly from what you see when you complete the exercises. This lab exercise does not include Git (version control). Please note that if you continue developing in your Coalesce account after this lab, none of your work will be saved or committed to a repository unless you set up before developing.   |
| :---- |

Let's get familiar with Coalesce by walking through the basic components of the user interface.

​​Once you've activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started.

   ![image20](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image20.png)
   ![image21](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image21.png)

1. Launch your Development Workspace by clicking the Launch button and navigate to the Build Interface of the Coalesce UI. This is where you can create, modify, and publish nodes that will transform your data.

   Nodes are visual representations of objects in Snowflake that can be arranged to create a Directed Acyclic Graph (DAG or Graph). A DAG is a conceptual representation of the nodes within your data pipeline and their relationships to and dependencies on each other.

2. In the Browser tab of the Build interface, you can visualize your node graph using the Graph, Node Grid, and Column Grid views. In the upper right hand corner, there is a person-shaped icon where you can manage your account and user settings.

   ![image22](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image22.png)

3. By clicking the question mark icon, you can access the Resource Center to find product guides, view announcements, request support, and share feedback.

   ![image24](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image24.png)

4. The Browser tab of the Build interface is where you’ll build your pipelines using nodes. You can visualize your graph of these nodes using the Graph, Node Grid, and Column Grid views. While this area is currently empty, we will build out nodes and our Graph in subsequent steps of this lab.

   ![image22](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image22.png)

5. Next to the Build page is the Deploy page. This is where you will push your pipeline to other environments such as Testing or Production.

   ![image25](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image25.png)

6. Next to the Deploy page is the Docs page. Documentation is an essential step in building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. To help with this, Coalesce automatically generates and updates documentation as you work.

   ![image26](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image26.png)

## Configure Storage Locations and Mappings

Before you can begin transforming your data, you will need to configure storage locations. Storage locations represent a logical destination in Snowflake for your database objects such as views and tables.

1. To add storage locations, navigate to the left side of your screen and click on the Build settings cog.

   ![image27](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image27.png)

2. Click the “Add New Storage Locations” button and name it SALES for your raw customer data.

   ![image28](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image28.png)

3. Now map your storage location to their logical destinations in Snowflake. In the upper left corner next to your Workspace name, click on the pencil icon to open your Workspace settings. Click on Storage Mappings and map your SALES location to the CORTEX_HOL database and the RAW_POS schema. Select Save to ensure Coalesce stores your mappings.

   ![image29](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image29.png)

4. Go back to the build settings and click on Node Types and then click the toggle button next to View to enable View node types. Now you’re ready to start building your pipeline.

   ![image30](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image30.png)

## Adding Data Sources

Let’s start to build the foundation of your data pipeline by creating a Graph (DAG) and adding data in the form of Source nodes.

1. Start in the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and select Add Sources.

   **![image31](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image31.png)**

2. Select all of the tables from the SALES storage location. You can click on the check box next to the storage location name to select all of the tables.

   ![image32](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image32.png)

3. You'll now see your graph populated with your Source nodes. Note that they are designated in red. Each node type in Coalesce has its own color associated with it, which helps with visual organization when viewing a Graph.

   ![image33](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image33.png)

4. Next, we'll load an Iceberg table for our final table source. Select the + incon again and nvaigate to Add Node > Iceberg > External Iceberg Table.

   ![image125](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image125.png)

5. Within the node, open the ICeberg Options configuration on the right side of the screen.

   ![image126](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image126.png)

6. First, change the type of catalog to **Object storage**, as the Iceberg file we'll be obtaining is currently in AWS S3.

   ![image127](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image127.png)

7. Within the setup code, you created an external volume to connect to the AWS S3 bucket. Input the name of that volume here: **cortex_iceberg_external_volume**.

   ![image128](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image128.png)

8. Next, you'll pass the name of the catalog integration that was created in the setup code as well: **cortex_iceberg_catalog_integration**.

   ![image129](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image129.png)

9. Finally, the metadata path for the file is included here. Paste this code into the parameter:

   ```SQL
   iceberg/metadata/00002-2c8a53e5-9fae-4f92-9f62-f914b3b680be.metadata.json
   ```

   ![image130](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image130.png)

10. Toggle off the Schedule refresh toggle, as we will be working with a static dataset.

      ![image131](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image131.png)

11. Select Create and Run to create the Iceberg table as an object in Snowflake and populate it with data.

12. With the table populated, select the Re-Sync Columns button to obtain the new columns in the table, replacing any previous, unneeded columns. Select OK to apply the column changes.

   ![image132](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image132.png)

## Creating Stage Nodes

Now that you’ve added your Source nodes, let’s prepare the data by adding business logic with Stage nodes. Stage nodes represent staging tables within Snowflake where transformations can be previewed and performed. Let's start by adding a standardized "stage layer" for all sources.

1. Press and hold the Shift key to multi-select all of your Source nodes from the node pane on the left side of your screen. Then right click and select Add Node > Stage from the drop down menu. You will now see that all of your Source nodes have corresponding Stage tables associated with them. If the 14M/38M data sources are being used, drop that suffix in the Node name in this case STG_ORDER_HEADER

   ![image34](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image34.png)

2. Now change the Graph to a Column Grid by using the top left dropdown menu. By using the Column Grid view, you can search and group by different column headers.

   ![image35](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image35.png)

3. Click on the Storage Location header and drag it to the sorting bar below the View As dropdown menu. You have now grouped your rows by Storage Location. Then expand all the columns from your SALES location.

   ![image36](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image36.png)

4. Change back to the Graph view from Column Grid to with the dropdown menu.

   ![image37](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image37.png)

5. Now let’s create the foundation for your data pipeline. In the upper right hand corner, press the Create All button to create your tables in Snowflake. Then click the Run All button to populate your nodes with data.

   ![image38](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image38.png)

## Exploring the Node Editor

Your Node Editor is used to edit the node object, the columns within it, and its relationship to other nodes. There are a few different components to the Node Editor.

1. Double click on your STG_CUSTOMER node to open up your Node Editor. The large middle section is your Mapping grid, where you can see the structure of your node along with column metadata like transformations, data types, and sources.

   ![image39](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image39.png)

2. On the right hand side of the Node Editor is the Config section, where you can view and set configurations based on the type of node you’re using.
      ![image40](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image40.png)

3. At the bottom of the Node Editor, press the arrow button to view the Data Preview pane.

   ![image41](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image41.png)

## Applying Single Column Transformations

Now let’s apply a single column transformation in your STG_CUSTOMER node by applying a consistent naming convention to our customer names. This will ensure that any customer names are standardized.

1. Select the FIRST_NAME column and click the blank Transform field in the Mapping grid. Enter the following transformation and then press Enter:

   ```SQL
   UPPER({{SRC}})
   ```

   ![image42](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image42.png)

2. Repeat this transformation for your LAST_NAME column:

   ```SQL
   UPPER({{SRC}})
   ```

   ![image43](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image43.png)

3. Click the Create and Run buttons to create and repopulate your node.

   ![image44](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image44.png)

## Joining Nodes

In order to prepare the data within your data pipeline, it needs to be processed more with an additional stage layer.

1. Return to the Browser tab and select your STG_ORDER_HEADER data with your STG_ORDER_DETAIL data by holding down the Shift key and clicking on both nodes.

   ![image45](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image45.png)

2. Right click on either node and select Join Nodes, and then select Stage. This will bring you to the node editor of your new Stage node.

   ![image46](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image46.png)

3. Select the Join tab in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement.

   ![image47](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image47.png)

4. Add column relationships in the last line of the existing Join statement using the code below:

   ```SQL
      FROM {{ ref('WORK', 'STG_ORDER_HEADER') }} "STG_ORDER_HEADER"
      INNER JOIN {{ ref('WORK', 'STG_ORDER_DETAIL') }} "STG_ORDER_DETAIL"
      ON "STG_ORDER_HEADER"."ORDER_ID" = "STG_ORDER_DETAIL"."ORDER_ID"
   ```

   **![image48](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image48.png)**

5. Since both nodes contain a ORDER_ID and DISCOUNT_ID columns, we will need to rename or remove one of the columns to avoid getting a duplicate column name error. Let’s delete the ORDER_ID and DISCOUNT_ID from the STG_ORDER_DETAIL.

   ![image49](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image49.png)

6. You can rename the node by clicking on the pencil icon next to the node name and give the node a name of your choice. Rename the node to STG_ORDER_MASTER.

   ![image50](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image50.png)

7. To create your Stage Node, click the Create button in the lower half of the screen. Then click the Run button to populate your STG_ORDER_MASTER node and preview the contents of your node by clicking Fetch Data.

   ![image51](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image51.png)

## Creating Dimension Nodes

Now let’s experiment with creating Dimension nodes. These nodes are generally descriptive in nature and can be used to track particular aspects of data over time (such as time or location). Coalesce currently supports Type 1 and Type 2 slowly changing dimensions, which we will explore in this section.

1. Select STG_CUSTOMER and right click, create Dimension node. Because the demographics of your customer base changes over time, creating a Dimension node on top of this data will allow you to capture customer changes over time.

   ![image52](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image52.png)

2. Within the Option configuration section, under Business Key, check the box next to CUSTOMER_ID and press the > button to select it as your business key.

   ![image53](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image53.png)

3. To create a Type 2 dimension, select the FIRST_NAME, LAST_NAME, EMAIL, PHONE_NUMBER columns under Change Tracking. This will allow you to track changes to these columns over time.

   **![image54](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image54.png)**

4. In the lower right of your screen, click the Create button to create your Dimension node and then the Run button to populate it with data.

   ![image55](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image55.png)

5. Now let’s create a Type 1 dimension which will help you focus on particular customer details. Navigate back to the Browser tab and select your STG_MENU node. Right click and select Add Node > Dimension from the dropdown menu.

   ![image56](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image56.png)

6. In your DIM_MENU node, navigate to the right and expand the Options drop down in the Config section. Under Business Key, check the box next to MENU_ITEM_ID and press the > button to select it as your business key.

   ![image57](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image57.png)

7. In the lower right of your screen, click the Create button to create your Dimension node and then the Run button to populate it with data.

   ![image58](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image58.png)

## Creating A Fact Node

Fact nodes represent Coalesce's implementation of a Kimball Fact Table, which consists of the measurements, metrics, or facts of a business process and is typically located at the center of a star or Snowflake schema surrounded by dimension tables.

Let’s create a Fact table to allow us to query the metrics surrounding our customer orders.

1. In your Browser tab, select the STG_ORDER_MASTER node. Then right click and select Add Node > Fact.

   ![image59](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image59.png)

2. Navigate to the right and expand the Options drop down in the Config section. Under Business Key, check the boxes next to ORDER_DETAIL_ID. Then press the > button to select them as your business key.

   ![image60](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image60.png)

3. Click the Create and Run buttons to run and populate your FCT_ORDER_MASTER node.

   ![image61](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image61.png)

## Bulk Editing and Propagating Columns Across Pipelines

While you were building your pipeline, you may have noticed there was a VARIANT column containing JSON data in our STG_MENU node. You’ll want to use this data to enrich your nodes and make more informed decisions. This means you first need to derive the JSON mappings from the VARIANT column into individual columns. Then, you need to propagate those columns to all of your downstream nodes in order to use them for decision making.

1. Within the STG_MENU node, select the MENU_ITEM_HEALTH_METRICS_OBJ column. You’ll notice the Data type is VARIANT. Right click on the column and select Derive Mappings > From JSON. Coalesce will automatically parse this data into their own separate columns.

   ![image62](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image62.png)

2. The columns contain a prefix inherited from the VARIANT column. You can bulk edit these columns and quickly update the name to remove the unneeded prefix.

   ![image63](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image63.png)

3. Select the `menu_item_health_metrics_ingrediants` column, hold shift, and select `menu_item_id`. This will highlight all of the columns you just parsed. Right click on any of the columns and select Bulk Edit.

   ![image64](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image64.png)

4. In the Column Editor on the right side of the screen, select Column Name from the Select an attribute area.

   ![image65](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image65.png)

5. Within the code editor, write the following code:

      ```SQL
      {{ column.name | replace("menu_item_health_metrics_", "") | upper }}
      ```

    ![image66](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image66.png)

6. By selecting preview, you can view the changes that will be made to your column. Select Update to make the changes.

    ![image67](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image67.png)

7. You will now have two MENU_ITEM_ID columns in the node. Delete the column you just renamed.

    ![image68](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image68.png)

8. Now that you have derived our mappings from a JSON column and have bulk edited them to be more understandable, you can propagate those columns to all of the applicable downstream nodes.
9. Select the ingredients column, hold down shift, and select the IS_NUT_FREE_FLAG column to highlight all of the columns you want to propagate.

    ![image69](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image69.png)

10. Right click on any of the columns and select View Column Lineage.

    ![image70](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image70.png)

11. You can see all of the columns highlighted with the STG_MENU node. Click on the ellipses next to any of the columns and select Propagate Addition.

    ![image71](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image71.png)

12. Select the check box next to DIM_MENU in order to propagate the columns to the node.

    ![image72](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image72.png)

13. Select Preview and then Apply to finalize these changes. You can confirm the warning about making changes without first committing your work to Git.

   ![image73](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image73.png)

## Using Cortex Nodes for Processing Data

Now that we have our fact and dimension nodes, we are going to process the data from the STG_REVIEWS table, allowing us to create a sentiment score for the reviews from each of our customers.

1. Right click on the STG_REVIEWS and select Add Node -> CortexML -> Cortex Functions.

   ![image74](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image74.png)

2. Within the LLM_REVIEWS node, open the Cortex Package dropdown in the Config settings of the node.

   ![image75](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image75.png)

3. We are going to create a sentiment score based on the reviews left by our customers. These reviews are found in the REVIEW_TEXT column. Duplicate the REVIEW_TEXT column to keep the original text, as the sentiment score we will create, will be in a new column.

   ![image76](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image76.png)

4. Call the new column REVIEW_SENTIMENT. You can do this by double clicking on the new column name.

   ![image77](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image77.png)

5. Select the Sentiment toggle on the right side of the node in the Cortex Package configuration options. Then, select the REVIEW_SENTIMENT column we just created.

   ![image78](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image78.png)

6. And that’s it. Coalesce will handle the rest. Select Create and Run to create the object and populate it with data, which will include the sentiment score.

   ![image79](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image79.png)

7. Next, we’ll process this data a bit more, so we can see the average review and sentiment for our customers. Right click on the LLM_REIVEWS node and select Add Node -> Stage.

   ![image80](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image80.png)

8. Rename the node to STG_CUSTOMER_SENTIMENT.

   ![image81](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image81.png)

9. Next, remove the following columns from the node:
   * REVIEW_ID
   * REVIEW_TEXT
   * REVIEW_DATE

   ![image82](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image82.png)

10. We want to know the average sentiment and star rating for each of our customers, as many of our customers have left multiple reviews. We can use the Coalesce bulk editing transformation functionality for this. Select the STAR_RATING and REVIEW_SENTIMENT columns and right click on either one and select Bulk Edit.

    ![image83](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image83.png)

11. In the attributions section, select Transform, as this is the operation we want to apply in bulk.

    ![image84](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image84.png)

12. In the transform SQL editor, copy and paste the following snippet of code.

      ```SQL
      AVG({{SRC}})
      ```

      ![image85](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image85.png)

13. Select the Preview button to view the changes and then select Update to apply the changes to your node.

      ![image86](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image86.png)

14. Finally, since we are using aggregate functions in SQL, we need to use a GROUP BY for our non-aggregated column. In the Join SQL Editor, start a new line and supply the following SQL:
    GROUP BY ALL

    ![image87](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image87.png)

15. Now Create and Run the node.
16. Navigate back to the Browser and now we can join the DIM_CUSTOMER node to the STG_CUSTOMER_SENTIMENT node we just created. Select both nodes and right click on either node and select Join Nodes -> View.

    ![image88](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image88.png)

17. Rename the node to V_CUSTOMER.
18. Navigate to the Join table to configure the join. In this case, we need to join on CUSTOMER_ID. Use the code below to configure the join:

      ```SQL
      FROM {{ ref('WORK', 'STG_CUSTOMER_SENTIMENT') }} "STG_CUSTOMER_SENTIMENT"

      LEFT JOIN {{ ref('WORK', 'DIM_CUSTOMERS') }} "DIM_CUSTOMERS"

      ON "STG_CUSTOMER_SENTIMENT"."CUSTOMER_ID" = "DIM_CUSTOMERS"."CUSTOMER_ID"
      ```

    ![image89](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image89.png)

19. Back in the mapping grid, remove one of the CUSTOMER_ID columns, as the table can’t contain two of the same named column.
20. Select Create when ready to create your view. Congratulations. You now have a view that users can query to learn about the sentiment of your customers.

## Creating ML Processing Node

1. Open the DIM_MENU node and update the business key by adding ingredients as a column. Since we added this as a column in the previous section, we need to update the node. Create and run the node.

   ![image90](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image90.png)

2. To finish preparing your data for ML forecasting, we’ll perform one more join. Navigate back to the Browser and select the FCT_ORDER_MASTER node and the DIM_MENU node by holding down the shift key and clicking on both nodes. Right click on either node and select Join Nodes, and then select Stage. This will bring you to the node editor of your new Stage node.

   ![image91](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image91.png)

3. Select the Join tab in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement. Add column relationships in the last line of the existing Join statement using the code below:

      ```SQL
      FROM {{ ref('WORK', 'DIM_MENU') }} "DIM_MENU"
      INNER JOIN {{ ref('WORK', 'FCT_ORDER_MASTER') }} "FCT_ORDER_MASTER"
      ON "DIM_MENU"."MENU_ITEM_ID" = "FCT_ORDER_MASTER"."MENU_ITEM_ID"
      ```

   **![image92](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image92.png)**

4. You can rename the node by clicking on the pencil icon next to the node name and give the node a name of your choice. Rename the node STG_ORDER_MASTER_ITEMS.

   ![image93](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image93.png)

5. Back in the mapping grid, delete one of the duplicate `MENU_ITEM_ID` columns, as well as all the `SYSTEM_` columns and then create and run the node.

   ![image94](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image94.png)

## Transforming Data in a Node

Now we will prepare the data for our ML forecast by selecting the columns we want to work with and applying single column transformations to that data.

The ML Forecast node we will use later in the lab will rely on three columns to produce a forecast:

* Series column
* Timestamp column
* Target column

We will structure our  STG_ORDER_MASTER_ITEMS node so that it only contains the columns we will need for our ML Forecast Node.

* MENU_ITEM_NAME -> Series column
* ORDER_AMOUNT -> Target column
* ORDER_TS -> Timestamp column

1. To structure the node with the columns that we need, the first thing we will do is rename the columns we want to keep. Select the ORDER_AMOUNT, ORDER_TS, and MENU_ITEM_NAME  columns, and right click on either of the columns and select, Bulk Edit.

   ![image95](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image95.png)

2. In the Bulk Editor, select the Column Name attribute and copy and paste the code below to apply a FORECAST prefix to each of the columns we want to keep. Select Preview -> Update to rename the columns.

      ```SQL
      FORECAST_{{column.name}}
      ```

   ![image96](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image96.png)

3. Click on the Column Name column header in the mapping grid to sort the column names in alphabetical order. Using the shift key, select all of the columns that do not contain a `FORECAST_` prefix, right click on them, and delete them.

   ![image97](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image97.png)

4. With the columns we need for our forecast in this node, we will now apply some column transformations. Currently the Timestamp column records values down to the second interval. For this lab, we want to predict order values at the day level. Let’s add this transformation.
   Select the FORECAST_ORDER_TS column and click into the transform field in the mapping grid. You can cast a timestamp as a date, removing the seconds, minutes, and hours, by using the transformation below:

      ```SQL
      {{SRC}}::date
      ```

   ![image98](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image98.png)

5. Next, we need to aggregate our sales at the day level. We can use a sum function for this. You can use the transformation here to sum the ORDER_AMOUNT column.

      ```SQL
      sum({{SRC}})
      ```

   ![image99](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image99.png)

6. We will leave the MENU_ITEM_NAME column as it is. However, because we have included an aggregate function, we will need to use a GROUP BY in the node. Navigate to the JOIN tab so we can finish preparing this node.

   ![image100](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image100.png)

7. The first thing we will do is add a filter to this node, as the dataset we are working with is quite large and has dozens of item names to predict on. We will filter the dataset to allow our ML Forecast node to run faster.
   To do this, within the editor, hit the enter key and add a new line to the editor. Then, type WHERE to add a filter clause.

   For the conditions in our filter, we will first filter where the ORDER_TS column is greater than or equal to 2022-01-01. We will also add a filter where the ITEM_CATEGORY is equal to Beverage. You can copy and paste the code for this filter below:

      ```SQL
      WHERE "ORDER_TS" >= '2022-01-01' and "ITEM_CATEGORY" = 'Beverage'
      ```

   ![image101](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image101.png)

8. Staying in the Join tab, we will add a GROUP BY statement. Within the editor, add a new line by hitting the enter key. Next, type GROUP BY within the editor, followed by the two non-aggregated columns, MENU_ITEM_NAME and ORDER_TS. You can copy the code for this below:
   GROUP BY ALL

   ![image102](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image102.png)

9. Create and Run the node to populate it with data for the next section of the lab guide.

## Adding an ML Forecast Node

As the final step of building out this data pipeline for getting your ML Forecast Node operational, we will add in the ML Forecast Node to the pipeline.

1. Navigate back to the Browser to see your data pipeline. Select the STG_ORDER_MASTER_ITEMS node that we have been working with and right click on it. Select Add Node -> CortexML -> Forecast.

   ![image103](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image103.png)

2. Within the node, there will be three additional columns that have been added, FORECAST, LOWER_BOUND, and UPPER_BOUND. These are generated automatically by the ML Forecast node and will be used to forecast the values of the configured node.

   Within the config section, select the Forecast Model Input dropdown. This is where we will configure the node to forecast the values we want to predict. Since there are multiple menu items that we will be predicting order amounts for, we will leave the Multi-Series Forecast toggle to on.

   **![image104](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image104.png)**

3. For the Model Instance Name, you can name the model any meaningful name you want, but for the sake of this lab, we will call this model `ORDER_ITEM_FORECAST`.

   ![image105](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image105.png)

4. For the Series Column input, we will use the FORECAST_MENU_ITEM_NAME column. This Series Column is the column that is used as the series for predicting values for.

   ![image106](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image106.png)

5. For the Timestamp Column, we will use the FORECAST_ORDER_TS column.

   ![image107](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image107.png)

6. For the Target Column, we will use the FORECAST_ORDER_AMOUNT column. The Target Column is the column that contains the values we want to predict.

   ![image108](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image108.png)

7. In this node type, the exogenous variables toggle is turned on. Exogenous Variables are additional features or columns that the model can use to help refine and predict the output for the model. Our node doesn’t contain any additional columns that are not being used by the required parameters, so we will toggle this off.

   We are presented with a Days to Forecast input that defaults to 30 days. We can update this value to any value of our choosing, but for the sake of this lab guide, will leave the value at 30.

   ![image109](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image109.png)

8. You have now successfully configured the ML Forecast node. Create and Run the node to populate it within Snowflake.

   ![image110](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image110.png)

## Viewing your ML Forecast Node in Snowflake

Now that we have configured, created, and run the ML Forecast node within our Snowflake account, we can navigate to Snowflake to see how our forecast values show up.

1. Your node should be created within the PC_COALESCE_DB.PUBLIC database and schema, and is likely called ML_FCTS_ORDER_MASTER_ITEMS, unless you manually changed the name during the last section.

   Within Snowflake, run a select statement that queries all of the data within the table. You can copy and paste the code below, assuming you are using the default table name for your ML Forecast node:

   ```SQL
      select *
      from pc_coalesce_db.public.ml_fcsts_order_master_items
   ```

    ![image111](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image111.png)

2. Next, select the Chart option next to the results of your query.

    ![image112](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image112.png)

3. If the chart type is not a Line, charge the chat to a Line Chart

    ![image113](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image113.png)

4. The first line (blue line), will denote our historical data that we’ve already captured and forecast our values with. Make sure the Blue line has `FORECAST_ORDER_AMOUNT` as the column value.

    ![image114](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image114.png)

5. The X-Axis will be the FORECAST_ORDER_TS column, allowing us to see our order amounts over time

    ![image115](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image115.png)

6. Finally, select Add column beneath the X-Axis and select FORECAST as the column. This is the column that Coalesce automatically generated to predict the values of our order amounts in the future. This will likely be denoted by a yellow line.

   ![image116](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image116.png)

7. Now you can see how Coalesce used Snowflake Cortex functions to predict new values of our order amounts for the next 30 days.

   ![image117](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image117.png)

## Conclusion and Next Steps

Congratulations on completing your lab. You've mastered the basics of Coalesce and are now equipped to venture into our more advanced features. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

### What We’ve Covered

* How to navigate the Coalesce interface
* Configure storage locations and mappings
* How to add data sources to your graph
* How to prepare your data for transformations with Stage nodes
* How to join tables
* How to apply transformations to individual and multiple columns at once
* How to configure and understand the ML Forecast node

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our [documentation](https://docs.coalesce.io/docs) and [resources](https://coalesce.io/resources/). For more technical guides on advanced Coalesce and Snowflake features, check out Snowflake Quickstart guides covering [Dynamic Tables](https://quickstarts.snowflake.com/guide/building_dynamic_tables_in_snowflake_with_coalesce/index.html?index=..%2F..index#1) and [Cortex ML functions](https://quickstarts.snowflake.com/guide/ml_forecasting_ad/index.html?index=..%2F..index#0).

### Additional Coalesce Resources

* [Getting Started](https://coalesce.io/get-started/)
* [Documentation](https://docs.coalesce.io/docs) & [Quickstart Guide](https://docs.coalesce.io/docs/quick-start)
* [Video Tutorials](https://fast.wistia.com/embed/channel/foemj32jtv)
* [Help Center](mailto:support@coalesce.io)

Reach out to our sales team at [coalesce.io](https://coalesce.io/contact-us/) or by emailing [sales@coalesce.io](mailto:sales@coalesce.io) to learn more.

[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake Marketplace]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce

---

## Getting Started Building Pipelines on Snowflake with Coalesce

## Overview

This entry-level Hands-On Lab exercise is designed to help you master the basics of Coalesce. In this lab, you'll explore the Coalesce interface, learn how to easily transform and model your data with our core capabilities, and understand how to deploy and refresh version-controlled data pipelines.

### What You’ll Need

- A Snowflake account (either a [trial account][] or access to an existing account)
- A Coalesce account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account)
- Basic knowledge of SQL, database concepts, and objects
- A [GitHub account][] with access to the [companion repository][] (optional, not required to complete the majority of this lab)
- The [Google Chrome browser][]

### What You'll Build

- A Directed Acyclic Graph (DAG) representing a basic star schema in Snowflake

### What You'll Learn

- How to navigate the Coalesce interface
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to join tables
- How to apply transformations to individual and multiple columns at once
- How to build out Dimension and Fact nodes
- How to make changes to your data and propagate changes across pipelines
- How to work with Git
- How to deploy and refresh your data pipeline

By completing the steps we've outlined in this guide, you'll have mastered the basics of Coalesce and can venture into our more advanced features.

## About Coalesce

Coalesce is the first cloud-native, visual data transformation platform built for Snowflake. Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

With Coalesce, you can:

- Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
- Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Snowflake functionality
- Analyze the impact of changes to pipelines with built-in data lineage down to the column level
- Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
- Ensure consistent data standards and governance across pipelines, with data never leaving your Snowflake instance

### How Is Coalesce Different?

Coalesce's unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### Core Concepts in Coalesce

#### Snowflake

Coalesce currently only supports Snowflake as its target database, As you will be using a trial Coalesce account created with [Snowflake Marketplace][], your basic database settings will be configured automatically and you can instantly build code.

#### Organization

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

#### Projects

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Git repository and Snowflake account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

#### Workspaces vs. Environments

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Snowflake schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, PreProd, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

### About This Guide

The exercises in this guide use the sample data provided with Snowflake trial accounts. This data is focused on fictitious commercial business data including supplier, customer, and orders information. For a full overview of the TPC-H schema, please visit [Snowflake's documentation][].

## Before You Start

To complete this lab, please create free trial accounts with Snowflake and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

We recommend using Google Chrome as your browser for the best experience.

:::info[Complete All Steps]

Not following these steps will cause delays and reduce your time spent in the Coalesce environment.

:::

### Step 1: Create a Snowflake Trial Account

1. Fill out the [Snowflake trial account form][]. Use an email address that is not associated with an existing Snowflake account.

   ![image1](./building-data-pipelines-with-iceberg-llm-functions/assets/image1.png)

2. After registering, you will receive an email from Snowflake with an activation link and URL for accessing your trial account. Finish setting up your account following the instructions in the email.

### Step 2: Create a Coalesce Trial Account from Snowflake Marketplace

1. Sign in to Snowsight. Make sure you are using the ACCOUNTADMIN role (or a role with privileges to create a database, warehouse, user, and role) and that your Snowflake email address is verified.
2. In the navigation menu, select **Marketplace > Snowflake Marketplace** and search for Coalesce.

      

3. Open the Coalesce listing and select Start free trial in the upper right. To see what the trial includes, review the Your free trial tab.
4. In the Connect to Coalesce dialog, review the information to be shared and the Snowflake objects that will be created, then select Connect to Coalesce.
5. You will be redirected to Coalesce to finish activating your trial account. Fill in your information to complete activation.

Congratulations. You’ve successfully created your Coalesce trial account.

Once you’ve activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started. Click on the **Launch** button next to your Development Workspace to get started.

## Navigating the Coalesce User Interface

:::info[About this lab]

Screenshots (product images, sample code, environments) depict examples and results that may vary slightly from what you see when you complete the exercises.

This lab exercise does not include Git (version control). Please note that if you continue developing in your Coalesce account after this lab, none of your work will be saved or committed to a repository unless you set up before developing.  

:::

Let's get familiar with Coalesce by walking through the basic components of the user interface.

Upon launching your Development Workspace, you’ll be taken to the Build Interface of the Coalesce UI. The Build Interface is where you can create, modify, and publish nodes that will transform your data.

Nodes are visual representations of objects in Snowflake that can be arranged to create a Directed Acyclic Graph (DAG or Graph). A DAG is a conceptual representation of the nodes within your data pipeline and their relationships to and dependencies on each other.

1. In the Browser tab of the Build interface, you can visualize your node graph using the Graph, Node Grid, and Column Grid views. In the upper right hand corner, there is a person-shaped icon where you can manage your account and user settings.

    

2. The Browser tab of the Build interface is where you’ll build your pipelines using nodes. You can visualize your graph of these nodes using the Graph, Node Grid, and Column Grid views. While this area is currently empty, we will build out nodes and our Graph in subsequent steps of this lab.

    

3. Next to the Build page is the Deploy page. This is where you will push your pipeline to other environments such as Testing or Production. Like the Browser tab, this area is currently empty but will populate as we will build out Nodes and our Graph in subsequent steps of this lab.

    

4. Next to the Deploy page is the Docs page. Documentation is an essential step in building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. To help with this, Coalesce automatically generates and updates documentation as you work.

    

## Adding Data Sources

Let’s start to build a Graph (DAG) by adding data in the form of Source nodes.

1. Start in the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and select Add Sources.

    

2. Click to expand the tables within `COALESCE_SAMPLE DATABASE.TPCH_SF1` and select all of the corresponding source tables underneath except for `PARTSUPP` (7 tables total). Click **Add Sources** on the bottom right to add the sources to your pipeline. You will use the `PARTSUPP` source later on in this lab.

    

3. You'll now see your graph populated with your Source nodes. Note that they are designated in red. Each node type in Coalesce has its own color associated with it, which helps with visual organization when viewing a Graph.

    

## Creating Stage Nodes

Now that you’ve added your Source nodes, let’s prepare the data by adding business logic with Stage nodes. Stage nodes represent staging tables within Snowflake where transformations can be previewed and performed.  

Let's start by adding a standardized "stage layer" for all sources.

1. Press and hold the Shift key to multi-select all of your Source nodes. Then right click and select **Add Node > Stage** from the drop down menu.

    

    You will now see that all of your Source nodes have corresponding Stage tables associated with them.

    

2. Now change the Graph to a Column Grid by using the top left dropdown menu:

     

     By using the Column Grid view, you can search and group by different column headers.

    

3. Click on the Storage Location header and drag it to the sorting bar below the View As dropdown menu. You have now grouped your rows by Storage Location.

    

4. Expand all the columns from your `WORK` location and select them. Then right click and select **Bulk Edit Columns** from the dropdown menu. Coalesce makes it quick and straightforward to apply transformations to your data by bulk editing columns.

    

5. Select Nullable as an option and then True in the dropdown menu. This will make all columns in your WORK location nullable which will minimize any potential loading issues with the data.

    

6. Click the Preview button to preview your changes, and then click the Update button to apply them.

    

7. Repeat this step by scrolling to the right of your Column Grid and double clicking on the Data Type header so that VARCHAR values appear at the top of the grid. Press the Shift key and select all of the VARCHAR columns. Then right click and select Bulk Edit from the dropdown menu.  

    

8. Select Data Type under Attributes and then type in STRING next to the Data Type field. By changing the data type to STRING as a bulk edit, we can minimize any potential issues with changes to our source data.

    

9. Click the Preview button once more and then press the Update button to make your changes. Then return to your Browser tab and change from Column Grid to Graph view with the dropdown menu.

    

10. Press the Create All button and then the Run All button to update your Graph.  

    

## Exploring the Node Editor

Now that we have a standard layer for performing basic transformations, let's add a more specific transformation by joining two of our Stage nodes. We’ll complete this in the Node Editor, which is used to edit the node object, the columns within it, and its relationship to other nodes

1. Press the Shift key and select the STG_LINEITEM and STG_ORDERS nodes. Once selected, right click and select Join Nodes > Stage to create a Stage node.

    

2. This action will cause a separate Node Editor tab to open up, which will appear next to your Browser tab. This is the Node Editor for the STG_LINEITEM_ORDERS node you just created. There are a few different components to this tab, the first section is the Mapping grid, where you can see the structure of your node along with column metadata like transformations, data types, and sources.

    

3. On the right hand side of the Node Editor is the Config section, where you can view and set configurations based on the type of node you’re using.

    

4. At the bottom of the Node Editor, press the arrow button to view the Data Preview pane.

    

## Joining Nodes

Let’s now join your STG_ORDERS data with your STG_LINEITEM data.

1. Select the Join tab of your STG_LINEITEM_ORDERS node in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement.

    

2. Add column relationships in the last line of the existing Join statement using the code below:

    ```sql
    FROM {{ ref('WORK', 'STG_LINEITEM') }} "STG_LINEITEM"
    INNER JOIN {{ ref('WORK', 'STG_ORDERS') }} "STG_ORDERS"
    ON "STG_LINEITEM"."L_ORDERKEY" = "STG_ORDERS"."O_ORDERKEY"
    ```

3. To check if your statement is syntactically correct, press the white Validate Select button in the lower half of the screen. The Results panel should show a green checkmark, confirming that the SQL is accepted. If not, you will see a database message indicating the error.

    

4. To create your Stage Node, click the Create button in the lower half of the screen. Then click the Run button to populate your STG_LINEITEM_ORDERS node and preview the contents of your node by clicking Fetch Data.

    

## Applying Single Column Transformations

Let’s apply a single column transformation in your STG_LINEITEM_ORDERS node.

1. Click on Mapping to return to the Mapping Grid. Scroll down to the O_TOTALPRICE column and right click to duplicate the column.

    

2. Double click on the duplicated column and rename it to `O_TOTAL_ORDER_PRICE`. Then enter the transformation below in the Transform field to the right of the column name.

    `CASE WHEN "STG_LINEITEM"."L_LINENUMBER" = 1 THEN {{SRC}} ELSE 0 END`

    The token `{{SRC}}` serves as a shorthand for our source node, allowing us to perform the transformation faster without having to code individual source names by hand.

    

3. Click the Create and Run buttons once more to repopulate your node.

    

4. Click back to Mapping and scroll down in the Mapping grid to select the `O_ORDERPRIORITY` column. Rename the column as `O_ORDERPRIORITY_NUM` and enter the transformation `SPLIT_PART("STG_ORDERS"."O_ORDERPRIORITY", '-', 1 )` in the Transform field. This transformation formula takes the original `O_ORDERPRIORITY` column in `STG_ORDERS`, splits its value at instances of '-', and returns the first portion of the split.

   Then right click on `O_ORDERPRIORITY_NUM` column and select Duplicate Column from the drop down menu.

    

5. Rename the duplicate column as `O_ORDERPRIORITY_DESC`.

    

6. In the transform field for `O_ORDERPRIORITY_DESC`, adjust the transformation to read `SPLIT_PART("STG_ORDERS"."O_ORDERPRIORITY", '-', 2 )`. Notice that this is the same transformation you entered previously, except this will now return the second portion of the split.  

   Click the ellipsis on the Create button and select Validate Create to validate that your create statement will run. On confirmation, click Create and then click the Run button. Once the Run has completed, scroll to the right in the Data Preview pane to preview your new `O_ORDERPRIORITY_NUM` and `O_ORDERPRIORITY_DESC` columns.

    

7. Now let’s apply another single column transformation by concatenating two different columns from different Stage nodes. Scroll to the bottom of the Mapping grid and double click on the gray column named Column Name.

    

8. Name this column `DAYS_TO_SHIP` and press the Enter key to create a new column. Under Data Type, enter `NUMBER`.

    

9. In the Transform field, enter the following transformation to calculate the difference between order dates and shipping dates listed in the `STG_ORDERS` and `STG_LINEITEM` nodes.

    `DATEDIFF('DAY',"STG_ORDERS"."O_ORDERDATE","STG_LINEITEM"."L_SHIPDATE")`

    

10. Click the Create and Run buttons to execute the statements. Once the Run has completed, scroll to the right in the Data Preview pane to view your newly created `DAYS_TO_SHIP` column.  

## Creating Dimension Nodes

Now let’s experiment with creating Dimension nodes. These nodes are generally descriptive in nature and can be used to track particular aspects of data over time (such as time or location). Coalesce currently supports Type 1 and Type 2 slowly changing dimensions, which we will explore in this section.

1. In your `STG_LINEITEM_ORDERS` node, navigate to the Mapping grid. Select the `O_CUSTKEY` column and drag it to the top of the Mapping grid. Repeat this motion by dragging the `L_SUPPKEY` and `L_PARTKEY` columns to the top of the Mapping grid.

    

2. Let’s rename some of the columns in our node for readability. Click Column Name to sort. Press and hold the Shift key to multi-select all columns that begin with the L\_ prefix. Then right click and select Bulk Edit from the drop down menu.  

    

3. Select Column Name as your attribute and enter the code `LINEITEM_{{column.name[2:]}}` to rename the columns. This particular Jinja code snippet will concatenate the prefix and remove the first 2 characters from the original column name.

     

4. Repeat this process by bulk renaming all columns with an O\_ prefix using the code `ORDER_{{column.name[2:]}}`.

    

5. Click Preview to check your renaming rule, and then press Update to apply it to your selected columns.  

    

6. Click Create and Run to update your `STG_LINEITEM_ORDERS` node.

    

7. Now let’s create a Dimension node for each of these foreign key columns by navigating back to the Browser tab. Press the Shift key and select `STG_CUSTOMER`, `STG_SUPPLIER` and `STG_PART` source nodes on the graph. Then right click and select Add Node > Dimension.

    

8. A Dimension node will appear for each of your selected Stage nodes. Under the Run All button, click the first icon to optimize the layout of your graph.

    

9. Double click on your new DIM\_PART node to open up the Node Editor. In the Config section on the right hand side of the screen, click to expand Options. Under Business Key, check the box next to `P_PARTKEY` and press the > button to select it as your business key. This will remain a Type 1 dimension, so we will not select columns for change tracking over time.

    

10. Return to the Browser tab and double click on the `DIM_CUSTOMER` node. In the Config section of your editor, expand Options and select `C_CUSTKEY` as your business key.

    

    To create a Type 2 dimension, select the `C_NAME` and `C_ADDRESS` columns under Change Tracking to track changes to these columns over time.

    

11. Return to the Browser tab once more and double click to open up the `DIM_SUPPLIER` node. Select `S_SUPPKEY` as your business key to create a Type 1 dimension.

    

12. Return to your graph and click Create All and Run All in the upper right hand corner.

## Bulk Editing Columns

Now let’s apply a bulk transformation to some of the columns in your `DIM_CUSTOMER` node. Double click on `DIM_CUSTOMER` in your graph to open up your Node Editor.

1. In the Mapping grid, click twice on Data Type to bring all VARCHAR columns to the top of the grid. Press and hold the Shift key to multi-select all the VARCHAR columns beginning with the `C_` prefix. Right click on your selections and click on Bulk Edit in the dropdown menu.

    

2. Click on the Column Editor on the right hand side of the screen. Select Data Type and Transform under Attributes. Clarify the Data Type as STRING and enter the transformation `UPPER({{SRC}})`in the Transform field.  

    

3. Click the Preview button and then press Update to apply the bulk transformation.

    

4. Return to the Browser tab and right click the `STG_LINEITEM_ORDERS` node to create a new Stage node, which will be a lookup table for our Dimension nodes.  

    

5. Open the new Stage node and rename it `STG_LINEITEM_ORDERS_DIM_LOOKUP` in the Node Editor. On the left hand side of the screen in the Node and Column selector, select the `DIM_CUSTOMER` node. Drag the `DIM_CUSTOMER_KEY` column from the column selector over to the bottom of your Mapping grid.

    

6. Repeat this action by selecting the `DIM_PART` node and dragging the `DIM_PART_KEY` column into the Mapping grid.

    

7. Repeat this action once more by selecting the `DIM_SUPPLIER` node and dragging the `DIM_SUPPLIER_KEY` column into the Mapping grid.

    

8. Select the Join tab and delete the existing code shown.

    

9. Click Generate Join and then Copy to Editor to automatically import your join statement.

    ```sql title="Copy to editor"
    FROM {{ ref('WORK', 'STG_LINEITEM_ORDERS') }}
    "STG_LINEITEM_ORDERS"

    LEFT JOIN {{ ref('WORK', 'DIM_CUSTOMER') }}
    "DIM_CUSTOMER"
    ON "STG_LINEITEM_ORDERS"./*COLUMN*/ = "DIM_CUSTOMER".
    "C_CUSTKEY"

    LEFT JOIN {{ ref('WORK', 'DIM_PART') }} "DIM_PART"
    ON "STG_LINEITEM_ORDERS"./*COLUMN*/ = "DIM_PART".
    "P_PARTKEY"

    LEFT JOIN {{ ref('WORK', 'DIM_SUPPLIER') }}
    "DIM_SUPPLIER" 
    ```

    

10. Manually resolve the join by adding `ORDER_CUSTKEY`, `LINEITEM_PARTKEY` and `LINEITEM_SUPPKEY` into the empty column names. Then click the Create and Run buttons to create and populate your `STG_LINEITEM_ORDERS_DIM_LOOKUP` node.

    

Note that If this source data had changed for customer, we would just need an additional line of code in the join.

## Creating and Updating Fact Nodes

Now let’s create a Fact node. Fact nodes represent Coalesce's implementation of a Kimball Fact Table, which consists of the measurements, metrics, or facts of a business process and is typically located at the center of a star or Snowflake schema surrounded by dimension tables.

1. Click on Mapping and then click on Data Type. Hold the Shift key and select the NUMBER and DATE columns only. Then right click to create a Fact node from just these columns, as Fact tables typically only contain measurements (as opposed to text, which would potentially be in a Dimension node).

    

2. Once your new Fact node has opened, rename it to `FCT_ORDER_DETAILS`. Return to your Browser tab and click Create and then Run to populate it with data.

    

Congratulations. You have now built out a small data mart.

## Propagating Changes Across Pipelines

Of course, business requirements are always changing and the business has now decided that they want to include supply costs as part of their analysis. Let’s explore how you can manage your pipeline under these evolving circumstances.

1. Click the \+ button next to the Search bar on the left side of the Browser tab and select Add Sources.

    

2. Expand the sample dataset and select the PARTSUPP source. Then click the Add 1 source button.

    

3. Double click on the new PARTSUPP source node to open it. You’ll see two columns, `PS_AVAILQTY` and `PS_SUPPLYCOST`, that need to appear in the `FCT_ORDER_DETAILS` table.

   

4. Return to the Browser and double click on your `STG_LINEITEM` node to open it. In the left hand portion of the Mapping grid, select the PARTSUPP node. Select the `PS_AVAILQTY` and `PS_SUPPLYCOST` columns and drag them into the bottom of the Mapping grid, effectively moving these columns into your `STG_LINEITEM` node.

    

5. Click on the Join tab and then click on Generate Join. Copy the last line to the Join tab and complete the statement with the `L_PARTKY` and `PS_PARTKEY` columns to join `PARTSUPP` with `STG_LINEITEM`.

    

6. Return to the Browser tab and click the Create and Run buttons, and then click the spiral icon to optimize your Graph.
7. Now let’s take a look at our column lineage and ensure that all of our columns are carried through our pipeline. Switch back to your `STG_LINEITEM` node and select the `PS_AVAILQTY` and `PS_SUPPLYCOST` columns. Then right click and select View Column Lineage from the dropdown menu.

    

8. Once in the Column Lineage view, select both columns and click the ellipses to select Propagate Addition. This will allow you to propagate these columns to the downstream nodes in your pipeline.

    

9. Check the boxes to add these columns to the last two successor nodes in your pipeline.

    

10. Click the Preview button to confirm that your two columns have been added to the bottom of each successor node. Then Click Apply and Confirm to propagate your additions.

    

11. By clicking on the DAYS\_TO\_SHIP column, you can view the concatenation and split lineage of this column. Then return to the **Browser** tab and click the **Run All** button to refresh your pipeline.

    

## Optimizing, Running and Validating Your DAG

1. Return to the Browser tab of the Build Interface. Click the  icon under Run All to visually optimize the organization of your nodes.

    

2. From the Run All dropdown, select Validate Create All and confirm there are no errors with the create statements in your pipeline. Upon confirmation, execute a Create All. Repeat these validation and execution steps with Run All.

    

3. Switch to your Snowflake account and view the tables and data within the Snowflake platform to ensure everything is as intended. You will find your build in PC\_COALESCE\_DB.PUBLIC (database.schema) as shown in the screenshot below.

    

## Conclusion and Next Steps

Congratulations on completing this entry-level lab exercise. You've mastered the basics of Coalesce and are now equipped to venture into our more advanced features. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

### What We’ve Covered

- How to navigate the Coalesce interface  
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes  
- How to join tables  
- How to apply transformations to individual and multiple columns at once  
- How to build out Dimension and Fact nodes  
- How make and propagate changes to your data across pipelines

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our [documentation][] and [resources][].

### Additional Resources

- [Getting Started][]
- [Quickstart Guide][]
- [Video Tutorials][]
<!-- vale off -->
Reach out to our sales team at [coalesce.io](https://coalesce.io/contact-us/) or by emailing [sales@coalesce.io](mailto:sales@coalesce.io) to learn more.

[trial account]: https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=CjwKCAiAk--dBhABEiwAchIwkYotxMl4v5sRhNMO6LxCXUTaor9yH2mAbOwQHVO0ZNJQRuNyHrmYwRoCPucQAvD_BwE
[GitHub account]: https://github.com/
[companion repository]: https://github.com/coalesceio/hands-on-lab
[Google Chrome browser]: https://www.google.com/chrome/
[Snowflake's documentation]: https://docs.snowflake.com/en/user-guide/sample-data-tpch.html
[resources]: https://coalesce.io/resources/
[documentation]: /
[Getting Started]: https://coalesce.io/get-started/
[Quickstart Guide]: /docs/get-started/quick-start
[Video Tutorials]: https://fast.wistia.com/embed/channel/foemj32jtv
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake Marketplace]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake trial account form]: https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=Cj0KCQiAtvSdBhD0ARIsAPf8oNm6YH7UeRqFRkVeQQ-3Akuyx2Ijzy8Yi5Om-mWMjm6dY4IpR1eGvqAaAg3MEALw_wcB

---

## Connecting to External Iceberg Tables with Coalesce

## Overview

Learn how to work with data in Iceberg Table format that exists in object storage or AWS Glue.

### Resources

- [Iceberg Tables Package][]
- [Sign up for Coalesce][]

[Iceberg Tables Package]: /docs/marketplace/package/coalesce_snowflake_iceberg-tables
[Sign up for Coalesce]: https://coalesce.io/get-started/

<YouTube id="CgNLydgbAm4" />

---

## Essential Foundations for Databases and Data Warehousing

## Overview

Coalesce provides the "T" or transform, component of an extract, load, and transform, or ELT approach to data integration and data warehousing. With an ELT approach, the transform activities are completed directly on the database in which the data has been previously loaded and use the power of the database rather than servers proprietary to the transformation tool to complete the transformations.

To optimize your experience with Coalesce, it is crucial that you have a good understanding of databases, data warehousing, and SQL.

:::info[Learning Resources]

This article links out to learning resources created, maintained, and owned by third party resources not affiliated with Coalesce.

:::

## Databases and Data Warehousing

### What Is a Database?

A database is a structured collection of data that is stored and accessed electronically. It is designed to manage, store, and retrieve information efficiently. Databases support the operations and management of data in various forms, such as text, numbers, and multimedia.

Databases are essential for a wide range of applications, from simple systems that store user information for websites to complex data warehousing solutions that aggregate vast amounts of information from multiple sources for business intelligence and analytics purposes. They are managed through database management systems (DBMS), which provide the tools and functionalities needed for data insertion, querying, update, and administration, ensuring data integrity, security, and performance.

If you are new to databases or just need to brush up on your core knowledge of them, the video tutorial and resources linked below provide foundational insights that serve as stepping stones to deepen your knowledge of this area.

<YouTube id="Tk1t3WKK-ZY" />

:::note[Learn more]

- [What is a Database?][] An overview from AWS.
- [What are Databases?][] An in-depth explanation from Microsoft Azure.
- [Database Overview.][] A comprehensive look from Wikipedia.

:::

### What Is Data Warehousing?

Data warehousing refers to the process of consolidating data from multiple sources into a single, centralized repository designed for analysis. It is a foundational component of business intelligence systems, enabling organizations to aggregate vast amounts of data in a structured format to support decision-making processes. By integrating data from various operational databases, data warehouses provide a unified view of the information, facilitating advanced analytics, reporting, and data mining.

The architecture of a data warehouse is optimized for fast data retrieval and analysis, rather than transaction processing, making it an ideal environment for identifying trends, patterns, and insights that can drive strategic business decisions. Through the use of data warehousing, businesses can improve their operational efficiency, understand customer behavior more deeply, and gain a competitive edge by leveraging data-driven strategies.

If you are new to data warehousing, or are looking for a refresher on the practice, we encourage you to check out the video tutorial and resources linked below.

<YouTube id="AHR_7jFCMeY" />

:::note[Learn More]

- [What is a Data Warehouse?][] An overview from AWS.
- [Data Warehouse Overview.][] A comprehensive look from Wikipedia.
- [Cloud Data Warehousing for Dummies.][] A beginner's guide to data warehousing in the cloud from Snowflake.

:::

## SQL

SQL is the standard language for managing and querying data in relational databases. To maximize your productivity within Coalesce, a solid understanding of Structured Query Language (SQL) is essential.

### What Is SQL?

Structured Query Language, or SQL, is a standardized programming language specifically designed for managing and manipulating relational databases. It is used for tasks such as querying data, updating records, and managing database schemas, making it an essential tool for database administrators and developers.

SQL enables users to create, read, update, and delete database records through its various commands, clauses, and syntax. This language provides a powerful and efficient means to retrieve and organize data across multiple tables, implement data security measures, and ensure data integrity within databases.

Given its wide adoption, SQL is the core language of virtually all relational database management systems (RDBMS), including popular platforms like MySQL, PostgreSQL, SQL Server, and Oracle.

### Learn SQL

If you are new to SQL, or just need to brush up on your knowledge of it, the video tutorial and resources linked below provide a great foundational introduction as well as advanced insights into SQL.

<YouTube id="zsjvFFKOm3c" />

:::note[Learn More]

- [What is SQL (Structured Query Language)?][] A comprehensive introduction from AWS
- [SQL Tutorial][] Detailed lessons and examples from w3schools
- [SQLZoo][] Interactive SQL tutorials for a hands-on approach to learning

:::

[What is a Database?]: https://aws.amazon.com/what-is/database/
[What are Databases?]: https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-are-databases
[Database Overview.]: https://en.wikipedia.org/wiki/Database
[What is a Data Warehouse?]: https://aws.amazon.com/what-is/data-warehouse/
[Data Warehouse Overview.]: https://en.wikipedia.org/wiki/Data_warehouse
[Cloud Data Warehousing for Dummies.]: https://www.snowflake.com/resource/cloud-data-warehousing-dummies/
[What is SQL (Structured Query Language)?]: https://aws.amazon.com/what-is/sql/
[SQL Tutorial]: https://www.w3schools.com/sql/
[SQLZoo]: https://sqlzoo.net/

---

## Databricks & Coalesce: Build a Weather Analytics Pipeline from AccuWeather Data

## Before You Begin

- Go through our [Databricks Connection Guide][] to get your account set up with Coalesce.
- Add the [AccuWeather data source][] to your catalog.
  - `forecast_daily_calendar_metric`
- Create a [Project][] and [Workspace][].
  - You can skip creating a version control account.

:::info[Need Databricks Access?]

Reach out to our [sales team][].

:::

## Add a Data Source

In your Workspace, you are on the [Build Interface][]. This is where you'll build your data pipelines.

1. Make sure you are on Nodes.
2. Click on the **"+" sign > Add Sources**.

    

3. Add Sources to SQL Pipeline will open. Find the `forecast_daily_calendar_metric` table. Double-clicking the source will give you a preview of the data.

    

4. Then click **Add Source**.
5. The source Node is added to your DAG. Now you can start building.

## Add a Stage Node

1. Right-click on the source **Node > Add Node > Stage**.

    

2. It will open the added stage Node in the [Node Editor][].
3. Next to the name, click the pencil and name it `stg_weather_metrics`. Databricks Node names are formatted as lowercase.
4. Then click **Create** and **Run**.

    

5. You’ve just populated your first table in Databricks. Click on the **Results** tab. You can see each stage complete and the SQL executed.

:::info[What Create and Run Do]

Create renders and executes all SQL stages of the [create template][] defined in the [Node Type][] definition. Run renders and executes all SQL stages of the [run template][] defined in the [Node Type][] definition.

:::

## Add a Transformation

1. If the Node isn’t open, you can open the `stg_weather_metrics` by double clicking the Node or right-click and select Edit.
2. Sort the column by name, by clicking on the **Column Name** table header.
3. Scroll to the column named `temperature`.
4. Double-click in the **Transform** column next to `temperature`.
5. Add `ROUND({{SRC}}, 1)`. This uses standard SQL and Helper Tokens. `SRC` evaluates to the fully qualified name and location. After adding, it should update to the correct `SRC`. For example, ``ROUND(`forecast_hourly_metric`.`temperature`, 1)``.

    

6. Click **Validate Select**. Validate Select lets you validate your transformations and joins.
7. Scroll down in the Results panel to see the SQL applied to `temperature`.

    

8. Click **Create** and **Run**.
9. In the Data panel, scroll over to `temperature`, and see that the value has been rounded to one decimal place.

## Add a Dimension Node

You’ll create a Dimension Node that will track how weather metrics change over time and each location. First we'll add some transformations to our stage Node to prep the data.

1. Make sure you're in the `stg_weather_metrics` Node.
2. You need to delete any columns not being used. Un-sort the columns by clicking the Column Name until the arrows disappear. Click and drag the following columns so they are at the top:
    1. `city_name`
    2. `country_code`
    3. `datetime_valid_local`
    4. `temperature`
    5. `humidity_relative`
    6. `wind_speed`
    7. `precipitation_type_desc`
    8. `cloud_cover_perc_total`
3. Click on the first column to be deleted, hold Shift or Command then select the last column to be deleted. **Right-click** and **Delete** the columns.

    <Wistia id="vuatsh40q3" subtitle="A demo showing selecting multiple columns for deletion."/>

4. Rename the column `datetime_valid_local` to `measurement_time`. Double click on `measurement_time` to rename it.
5. Rename `cloud_cover_perc_total` to `cloud_cover_percent`.
6. Add the following transformations to standardize the data:
    1. `temperature` - `ROUND({{SRC}}, 1)` to round to one decimal place, if not already added.
    2. `humidity_relative` - `ROUND({{SRC}}, 1)` to round to one decimal place.
    3. `precipitation_type_desc` - `COALESCE({{SRC}}, 'None')` to replace null values.
7. Then double click in the **Data Type** for `cloud_cover_percent` and type **INTEGER**.

    

8. Click **Validate Select** to preview the SQL that will be run.
9. Then click **Create** and **Run**.
10. Return to your DAG and create a Dimension Node off the `stg_weather_metrics`. Name it `dim_weather_metrics`.
11. Under **Options**, select the following as **Business Keys** by selecting them in the box, then click the arrow to move them over.
    1. `city_name`
    2. `country_code`
    3. `measurement_time`
12. In **Change Tracking**, select the following:
    1. `temperature`
    2. `precipitation_type_desc`
    3. `wind_speed`
    4. `cloud_cover_percent`
13. Then click **Create** and **Run**.

## Create a Fact Node

You'll create a Fact Node to track historical data. You'll also use the Join tab.

1. Create a Stage Node called `stg_weather_facts` off the Dimension Node `dim_weather_metrics`.
2. You'll need to create a Join. The Join tab can be used to accept any valid SQL. We are joining the source data, `forecast_hourly_metric` to `dim_weather_metrics`. The conditions are the `city_name`, `country_code`, and `measurement_time` must match. This will join all results even if there are no matches. They will be null values.

    ```sql
    FROM {{ ref('ACCUWEATHER', 'forecast_hourly_metric') }} `forecast_hourly_metric`
    LEFT JOIN {{ ref('<YOUR-STORAGE-LOCATION-NAME>', 'dim_weather_metrics') }} `dim_weather_metrics`
        ON `forecast_hourly_metric`.`city_name` = `dim_weather_metrics`.`city_name`
        AND `forecast_hourly_metric`.`country_code` = `dim_weather_metrics`.`country_code`
        AND `forecast_hourly_metric`.`datetime_valid_local` = `dim_weather_metrics`.`measurement_time`
    ```

    

3. Create and Run the Node.
4. Click on the browser tab to see that the `stg_weather_facts` has a reference to the source and dimension Node.

    

5. Create a Fact Node from the `stg_weather_facts`.
6. Name it `fct_weather_facts`.
7. Set the Business Keys as:
   1. `city_name`
   2. `country_code`
   3. `measurement_time`
8. Create and Run the Node.

<figure>
  
  <figcaption>The final DAG</figcaption>
</figure>

## Wrap Up

Here's what you did:

You Joined the source Node to the `stg_weather_facts` to get raw measurements like `humidity` and `precipitation_type_desc`. This is the data you are tracking.

`Stg_weather_facts` will have the `dim_weather_metrics_key`. This makes sure each row has the correct historical weather data. Another way of saying this is: "On this day, time, and place, the weather description was X." Another example, "This forecast measurement ties to `dim_weather_metrics_key` at 124 to 124 in the fact Node and that record said it was cloudy, 68°F, and humid at that time."

### Databricks Query

You can also query this in Databricks.

```sql title="Avg Humidity by cloud cover type"
SELECT
  d.cloud_cover_percent,
  COUNT(*) AS measurement_count,
  ROUND(AVG(f.humidity_relative), 1) AS avg_humidity
FROM fct_weather_facts f
JOIN dim_weather_metrics d
  ON f.dim_weather_metrics_key = d.dim_weather_metrics_key
GROUP BY d.cloud_cover_percent
ORDER BY d.cloud_cover_percent;
```

```sql title="Daily temperature trends by city"
SELECT
  f.measurement_time::DATE AS date,
  f.city_name,
  ROUND(AVG(d.temperature), 1) AS avg_temp
FROM fct_weather_facts f
JOIN dim_weather_metrics d
  ON f.dim_weather_metrics_key = d.dim_weather_metrics_key
GROUP BY date, f.city_name
ORDER BY date, f.city_name;
```

<figure>
  
  <figcaption>Daily temperature trends by city</figcaption>
</figure>

[sales team]: https://coalesce.io/contact-us/
[Databricks Connection Guide]: /docs/setup-your-project/connection-guides/databricks
[AccuWeather data source]: https://marketplace.databricks.com/provider/f0a29bd4-946b-477e-a43a-1c3f2a6e6506/AccuWeather
[Project]: /docs/setup-your-project/create-your-project
[Workspace]: /docs/setup-your-project/create-a-workspace
[Build Interface]: /docs/build-your-pipeline/the-build-interface/
[Node Editor]: /docs/build-your-pipeline/the-build-interface/node-editor
[create template]: /docs/build-your-pipeline/user-defined-nodes/
[Node Type]: /docs/get-started/coalesce-fundamentals/nodes
[run template]: /docs/build-your-pipeline/user-defined-nodes/

---

## Getting Started Building Pipelines on Databricks with Coalesce

## Overview

1. This entry-level Hands-On Lab exercise is designed to help you master the basics of Coalesce. In this lab, you’ll explore the Coalesce interface, learn how to easily transform and model your data with our core capabilities, and understand how to deploy and refresh version-controlled data pipelines.

### What You’ll Need

* A Databricks trial account
* A Coalesce trial account
* Basic knowledge of SQL, database concepts, and objects
* The Google Chrome Browser

### What You’ll Build

A Directed Acyclic Graph (DAG) representing a basic star schema in Databricks

### What You’ll Learn

* How to navigate the Coalesce interface
* How to add data sources to your graph
* How to prepare your data for transformations with Stage nodes
* How to join tables
* How to apply transformations to individual and multiple columns at once
* How to build out Dimension and Fact nodes
* How to make changes to your data and propagate changes across pipelines
* By completing the steps we’ve outlined in this guide, you’ll have mastered the basics of Coalesce and can venture into our more advanced features.

## About Coalesce

Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

* With Coalesce, you can:
* Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
* Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Databricks functionality
* Analyze the impact of changes to pipelines with built-in data lineage down to the column level
* Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
* Ensure consistent data standards and governance across pipelines, with data never leaving your Databricks instance

### How Is Coalesce Different?

Coalesce’s unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### Core Concepts in Coalesce

#### Multi-Platform

Coalesce supports multiple data platforms as the target of your work. As you will be using a trial Coalesce account, your basic database settings will be configured in the first few sections of the lab guide.

### Organization

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

### Projects

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Git repository and Databricks account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

### Workspaces VS Environments

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Databricks schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, UAT, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

### About This Guide

The exercises in this guide use the sample data provided with Databricks trial accounts. This data is focused on fictitious commercial business data including supplier, customer, and orders information.

![image1](databricks-quickstart/image1.png)

## Before You Start

1. To complete this lab, please create free trial accounts with Databricks and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

2. We recommend using Google Chrome as your browser for the best experience.

### Step 1: Create a Databricks Trial Account

1. Select the Express Setup for a Databricks trial account form . Use an email address that is not associated with an existing Databricks account.

    ![image2](databricks-quickstart/image2.png)

2. Give your account a meaningful name. You will have access to your Databricks trial account for 30 days. Then select the country you will be using your Databricks trial in.

    ![image3](databricks-quickstart/image3.png)

3. Once complete, Databricks will set up your account and you will land on the Databricks homepage. Your Databricks trial account is ready to go. Make sure you save the URL of your account so you can access it in the future.

    ![image4](databricks-quickstart/image4.png)

### Step 2: Create a Coalesce Trial Account

1. Next, you’ll need to sign up for a free Coalesce trial account. To do this, navigate to the Coalesce .

2. Select the data platform you want to connect to, for this lab, select Databricks. Then, select the number of SQL users you want to support. Since you’re likely doing this lab on your own, select 1-5.

    ![image5](databricks-quickstart/image5.png)

3. Next, fill in the information to set up your trial account. If you have an existing Coalesce account, use an email that is not associated with an existing Coalesce account.

    ![image6](databricks-quickstart/image6.png)

4. Select the Submit button. Within a few minutes, you will receive your trial account email, sent to the email address provided in the submission form.

    ![image7](databricks-quickstart/image7.png)

5. Click the ACTIVATE TRIAL button in the email. You will be taken to an activation page where you will fill out your account based information.

    ![image8](databricks-quickstart/image8.png)

6. Select activate Trial and you will land inside of the Coalesce platform!

7. Congratulations! You’ve successfully created your Coalesce trial account.

8. Once you’ve activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

## Creating a Project in Coalesce

Now that you have a Coalesce and Databricks trial account, it’s time to connect the two together. The first step is creating a project that uses Databricks as the supported platform.

1. In the upper left corner of the projects page, select the plus + button to create a new project.

    ![image9](databricks-quickstart/image9.png)

2. Give your project a name, select Databricks as your platform, and give an optional description.

    ![image10](databricks-quickstart/image10.png)

3. Select the Next button, where you’ll be asked about setting up version control. Coalesce supports a variety of Version Control platforms and we recommend this for anyone using Coalesce in their job. However, as it requires additional set up, and setting up version control is not a requirement to build, we will be skipping this step. Select the Skip and Create button in the lower left corner of the modal.

    ![image11](databricks-quickstart/image11.png)

4. You’ll receive a warning message that you are about to proceed with creating a project that is not version controlled. Select Skip and Create to confirm.

    ![image12](databricks-quickstart/image12.png)

5. Congratulations, you now have a project in Coalesce that supports development on Databricks. In the next section, we’ll create a workspace that connects directly to your Databricks trial account.

## Creating a Workspace

Now that you have a project created to support Databricks development, you need to create a workspace to actually build your data products. That’s what a workspace is for. Let’s create a workspace and connect it to your Databricks account.

1. In your new project, select the blue Create Workspace button to create a new workspace.

    ![image13](databricks-quickstart/image13.png)

2. Give your workspace a name and optionally provide a description.

    ![image14](databricks-quickstart/image14.png)

3. Now in the projects page, you should see a new workspace! Click the gear icon next to the blue Launch button to configure your workspace and connect it to Databricks.

    ![image15](databricks-quickstart/image15.png)

4. This will open the workspace settings. The first thing we need to do is provide the Databricks account URL we are connecting to.

    ![image16](databricks-quickstart/image16.png)

5. To do this, navigate to your Databricks trial and select SQL Warehouses.

    ![image17](databricks-quickstart/image17.png)

6. Select the warehouse listed (Serverless Starter Warehouse) and select the Connection details tab. You will need to copy the Server Hostname to your clipboard. You can do this by selecting the copy icon next to the Server Hostname string.

    ![image18](databricks-quickstart/image18.png)

7. Return to Coalesce and provide the string in the Databricks account URL field. Once done, make sure to select Save to save the changes to your workspace.

    ![image19](databricks-quickstart/image19.png)

8. Next, navigate to the User Credentials tab in the workspace modal. This is where we will provide the authentication to our Databricks trial account.

    ![image20](databricks-quickstart/image20.png)

9. The default authentication type is Token (Cloud), which is what we will be using. Navigate back to your Databricks trial account one last time. This time, select your personal settings in the upper right corner of the screen and select Settings.

    ![image21](databricks-quickstart/image21.png)

10. Within the settings, select Developer from the User men and then select Manage next to Access tokens.

    ![image22](databricks-quickstart/image22.png)

11. On the Access tokens page, select the blue Generate new token button.

    ![image23](databricks-quickstart/image23.png)

12. Give the token a comment and set the lifetime. You can leave the default at 90 days.

    ![image24](databricks-quickstart/image24.png)

13. Select the blue Generate button. This will display your new access token. Be sure to copy this to your clipboard so we can paste it into Coalesce.

    ![image25](databricks-quickstart/image25.png)

14. Navigate back to Coalesce. Select the edit button next to the token field and paste in the access token you created in Databricks.

    ![image26](databricks-quickstart/image26.png)

15. You should now have the ability to save & retrieve the SQL warehouse available in your Databricks account, to use to run your Coalesce objects.

    ![image27](databricks-quickstart/image27.png)

16. Select the Serverless start warehouse available and then select the blue save button to save all of your changes.

    ![image28](databricks-quickstart/image28.png)

17. You have now connected your Coalesce trial account to your Databricks trial account and can start building your data products.

## Navigating the Coalesce User Interface

Let's get familiar with Coalesce by walking through the basic components of the user interface.

1. Upon launching your Development Workspace, you’ll be taken to the Build Interface of the Coalesce UI. The Build Interface is where you can create, modify and publish nodes that will transform your data.

2. Nodes are visual representations of objects in Databricks that can be arranged to create a Directed Acyclic Graph (DAG or Graph). A DAG is a conceptual representation of the nodes within your data pipeline and their relationships to and dependencies on each other.

3. In the Browser tab of the Build interface, you can visualize your node graph using the Graph, Node Grid, and Column Grid views. In the upper right hand corner, there is a person-shaped icon where you can manage your account and user settings.

    ![image29](databricks-quickstart/image29.png)

4. Next to the Build interface is the Deploy interface. This is where you will push your pipeline to other environments such as Testing or Production. Like the Browser tab, this area is currently empty but will populate as we will build out Nodes and our Graph in subsequent steps of this lab.

    ![image30](databricks-quickstart/image30.png)

5. Next to the Deploy interface is the Docs interface. Documentation is an essential step in building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. To help with this, Coalesce automatically generates and updates documentation as you work.

    ![image31](databricks-quickstart/image31.png)

## Adding Storage Locations

Storage locations are containers or pointers that allow you to designate which catalog and schema you want to represent within the Coalesce UI. We will add three storage locations in this lab.

1. To add a storage location, navigate to the build settings in the lower left corner of the screen.

    ![image32](databricks-quickstart/image32.png)

2. Within the build settings, navigate to Storage Locations.

    ![image33](databricks-quickstart/image33.png)

3. Select the Create Storage Location button in the upper right corner of the screen. The first storage location we’ll create is called BRONZE. In the text box, type in BRONZE and select the Create Storage Location button.

    ![image34](databricks-quickstart/image34.png)

4. We’ll follow this same process to create our Silver layer.

    ![image35](databricks-quickstart/image35.png)

5. And finally, we’ll create the gold layer.

    ![image36](databricks-quickstart/image36.png)

6. In the storage location menu page, select the ellipses next to the GOLD storage location and select Set as Default. This will make sure that, by default, all objects created in coalesce will be persisted in this location within Databricks.

    ![image37](databricks-quickstart/image37.png)

7. Now that we’ve created storage locations, we need to point these storage locations to physical destinations within our Data bricks account. Let’s do that next!

## Configuring Storage Mappings

In order to use any data in your Databricks account, you need to point your storage locations to physical destinations within Databricks. We call these Storage Mappings in Coalesce. Let’s configure these now, at the end of which, we can begin building our data pipelines!

1. Navigate back to the workspace settings, the same place where we configured the connection to your Databricks account.

    ![image38](databricks-quickstart/image38.png)

2. Navigate to the Storage Mapping menu item and you’ll map your Storage Locations with the following Catalogs and Schemas:

    * BRONZE -> samples -> tpch
    * SILVER -> workspace -> default
    * GOLD -> workspace -> default

3. You’ll get a notification that both the SILVER and GOLD layers are pointing to the same catalog and schema. While this isn’t always best practice, for the sake of this lab, this is all we need.

    ![image39](databricks-quickstart/image39.png)

4. Make sure to save the mappings you just configured!

## Adding Data Sources

Now that we have access to data, let’s start to build a Graph (DAG) by adding data in the form of Source nodes.

1. Start in the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and select Add Sources.

    ![image40](databricks-quickstart/image40.png)

2. Click to expand the tables within BRONZE storage locations and select all of the corresponding source tables underneath. Click Add Sources on the bottom right to add the sources to your pipeline.

    ![image41](databricks-quickstart/image41.png)

3. You'll now see your graph populated with your Source nodes. Note that they are designated in red. Each node type in Coalesce has its own color associated with it, which helps with visual organization when viewing a Graph.

    ![image42](databricks-quickstart/image42.png)

## Creating Stage Nodes

Now that you’ve added your Source nodes, let’s prepare the data by adding business logic with Stage nodes. Stage nodes represent staging tables within Databricks where transformations can be previewed and performed.

1. Let's start by adding a standardized "stage layer" for all sources.

2. Press and hold the Shift key to multi-select all of your Source nodes. Then right click and select Add Node > Stage from the drop down menu.

    ![image43](databricks-quickstart/image43.png)

3. You will now see that all of your Source nodes have corresponding Stage tables associated with them.

    ![image44](databricks-quickstart/image44.png)

4. Double click on the stg_customer node. This will open the node editor.

    ![image45](databricks-quickstart/image45.png)

5. There are three components to the node editor. We will be using all three throughout this lab, so let’s get you familiar. The first is the mapping grid, where we can view our columns, build transformations, supply documentation, and view other metadata.

    ![image46](databricks-quickstart/image46.png)

6. Next, to the left of your screen are the configuration options for the node. These options are standardized based on the definition provided when creating the node.

    ![image47](databricks-quickstart/image47.png)

7. Finally, at the bottom of your screen are the Create and Run buttons. Selecting these buttons will execute the DDL and DML respectively for each node.

    ![image48](databricks-quickstart/image48.png)

## Applying Single Column Transformations

1. Let’s explore how to apply SQL transformations to columns in the mapping grid.

2. Double click into the Transform column of the mapping grid next to the c_name column. This will open a SQL editor for this column. Let’s apply a simple UPPER function to this column.

    ```SQL
    UPPER(`customer`.`c_name`)
    ```

    ![image49](databricks-quickstart/image49.png)

3. Next, let’s add a new column to our table. Double click on the grey “column name” denotation at the bottom of the mapping grid. This will allow you to add a new column by naming it. In this case, let’s call the column c_baltype.

    ![image50](databricks-quickstart/image50.png)

4. When adding a new column, you also need to provide the data type of the column. In this case, we’ll be denoting what type of balance each customer has, so we’ll be using a STRING data type.

    ![image51](databricks-quickstart/image51.png)

5. Next, we’ll provide CASE WHEN logic to determine what type of balance each customer has. Again, double click into the Transform column of the mapping grid, next to the c_baltype column to open the SQL editor. Supply the following SQL statement

    ```SQL
    case when `customer`.`c_acctbal` > 10 then ‘Large’ else ‘Small’ end
    ```

    ![image52](databricks-quickstart/image52.png)

6. Once complete, select the blue Create button at the bottom of the screen. This will execute the DDL for the node.

    ![image53](databricks-quickstart/image53.png)

7. Coalesce allows you to view the SQL of each execution at any time. In this case, we can view the exact DDL that was written on your behalf.

    ![image54](databricks-quickstart/image54.png)

8. Now that the table is created, select the blue Run button to execute the DML of the table, to populate it with data.

    ![image55](databricks-quickstart/image55.png)

9. Coalesce will take into account all of the configuration options and SQL transformations you as a user have provided, and apply them to the DML. In this case, we can view the UPPER and CASE WHEN transformations we applied.

    ![image56](databricks-quickstart/image56.png)

10. Navigate back to the Browser in Coalesce. In the upper right corner of the screen, there is a blue Run All button. Next to it is an ellipsis. Select the ellipsis and choose, Create All. This will execute the DDL for all of the objects in the current pipeline.

    ![image57](databricks-quickstart/image57.png)

11. Once complete, select the Run All button to populate all of the tables with data.

    ![image58](databricks-quickstart/image58.png)

## Joining Nodes

Let’s now join your STG_ORDERS data with your STG_LINEITEM data.

1. Using the shift key on your keyboard, select the stg_orders and stg_lineitem nodes at the same time. Right click on either node and select Join Nodes -> Stage

    ![image59](databricks-quickstart/image59.png)

2. Coalesce will drop you inside of the node. Select the pencil icon in the upper left corner of the screen and rename the node to stg_order_master.

    ![image60](databricks-quickstart/image60.png)

3. Now, select the Join tab of your stG_order_master node in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement.

    ![image61](databricks-quickstart/image61.png)

4. Add column relationships in the last line of the existing Join statement using the bolded code below:

    ```SQL
    FROM {{ ref('GOLD', 'stg_lineitem') }} `stg_lineitem`
    INNER JOIN {{ ref('GOLD', 'stg_orders') }} `stg_orders`
    ON `stg_lineitem`.`l_orderkey` = `stg_orders`.`o_orderkey`
    ```

    ![image62](databricks-quickstart/image62.png)

5. Select the Create and Run buttons to build the table in Databricks and populate it with data.

6. Let’s perform this same process again with the stg_partsupp and stg_part nodes. Again, hold down the shift key and select both nodes, right click on either node, and select Join Nodes -> Stage.

    ![image63](databricks-quickstart/image63.png)

7. Rename the table to stg_part_master.

    ![image64](databricks-quickstart/image64.png)

8. Navigate to the join tab and the following join condition:

    ```SQL
    FROM {{ ref('GOLD', 'stg_part') }} `stg_part`
    INNER JOIN {{ ref('GOLD', 'stg_partsupp') }} `stg_partsupp`
    ON `stg_part`.`p_partkey` = `stg_partsupp`.`ps_partkey`
    ```

    ![image65](databricks-quickstart/image65.png)

9. Create and Run the node to build the table in Databricks and populate it with data. Now you have two joined tables that now exist in your Databricks account.

## Creating Dimension Nodes

Now let’s experiment with creating Dimension nodes. These nodes are generally descriptive in nature and can be used to track particular aspects of data over time (such as time or location). Coalesce currently supports Type 1 and Type 2 slowly changing dimensions, which we will explore in this section.

1. First, let’s remove any nodes we no longer need. Let’s remove the nation, stg_nation, region, and stg_region nodes by holding down the shift key and selecting each node. Right click on any of the highlighted nodes and select Delete Nodes.

    ![image66](databricks-quickstart/image66.png)

2. With the nodes we care about remaining, again, holding down the shift key, select the stg_customer and stg_supplier nodes, right click on either node, and select Add Nodes -> Dimension.

    ![image67](databricks-quickstart/image67.png)

3. This will create two dimension nodes that will be ready to configure. Double click in the dim_customer node.

    ![image68](databricks-quickstart/image68.png)

4. The node will look a little different than the stage nodes we’ve been working with. Specifically, within the configuration options, we can specify a business key, as well as change tracking columns for type two slowly changing dimensions. Select c_custkey as the business key for the node.

    ![image69](databricks-quickstart/image69.png)

5. Let’s also track changes to columns on this table using the change tracking columns. Select the c_name and c_address columns as the columns we want to track changes on.

    ![image70](databricks-quickstart/image70.png)

6. Click Create and Run the dim_customer node. After it is done running, you can view the best practice SQL that Coalesce generated to support type 2 SCDs.

    ![image71](databricks-quickstart/image71.png)

7. Let’s apply this same process to the dim_supplier node.

    ![image72](databricks-quickstart/image72.png)

8. Pass the s_suppkey column through as the business key

    ![image73](databricks-quickstart/image73.png)

9. This time, we won’t track any changes, so you can just Create and Run the node.

## Creating Fact Nodes

Coalesce supports Fact nodes out of the box, which allow you to easily denote tables that are storing fact or measure level data. Let’s learn how to add a fact node.

1. Select and right click on the stg_order_master table. Select Add Node -> Fact

    ![image74](databricks-quickstart/image74.png)

2. You’ll be dropped inside the Fact node. Within the configuration settings, you can select a business key. This is valuable because Coalesce can use this information to infer join conditions when nodes are joined together. In this case, use the l_orderkey as the business key.

    ![image75](databricks-quickstart/image75.png)

3. Create and Run the node and view your graph

    ![image76](databricks-quickstart/image76.png)

## Adding a View for Reporting

Coalesce supports views for various use cases within data pipelines. In order to use view nodes, you need to turn them on first.

1. Navigate to the build settings in the lower left corner of the screen and select the gear icon.

    ![image77](databricks-quickstart/image77.png)

2. Navigate to Node Types to view all of the nodes available in your workspace. You can see the View is toggled to off.

    ![image78](databricks-quickstart/image78.png)

3. To turn a view on, select the toggle to turn it on.

    ![image79](databricks-quickstart/image79.png)

4. Now, navigate back to the Browser and we’ll join a small star schema together into a view. By holding down the shift key, select the fct_order_master, dim_customer, and dim_supplier nodes, right click on either of them and select Join Nodes -> View.

    ![image80](databricks-quickstart/image80.png)

5. Inside the node, you’ll see all of the columns from all three tables.

    ![image81](databricks-quickstart/image81.png)

6. Let’s configure the join first. Navigate to the join tab to configure the join. Supply the following join condition below:

    ```SQL
    FROM {{ ref('GOLD', 'fct_order_master') }} `fct_order_master`
    LEFT JOIN {{ ref('GOLD', 'dim_customer') }} `dim_customer`
    ON `fct_order_master`.`o_custkey` = `dim_customer`.`c_custkey`
    LEFT JOIN {{ ref('GOLD', 'dim_supplier') }} `dim_supplier`
    ON `fct_order_master`.`l_suppkey` = `dim_supplier`.`s_suppkey`
    ```

7. Now navigate back to the mapping grid. We need to remove columns that have the same name, plus get rid of columns we no longer need. Select the Column Name header on the mapping grid to alphanumerical sort your columns

    ![image82](databricks-quickstart/image82.png)

8. Scroll down to the bottom of the mapping grid to view all of the system columns available. Select the very first system column listed, hold down the shift key, and select the last system column selected. Then, right click on any of the highlighted columns and select Delete Columns.

    ![image83](databricks-quickstart/image83.png)

9. You will need to confirm you want to delete these columns. Select Delete.

    ![image84](databricks-quickstart/image84.png)

10. Create the view! You will notice there is no run button, because views are just stored as queries, they don’t actually store data.

## Bulk Editing Columns, Tests, and Propagation

With your pipeline built, you can add tests to your nodes to ensure your data quality is intact. Additionally, once done, as new columns are added to your pipeline, propagating those changes easily can be incredibly important. Finally, it can be tedious to manually write and update data transformations for each individual column. With Coalesce, you can bulk edit columns quickly and easily. That’s what we’ll learn about in this last section.

1. In the Browser, double click on the stg_order_master node to open it.

2. Select the Data Type Column header in the mapping grid to sort the columns in alphanumerical order by data type.

    ![image85](databricks-quickstart/image85.png)

3. All of the STRING data types will be at the bottom of your screen. By holding down the shift key, we can select the first column containing a STRING data type, and then the last column containing a STRING data type.

    ![image86](databricks-quickstart/image86.png)

4. Right click on any of the highlighted columns and select Bulk Edit

    ![image87](databricks-quickstart/image87.png)

5. Within the Column Editor on the right side of the screen, select the attribute you want to update - in this case, Transform.

    ![image88](databricks-quickstart/image88.png)

6. A SQL editor will open where you can write the SQL statement that you want applied to all of the columns highlighted. We’ll use the SQL statement below, to apply lower casing to all of the columns in our columns.

    ```SQL
    lower({{SRC}})
    ```

    ![image89](databricks-quickstart/image89.png)

7. Select the Preview button to view your changes and select Update to persist the changes to your columns.

    ![image90](databricks-quickstart/image90.png)

8. Next, in the Testing tab in the upper right corner of the node, you can write your own SQL based tests, or use out of the box, null or unique column tests. Navigate to the Column section of the Testing tab.

    ![image91](databricks-quickstart/image91.png)

9. Select both Null and Unique as values you want to test for.

    ![image92](databricks-quickstart/image92.png)

10. Next, select Null for each of the following three keys:

    * l_orderkey
    * l_partkey
    * l_suppkey

11. These tests check to ensure that no null values are contained within these columns. You should also select the Unique test for the l_orderkey column.

    ![image93](databricks-quickstart/image93.png)

12. When you run the node, you’ll be able to see each execution step. Notice that the Unique test on l_orderkey failed. This is because for each line item contained in an order, the order key will be listed. This means if there were 5 line items in an order, the order ID will duplicate 5 times.

    ![image94](databricks-quickstart/image94.png)

13. To solve this, let’s create a business key that is unique to the table. Create a new column called business_key.

    ![image95](databricks-quickstart/image95.png)

14. Give it a data type of LONG.

    ![image96](databricks-quickstart/image96.png)

15. Next, let’s create the unique identifier for the table by providing a transformation that concatenates the l_orderkey and l_linenumber together.

    ```SQL
    concat(`stg_lineitem`.`l_orderkey`, `stg_lineitem`.`l_linenumber`)
    ```

    ![image97](databricks-quickstart/image97.png)

16. Create and Run the node to make the changes.

17. Drag the business_key node to the middle of the mapping grid - this allows us to arrange columns in a more helpful order.

    ![image98](databricks-quickstart/image98.png)

18. Back in the Testing tab, deselect the Unique test from l_orderkey and select the Unique test for the new business_key column.

    ![image99](databricks-quickstart/image99.png)

19. Now, rerun the node and ensure all the tests pass. Congratulations! You’ve successfully, built, transformed, and tested your data.

## Conclusion and Next Steps

Congratulations on completing this entry-level lab exercise! You've mastered the basics of Coalesce and are now equipped to venture into our more advanced features. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

## What we’ve covered

* How to navigate the Coalesce interface
* How to add data sources to your graph
* How to prepare your data for transformations with Stage nodes
* How to join tables
* How to apply transformations to individual and multiple columns at once
* How to build out Dimension and Fact nodes
* How make and propagate changes to your data across pipelines

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our  and .

Reach out to our sales team at coalesce.io

---

## DataOps Best Practices with Git and Coalesce

## Overview

### What Is DataOps?

DataOps has emerged in recent years as a new approach improving the speed, efficiency, and quality of data analytics processes. It is a methodology that combines cross-functional teams, automated data workflows, and standardized data management practices to quickly and effectively process, analyze, and deliver data-driven insights.

In this guide, you will find best practices for applying the principles of DataOps within the context of Coalesce and how to best implement them by fully leveraging our Git integration for version control.

By saving (or committing) any code changes in Coalesce to Git, you will ensure that previous versions of your data warehouse are preserved and accessible. These changes are accessible through a dedicated Git repository, which is a collection of files that store the different versions of your data warehouse.

We strongly recommend setting up Git within your Coalesce instance to perform proper version control and avoid losing your work.

## Introduction to Git

### What Is Git?

Git is a widely used version control solution that is used to store, branch, and merge developers' work. Git maintains a centralized code "database" (known as a Git repository) on a hosted provider such as [GitHub][], [Bitbucket][], [GitLab][] and [Azure DevOps Git][]. This centralized repository is copied / cloned to a Git client (such as Coalesce) and used to manage development operations.

### How Does Git Work?

Git allows developers working on the default / live version of their data warehouse (the main branch) to isolate code changes by creating new branches of work. Branches are then merged together to incorporate new changes into the main branch. All deployments to non-development environments (for example, Test / Production) are performed from the main branch.

### Branching and Merging

Branching provides flexibility in team environments by separating development from deployment. Types of branches include:

- An **integration** branch is used to merge development for system testing and reviews.
- A **feature** branch is a temporary branch that is created from the **integration** branch.
- Feature branches track all changes consisting of a "deployable unit of work" (for example, a new or modified pipeline of nodes). All development should be committed and unit tested in a feature branch.
- Several **feature** branches can be worked on in parallel by different members of the team.
- **Feature** branches are merged into an **integration** branch for system testing and review.
- When development is complete and ready for integration, a feature branch is merged back into an integration or main branch.

### Git Permissions

Git permissions and approvals can be set up directly on your Git platform (not through Coalesce). Any Git pipelines (also known as actions or runners) can be configured along with Coalesce's Command Line (CLI) utility to automate deployments and executions.

Git operations such as pull requests and merge conflict resolution can be done through Coalesce, or off of the platform via Git commands or other clients (VS Code).

## Using Version Control in Coalesce

### 1. Create a Project

The first step to setting up version control is by creating a Project in Coalesce. Projects are logical groupings of data warehousing efforts in Coalesce, and a good way to organize your work by a particular initiative or team focus.

When you create a Project, you'll be asked for a Git repository URL. Git repository URLs are used by Git hosting services (GitHub) as a way of pointing to a Git repository. However, everyone still has a full copy of the repository on their own machine. Coalesce generates and manages metadata which is converted and executed as SQL code,  its this metadata that is stored in Git as YAML files.

After adding the Git repository URL, you'll be asked to add your Git credentials. Your credentials are similar to a username and password, and a way of controlling access to the Git repository.

### 2. Create a Workspace Within Your Project

Workspaces are dedicated development areas within Coalesce where you can build out your data warehouse and apply transformations to data.

When you create a Workspace, you'll be asked to select a Git branch, and create a new branch from it.

Branches are one of the most powerful features of Git, as they allow different people to work on the same items at the same time, and later combine (merge) their changes. This is particularly useful if you need to work on a hotfix for your data warehouse, or need to add new items to it that shouldn't be exposed to end users just yet.

With feature branches, teams can make changes to the main data warehouse while other people work on hotfixes or new features. Any changes made to the main data warehouse can be synced with the new features, and vice-versa.

### 3. Transform Data in Your Workspace

Once your Workspace has been created, you can start creating or modifying your data warehouse. Although your workspace is linked to Git, Coalesce doesn't automatically save your changes to Git - changes are, however, saved somewhere else, and you won't risk losing your work. However, it is a good idea to commit your changes when you're at a good stopping point (for example, when creating a node or multiple nodes, mapping storage locations, building a subgraph, building a job). You should also consider committing if you need to work on another Workspace, or if you want to make your work visible to other members of your team.

### 4. Commit Your Changes to Git and Deploy to Non-Development Environment

A commit is a snapshot in time of your Workspace. You can use commits to go back to a previous version of a workspace, or to troubleshoot recent changes. Commits are also the mechanism Coalesce uses to deploy to non-development Environments.

When you trigger an environment deployment, you'll be asked to select a branch (a Workspace) and a commit (the snapshot in time of that Workspace). When you perform a commit, your work becomes visible to anyone else viewing the Workspace. Before a commit, only you can see those changes - so it is a good idea to commit often.

Anyone that has access to the Project your workspace is part of can also see those commits; they can use your commit as a starting point for a new workspace, and build on the work you've done.

<figure>
  
  <figcaption>Example of a high-level Git workflow in Coalesce</figcaption>
</figure>

## Step-By Step: Basic Git Setup & Configuration

### Starting a New Project

1. Create a new Project with a unique name and description.

    

2. Set up version control within your Project by creating and attaching a dedicated Git repository.

    

3. Next, link your Git account to your new Project and repository. Your Git credentials can be found in your User Settings if you have an existing account, or you can add a new account altogether.

    

4. Click on the question mark icon for instructions on [how to add your credentials][] depending on your Git provider. Make sure to test your account before you finish creating the Project.

    

5. To start building, create a new Workspace in your Project.

    

6. Create a new Git branch associated with your Workspace. Every Workspace is attached to its own Git branch, meaning that you can have multiple branches of work happening in parallel across different Workspaces within your Project.

    :::info[Naming Workspaces]

    Name your Workspace the same as your branch to minimize confusion. This will be helpful as you work with many different Workspaces and branches.

    :::

    

7. Launch your new Workspace and complete your [Build settings][] by connecting to data platform and adding storage locations.

    

8. Switch to your data platform (outside of Coalesce) and set up your environment to include the following:

    - Target databases
    - Target schemas
    - Users / Permissions
    - Warehouses
    - Source databases
    - Source schemas access

9. Create corresponding Storage Locations in Coalesce. There should be one for each unique database schema you plan to use in your data platform.

    

10. Click the gear icon in the lower left hand corner of the interface to open your Build settings and map any Storage Locations to databases / schemas.

    

    :::info[Workspace and Environments]

    There's a difference between a Workspace and an Environment in Coalesce.

    A Workspace is a development area where you build your pipelines. You will commit code from Workspaces to feature branches in Git, and then deploy your committed code to a target Environment (such as UAT, Production, etc).

    :::

11. Create at least one target Environment and also map the Storage Locations to the corresponding databases / schemas.

    

12. Rename DEV branch to MAIN, fill in the configuration and Commit

    

    When you first set up your Organization in Coalesce, you'll notice that a single Workspace called DEV is created. Rename this to "Main."

    :::info[Feature Branches for Development]

    If you plan to use feature branches for development, this step is required and usually completed by a user with Admin privileges.

    When Coalesce detects an empty Git repository, it will automatically generate a data.yml file to store all configuration information such as your connections, storage locations and mappings, node types, macros, and so on.

    Because of this, it's important to perform an initial commit to add a valid data.yml file to Git. This will version all of the configurations currently set in the Main branch, and any feature branch Workspaces created from Main will inherit these configurations.

    :::

    

13. Create feature branches by duplicating the Main branch. Use a meaningful naming convention for your Workspaces and branches with descriptions. Tag colors can be used to provide additional context to the user based on the organization's preferences. For example, all feature workspaces could be BLUE, main Workspaces GREEN, testing Environments YELLOW, and production Environments RED.

    

14. Commit your configuration to Git (see illustrative workflow below).

    

## Creating Your Development Strategy

Your development strategy will vary depending on team size, geographical location, communication frequency, and your organization's overall culture. We recommend developing a baseline strategy that can be used as a starting point and adjusted based on your needs.

### Step-by-Step: Creating a Feature Branch

1. Create a feature branch by creating or duplicating a Workspace. Note that each branch is associated with a unique Workspace.

    

2. Re-confirm user credentials within your Workspace and test your connectivity before proceeding.

    

:::info[Update Target Storage Locations]

Update your target Storage Location mappings to point to a different set of database and schema pairs. This is recommended as a best practice so that developers are working in separate environments.

:::

## Starting the Development Process

### Step-by-Step: Developing in Your Feature Branch

1. Start developing your graph in the Workspace / feature branch that you just created.

    

2. Commit your code using the Git modal. The list of YAML files will include changes made by any developer using the same Workspace.

    

    When the Git modal is opened, a snapshot of the code for a particular Workspace is taken and is only refreshed by closing and re-opening the Git modal again. If changes were made in a different browser, they would not be reflected in the list.

    

    :::info[Commit Individual Files]

    Instead of committing all changes, individual YAML files can be excluded from the commit, by sliding the switch to the right of the file.

    The ‘Add All' button will add any excluded files back and the ‘Remove All' button will exclude everything. Excluded files will reappear as a change, the next time the Git modal is opened.

    The Fetch button can be used to re-sync the Git modal with any changes that have been made outside of Coalesce.

    To permanently discard changes (and effectively roll back the code to the last commit), choose the Discard All Changes button. Alternatively, the changes to individual YAML files, can be discarded by using the reverse icon next to the file. Discarding changes cannot be recovered and could potentially undo other developers' code who may have worked in the same Workspace.

    :::

    :::info[Commit Tips]

   - Make frequent commits using meaningful commit descriptions.
   - Ensure that all members of the development team are aware that a commit is planned, so that all code for the Workspace is in a ‘commit ready' state.
   - Consider allocating a single developer the task of performing commits to the Main branch, to avoid any possible conflicts.
   - Do not have more than one instance of the Git modal open for a single Workspace, at the same time.

    :::

3. Make an Initial commit of data.yml (containing configuration), into the ‘Main' branch.
4. Branch ‘Main' into a feature branch and begin development.

    

5. Make frequent commits as development continues on the particular feature.

    

6. When the feature is complete, merge the code back into ‘Main' and discard the feature Workspace.

    

7. Merge back to Main Workspace by opening the target Workspace (associated with ‘Main'). Then, from within the Git modal, choose the required commit from the source feature branch and click on ‘Merge'.

    Alternatively, if the latest commit is being merged, choose the ‘Merge Latest' button.

    :::info[Merge and Conflicts]

    If the code can be merged, it will happen instantly and there will not be an option to view the changes beforehand (Pull Requests are not supported in Coalesce).

    If there is a conflict, then the differences will be displayed in a separate window. It is recommended that all conflicts are handled outside of Coalesce (for example, in GitHub).

    :::

    

    :::info[Deploy to Target Environments]

    Only deploy to target Environments from a ‘Main' branch, rather than from feature branches.

    :::

## Deployment Process

1. Deploy code into a target Environment. Code can be deployed from a development Workspace into a target Environment, by using the GUI, the CLI tool or by using the API.
2. By choosing a particular commit to deploy, a comparison between the previous deploy (also considering the state of the target environment) and the later version of the code is made, before a ‘plan' of the changes is generated. The plan will contain all of the SQL steps that will be executed to deploy the changes that can be reviewed before the actual deployment takes place.

    

:::info[Storage Locations]

Be sure to set up all Storage Locations in advance.

:::

## Hotfix Management

Hotfixes are typically released in response to urgent issues that cannot wait for the next scheduled software update or release. They are designed to quickly resolve a specific problem without requiring a full software update or installation.

Hotfixes are usually smaller and less comprehensive than regular software updates, and they are often focused on fixing a single issue or a small set of related issues. They are usually distributed to customers as a standalone patch or a small package that can be easily installed on top of the existing software installation.

We recommend that you follow these steps to resolve hotfixes.

1. Determine the scope of the hotfix.

    

2. Branch from Main to create a hotfix branch.

    

3. Implement the code to resolve the hotfix.

    

4. Deploy to QA for testing and if everything looks good, continue to deploy to Production.

    

5. Merge main branch into the hotfix branch and verify everything is working properly.

    

6. Merge hotfix branch into the main branch and continue the release to QA and then to Production as usual.

## Conflict Merging

Merging branches allows for the new development and changes in two feature branches to be combined into one branch. If no common objects have been changed, then the merge is straightforward and can be managed in Coalesce.

If the same object has been changed on two different branches that are being merged, then a merge conflict is identified. The developer responsible for the merge needs to decide which version of the change should be integrated and deployed, and resolve the merge conflict.

The merge conflict should be resolved in the Git host portal or in Git client such as VS Code. Merges are only allowed when there are no uncommitted changes in the target branch. A warning will be displayed if there have been changes, which will either need to be discarded or committed before a merge attempt can take place.

If there have been changes made by another developer while the Git modal is open, merging could reverse their changes without a warning. Often, merge types are dictated by the complexity of changes since the last merge.

These processes are applicable to any target Environments that have been created. Commits in the Git modal will be tagged with the Environments that reflect that particular version of code.

### Step-by-Step: Resolving Merge Conflicts (Example Exercise)

In this example we will purposefully create a merge conflict and demonstrate how to resolve it in GitHub.

1. Begin in your primary Workspace and create a new Stage node.

    

2. Commit all changes in Git.

    
  
3. Now create a new Workspace and branch from the primary Workspace where changes were just committed.

    

4. Make a change to your node's description and commit the change to Git.

     

5. Switch to the primary Workspace and add something different to the description field of your node. Commit the change. This step will create a merge conflict to resolve off platform.

    

6. Now in the primary branch, try to merge the copied branch in. It should result in a merge conflict.

      

7. Instead of using the Coalesce merge conflict editor, close the pane and go to GitHub.

    

8. Once in GitHub you will see a prompt to create a pull request. Create the pull request.

    

9. After creating the pull request, a message will appear saying that the branches cannot be automatically merged due to conflicts. Click the "Resolve conflicts" button.

    

10. In the conflict editor, markers will indicate where the unresolvable differences were between the two files.

    

11. Delete the code you don't want to keep, as well as all of the Git generated conflict markers. After you do this, click the "Mark as Resolved" button in the upper right and then click "Commit Merge."

    

12. After you commit the merge, click the green button on the next screen that says "Merge Pull Request."

    

13. From here you will be able to see your changes reflected in Coalesce. To do this, go to the Git window and click "Resync Branch."

    

14. After the branch is re-synced, you can check in your node and verify that the change you selected in Git exists.

    

Congrats. You just resolved a merge conflict.

[GitHub]: https://github.com/
[Bitbucket]: https://bitbucket.org/
[GitLab]: https://gitlab.com/
[Azure DevOps Git]: https://dev.azure.com/
[how to add your credentials]: /docs/setup-your-project/setup-version-control
[Build settings]: /docs/build-your-pipeline/the-build-interface/build-settings/

---

## Migrating SQL to Coalesce with Copilot

<!-- vale off -->

## Overview

Learn how to migrate existing SQL code into Coalesce using Copilot. Whether you have stored procedures, views, ad-hoc queries, or legacy transformation logic, Copilot can analyze your SQL and generate a DAG of Nodes with transformations, columns, and relationships preserved. This guide walks you through the process from preparation through validation and refinement, with copy-paste examples you can use.

## What You Can Migrate

Copilot handles the following sources:

| Source | What to Extract | Notes |
| ------ | ----------------- | ----- |
| **Stored procedures** | The full procedure | Paste the whole procedure. Copilot creates a Node for each CTE and the final INSERT. Add any uncaptured setup or cleanup to Pre-SQL or Post-SQL. |
| **Views** | The `SELECT` statement from `CREATE VIEW x AS` | Often the cleanest input; views are usually self-contained. Works best when the view has no external dependencies or when you already have matching source Nodes in your Workspace. |
| **Ad-hoc queries** | The full query | Paste as-is if it uses standard SQL. |
| **CTE-based scripts** | The full script | CTEs map well to Stage Nodes; each CTE can become a Node. |
| **dbt models** | The compiled SQL or model query | Remove Jinja; paste the resulting SQL. |
| **Legacy ETL code** | Transformation logic from WhereScape, SAP HANA, Informatica, or similar | Extract the SQL portions (SELECT, INSERT, MERGE, and similar statements; ignore control flow and scripting). Platform-specific syntax (date functions, string handling, and so on) may need adjustment for your target data platform. |

:::info Best Fit
Copilot excels at converting SQL with clear structure: CTEs, joins, aggregations, and standard transformations. The more modular your SQL, the cleaner the resulting DAG.
:::

## Prerequisites

Before you start, ensure you have:

- **Coalesce 7.27+ with Copilot** — [Enable Copilot](/docs/coalesce-ai/copilot#enable-copilot) in your Workspace.
- **[Workspace permissions][]** to create and modify Nodes — Copilot creates and edits Nodes on your behalf; you need write access.
- **Source Nodes already in your Workspace** — Copilot cannot create source Nodes. It matches table names in your SQL to existing sources; add your source tables first.
- **[Storage Locations][]** configured — Each generated Node needs a target schema. Without Storage Locations, Copilot can't assign where tables or views are deployed.
- **A version control checkpoint** — Create a checkpoint (commit or save point) before starting. If Copilot produces unexpected results, you can discard changes and revert.

## Step 1: Prepare Your SQL

Preparation means getting your SQL into a form Copilot can process efficiently. A few minutes of cleanup before pasting reduces manual fixes later and improves the quality of the generated DAG.

**Things you can do:**

- **Gather your SQL in one place** — Copy the full procedure, view definition, or script into a single file or editor. Copilot handles hundreds of lines; very large procedures may need to be split into logical chunks.
- **Simplify where possible** — Remove or comment out session variables, temporary setup, or environment-specific logic that doesn't belong in the transformation.
- **Use clear table references** — Avoid aliases like `t.col` when you can; use `table_name.column_name` instead. Aliases can break column lineage and require manual re-mapping after migration.
- **Match table names to your Workspace** — Ensure table names in your SQL match the names of your existing source Nodes in Coalesce.
- **Resolve templating first** — For dbt models, run `dbt compile` and paste the compiled SQL instead of Jinja; Copilot needs plain SQL.

:::info[SQL Preparation Examples]
The following are examples of preparing SQL for migration.
:::

### Migrating a Stored Procedure

You can paste the entire stored procedure into Copilot. Copilot creates a Node for each CTE and for the final INSERT, so a procedure with two CTEs and an INSERT becomes three Nodes.

**Example procedure:**

```sql
CREATE OR REPLACE PROCEDURE refresh_daily_sales()
RETURNS VARCHAR
LANGUAGE SQL
AS
$$
DECLARE
    run_date DATE DEFAULT CURRENT_DATE();
BEGIN
    CREATE OR REPLACE TEMP TABLE temp_run_params AS
    SELECT run_date AS filter_date;

    INSERT INTO target_sales
    WITH filtered_orders AS (
        SELECT * FROM orders WHERE order_date = (SELECT filter_date FROM temp_run_params)
    ),
    aggregated AS (
        SELECT customer_id, SUM(amount) AS total FROM filtered_orders GROUP BY customer_id
    )
    SELECT * FROM aggregated;

    RETURN 'Success';
END;
$$;
```

**Paste this whole procedure into Copilot.** Copilot creates a Node for `filtered_orders`, a Node for `aggregated`, and a Node for the final INSERT (three Nodes total).

#### Add Setup and Cleanup (If Needed)

Copilot creates Nodes from the CTEs and INSERT, but some procedure logic may not be captured—for example, temp table creation that runs before the transformation, or logging that runs after. Add that logic using [Pre-SQL and Post-SQL](/docs/build-your-pipeline/pre-post-sql):

| Procedure logic Copilot may not capture | Where to add it in Coalesce |
| --------------------------------------- | --------------------------- |
| Temp table creation, session setup | **Pre-SQL** on the first Node in the pipeline |
| Post-insert logging, cleanup, or audit | **Post-SQL** on the final Node |
| Dynamic SQL or cursor loops | Pre-SQL, Post-SQL, or a separate Node |

**Example:** The procedure above creates `temp_run_params` before the transformation. The first Node (`filtered_orders`) references it in its WHERE clause. Add Pre-SQL on that Node so the temp table exists when the Node runs:

```sql
CREATE OR REPLACE TEMP TABLE temp_run_params AS
SELECT CURRENT_DATE() AS filter_date;
```

For logging or audit steps (such as the `RETURN 'Success'`), add them to **Post-SQL** on the final Node.

### Migrating a View

#### Step 1: Retrieve the View Definition

Get the view's DDL from your data platform:

- **Snowflake:** `SELECT GET_DDL('VIEW', 'database.schema.view_name');`
- **BigQuery:** Query `INFORMATION_SCHEMA.VIEWS` for the `view_definition` column
- **Databricks:** `DESCRIBE EXTENDED catalog.schema.view_name` (returns view text), or query `INFORMATION_SCHEMA.VIEWS` for the `VIEW_DEFINITION` column (Unity Catalog)
- **Microsoft Fabric:** Use the view metadata in your workspace or script the view definition from the Fabric portal

You'll get something like:

```sql
CREATE OR REPLACE VIEW my_schema.customer_summary AS
SELECT
    customer_id,
    customer_name,
    region,
    SUM(order_amount) AS total_spent
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE o.order_date >= '2024-01-01'
GROUP BY customer_id, customer_name, region;
```

#### Step 2: Paste the View Into Copilot

Paste the full view definition into Copilot with instructions such as: "Convert this view into a Coalesce View Node. Use my existing source Nodes CUSTOMERS and ORDERS." Copilot creates a View Node with the transformation logic and column mappings. If your source Node names differ from the table names in the view (for example, the view uses `customers` but your Node is `SOURCE.CUSTOMERS`), update the SQL to match your Workspace names or specify the mapping in your instructions.

### Migrating a dbt Model

#### Step 1: Get Plain SQL

dbt models use Jinja templating. Copilot needs plain SQL, so compile first:

1. Run `dbt compile` in your project directory.
2. Open the compiled SQL in `target/compiled/<project>/models/<model_name>.sql`.
3. Copy the contents. All `{{ ref() }}`, `{{ config() }}`, and other Jinja will be resolved.

**Example:** A dbt model like this:

```sql
{{ config(materialized='view') }}
SELECT
    {{ ref('stg_customers') }}.customer_id,
    {{ ref('stg_customers') }}.customer_name,
    SUM({{ ref('fct_orders') }}.amount) AS total_spent
FROM {{ ref('stg_customers') }}
LEFT JOIN {{ ref('fct_orders') }} ON ...
GROUP BY 1, 2;
```

Becomes plain SQL after compile:

```sql
SELECT
    "stg_customers".customer_id,
    "stg_customers".customer_name,
    SUM("fct_orders".amount) AS total_spent
FROM "database"."schema"."stg_customers" "stg_customers"
LEFT JOIN "database"."schema"."fct_orders" "fct_orders" ON ...
GROUP BY 1, 2;
```

:::info Stored Procedures and Session Logic
Complex stored procedures sometimes include logic that Copilot doesn't capture, such as custom session variables or Pre-SQL and Post-SQL. Teams migrating large stored procedure estates often add missing logic to Pre-SQL and Post-SQL manually. Validate the generated Nodes against your original procedure behavior.
:::

## Step 2: Create a Checkpoint and Open Copilot

Before you start, create a safety net and open Copilot with the right settings.

1. **Create a version control checkpoint** — In the Coalesce App, open the version control modal (Git icon) and create a checkpoint (commit or save point). If Copilot produces unexpected results, you can discard all changes and revert to this point.
2. **Open Copilot** — Click the AI button  in the builder toolbar to open the Copilot chat panel.
3. **Enable Allow Edits** — Ensure **Allow Edits** (Edit Mode) is turned on in the Copilot panel. If it's off, Copilot can only suggest changes; it won't create or modify Nodes. You must enable it for the migration workflow to work.

## Step 3: Paste Your Instructions and SQL

Start a new chat and provide clear instructions before pasting your SQL. The instructions set the context so Copilot knows how to interpret your code.

### Example Instructions by Source Type

**For a stored procedure (CTE-based):**

> Reverse engineer this SQL into Coalesce Nodes. Create a Stage Node for each CTE. Use my existing source Nodes INVENTORY_MOVEMENTS and PRODUCTS. Map to my WORK Storage Location.

**For a view definition:**

> Convert this view definition into a Coalesce View Node. Use my existing source Node CUSTOMERS. Name the Node V_CUSTOMER_SUMMARY.

**For an ad-hoc query:**

> Create Nodes from this query. Match CUSTOMERS and ORDERS to my existing sources. Create a Stage Node for the result.

**For a dbt-style model:**

> Import this SQL as Coalesce Nodes. Each CTE should become a separate Stage Node. Use snake_case for Node names.

### What to Specify Upfront

Include any of these in your first message:

- **Node Type preference:** "Create Stage tables" (default; good for intermediate transformations), "Create Views only" (lighter weight; no materialization), or "Make the final Node a Fact table" (for final marts or reporting tables)
- **Naming convention:** "Prefix all Nodes with STG_" or "Use my org's naming: dd_mm_yyyy format"
- **Target Subgraph:** "Put all Nodes in the Inventory Subgraph"
- **Source mapping:** "Use SOURCE.INVENTORY_MOVEMENTS and SOURCE.PRODUCTS as predecessors"

### Paste the SQL

After your instructions, paste the SQL. Paste one script at a time; for multiple procedures or views, run separate Copilot sessions. Copilot will analyze it and create the corresponding Nodes. For a ~400-line stored procedure with multiple CTEs, Copilot typically generates a chain of Stage Nodes, each representing a logical step in your pipeline. Processing may take a minute or two for large inputs.

:::info Live Feedback
Copilot shows progress as it works. You'll see Nodes appear in your Workspace and the graph update in real time.
:::

## Step 4: Review the Generated Pipeline

After Copilot finishes, validate the results before deploying. Work through each check systematically.

### Check the Graph and Lineage

In the builder, the graph shows your pipeline as a DAG. If it doesn't update automatically after Copilot runs, reload it. Perform these checks:

1. Reload the graph or lineage view if it doesn't update automatically.
2. Verify your source Nodes connect to the new Nodes. There should be no orphaned Nodes (Nodes with no incoming or outgoing connections that don't belong).
3. Trace the flow: each CTE or logical step in your SQL should map to a Node. The final Node should represent your final SELECT.

### Inspect Columns and Transforms

Open each generated Node and review the columns. In the Node panel, switch to the column view (Columns dropdown). Perform these checks:

1. Look for the transforms in the Transform column. Those show where Copilot placed transformation logic from your SQL.
2. Compare column names and data types to your original SQL. Ensure business logic is preserved.
3. Check that joins and aggregations are correct. Open the Join tab and Config tab as needed.

    <figure>
    
    <figcaption>Check the Join tab</figcaption>
    </figure>

    <figure>
    
    <figcaption>Review the transforms</figcaption>
    </figure>

### Verify Column Lineage

If you used table aliases in your SQL (for example, `im.product_id` instead of `INVENTORY_MOVEMENTS.product_id`), Copilot may have placed the expression in the Transform field instead of mapping directly to the Source column. That breaks column lineage.

**To fix:** In the Coalesce UI, open the Node, find the affected columns, remove the transform, and map the column to the correct source column.

### Confirm Node Types and Storage

- **Node Types:** By default, Copilot often creates Stage Nodes. Ensure they match your intent: Stage (intermediate tables), View (virtual, no materialization), Persistent Stage (materialized staging), Dimension (slowly changing dimensions), or Fact (fact tables for analytics).
- **Storage Locations:** Verify each Node points to the correct Storage Location in the Node Config. Staging tables typically go to a WORK or STAGE schema; final tables may go to a different schema. Change the Storage Location in the Node Config if needed.

### Run a Quick Validation

Before deploying, run these validation steps:

1. Click **Validate Select** on a Node to ensure the SQL compiles. <!-- TODO: add screenshot of Validate button -->
2. Run the Node (Build or Refresh) in a development environment to confirm the output matches expectations.
3. Compare row counts or sample data to your original query if possible.

:::warning Review Before Deployment
Examine all generated SQL, Node configurations, and transformations before deploying to production. Test changes in a development environment first. See [Copilot Best Practices and Limitations](/docs/coalesce-ai/copilot/copilot-best-practices) for a full validation checklist.
:::

## Step 5: Refine and Enhance

Once the base pipeline is in place, you can use Copilot for follow-up changes. Ask in the Copilot chat; Copilot applies changes across the pipeline.

**Convert Node Types.** Ask Copilot to change Stage tables to Views, or to convert the final Node into a Fact table.

**Standardize transformations.** If you have inconsistent null handling (COALESCE, NVL, and so on) across columns, ask Copilot to apply a consistent pattern. For example: "For all VARCHAR columns, apply NVL for null handling and make it consistent."

**Rename columns.** Improve clarity for people who consume your data downstream: "Rename any column containing 'sales AMT' to 'sales_amount'." Copilot propagates renames through downstream Nodes.

**Insert Nodes in the middle.** To add a Persistent Stage between two existing Nodes, create the new Node, then ask Copilot to update the reference. For example: "Modify inventory_status so it references pstage_movements_filtered instead of movements_filtered." Copilot updates references and lineage.

**Adjust data types.** In the Copilot chat, request conversions such as VARCHAR to STRING, or add validation to incoming columns. Verify changes in the Node panel afterward.

**Discard and retry.** If the result isn't what you want, open the version control modal and discard all changes. Create a new checkpoint, then start a new Copilot conversation with revised instructions.

:::tip Copilot and UI Together
Use Copilot for bulk changes and structural updates, then switch to the Coalesce UI for fine-tuning individual column mappings and advanced Node configuration.
:::

## Example: Stored Procedure (Inventory Pipeline)

This example walks through migrating a CTE-based inventory pipeline. You'll paste instructions and SQL into Copilot, then validate the generated Nodes. Use it as a template for your own stored procedures.

### Before You Start

- **Checkpoint and Copilot** — Complete [Step 2: Create a Checkpoint and Open Copilot](#step-2-create-a-checkpoint-and-open-copilot) first. Create a version control checkpoint, open Copilot, and ensure **Allow Edits** is on.
- **Source Nodes** — Create or add source Nodes for `INVENTORY_MOVEMENTS` and `PRODUCTS` in your Workspace. Copilot cannot create source Nodes; it matches table names in your SQL to existing sources.
- **Table name alignment** — If your source Node names differ (for example, `SOURCE.INVENTORY_MOVEMENTS` vs `INVENTORY_MOVEMENTS`), update the SQL to match your Workspace or specify the mapping in your instructions.
- **Storage Location** — Ensure your Workspace has a Storage Location configured (for example, WORK or STAGE) so Copilot can assign where tables are deployed.

### Step 1: Paste Instructions and SQL

Copy the following instructions into a new Copilot chat, then paste the SQL below it:

> Reverse engineer this SQL into Coalesce Nodes. Create a Stage Node for each CTE. Use my existing source Nodes INVENTORY_MOVEMENTS and PRODUCTS. Map to my WORK Storage Location.

```sql
-- Step 1: Filter to recent movements and join with product details
WITH movements_filtered AS (
    SELECT
        im.movement_id,
        im.product_id,
        im.warehouse_id,
        im.quantity,
        im.movement_date,
        im.movement_type,
        p.product_name,
        p.category
    FROM INVENTORY_MOVEMENTS im
    JOIN PRODUCTS p ON im.product_id = p.product_id
    WHERE im.movement_date >= DATEADD(day, -90, CURRENT_DATE())
),

-- Step 2: Aggregate movements by product and warehouse
movements_by_product AS (
    SELECT
        product_id,
        warehouse_id,
        product_name,
        category,
        SUM(CASE WHEN movement_type = 'IN' THEN quantity ELSE -quantity END) AS net_quantity,
        COUNT(*) AS movement_count
    FROM movements_filtered
    GROUP BY product_id, warehouse_id, product_name, category
),

-- Step 3: Calculate inventory status and flag low stock
inventory_status AS (
    SELECT
        product_id,
        warehouse_id,
        product_name,
        category,
        net_quantity,
        movement_count,
        CASE
            WHEN net_quantity <= 0 THEN 'OUT_OF_STOCK'
            WHEN net_quantity < 10 THEN 'LOW_STOCK'
            ELSE 'IN_STOCK'
        END AS stock_status
    FROM movements_by_product
)

SELECT * FROM inventory_status
ORDER BY stock_status, net_quantity ASC;
```

### Step 2: Review the Generated Pipeline

After Copilot finishes, you should see three Stage Nodes (one per CTE) plus a final Node for the ordered result. Each Node connects to the previous one in the pipeline. Work through the checks in [Step 4: Review the Generated Pipeline](#step-4-review-the-generated-pipeline):

1. Verify source Nodes connect to the new Nodes and the graph shows a clear flow.
2. Open each Node and check the Transform column for correct column mappings; fix any alias-related transforms if needed.
3. Click **Validate Select** on the final Node to ensure the SQL compiles.
4. Run the Node in a development environment and compare output to your original procedure if possible.

From there, you can convert the final Node to a View or Fact table, or add more transformations using Copilot or the UI.

## Example: Ad-Hoc Query (Customer Spending)

This example shows how to migrate a standalone query with a join and aggregation. It's simpler than a stored procedure: one query becomes one (or a few) Nodes.

### What You Need

- **Checkpoint and Copilot** — Complete [Step 2: Create a Checkpoint and Open Copilot](#step-2-create-a-checkpoint-and-open-copilot) first. Create a version control checkpoint, open Copilot, and ensure **Allow Edits** is on.
- **Source Nodes** — Create or add source Nodes for `CUSTOMERS` and `ORDERS`. Copilot cannot create source Nodes; it matches table names in your SQL to existing sources.
- **Table name alignment** — Ensure table names in your SQL match your Workspace (for example, `CUSTOMERS` and `ORDERS`). If your sources use different names, adjust the SQL or specify the mapping in your instructions.

### Paste the Query

Copy the following instructions into a new Copilot chat, then paste the SQL below it:

> Create Nodes from this query. Match CUSTOMERS and ORDERS to my existing sources. Create a Stage Node for the result.

```sql
SELECT
    c.customer_id,
    c.customer_name,
    c.region,
    SUM(o.order_amount) AS total_spent,
    COUNT(o.order_id) AS order_count
FROM CUSTOMERS c
LEFT JOIN ORDERS o ON c.customer_id = o.customer_id
WHERE o.order_date >= '2024-01-01'
GROUP BY c.customer_id, c.customer_name, c.region
ORDER BY total_spent DESC;
```

### Review the Generated Pipeline

Copilot matches `CUSTOMERS` and `ORDERS` to your existing source Nodes and builds a transformation Node with the join and aggregation. You should see a single Stage Node (or a small chain) representing the query result. Work through the checks in [Step 4: Review the Generated Pipeline](#step-4-review-the-generated-pipeline):

1. Verify the source Nodes connect to the new Node and the graph shows the correct lineage.
2. Open the Node and check the Transform column; if you used aliases (`c.customer_id`, `o.order_amount`), Copilot may have placed expressions in the Transform field instead of mapping to source columns. Fix any broken lineage by re-mapping to the correct source column.
3. Click **Validate Select** to ensure the SQL compiles.
4. Run the Node and compare row counts or sample data to your original query.

## Common Pitfalls and Troubleshooting

The following table lists common issues and how to resolve them:

| Issue | Resolution |
| ----- | ---------- |
| Table aliases break column lineage or lineage looks wrong | After import, open the Node and use the column grid view to see all columns. Check for alias-related transforms; for each affected column, remove the transform (clear the Transform field) and re-map the column to the correct source column. You can fix multiple columns in bulk when using the column grid. |
| Stored procedure session logic missing | Add custom session variables or setup to [Pre-SQL and Post-SQL](/docs/build-your-pipeline/pre-post-sql) manually on the appropriate Node. |
| Copilot doesn't capture everything in complex stored procedures | Migrate in smaller chunks (split the procedure into logical sections), or plan to supplement with manual edits after migration. |
| Very large Nodes hit token limits | Break the migration into smaller SQL segments. Alternatively, migrate incrementally and propagate only new columns to downstream Nodes instead of re-reading entire Nodes. |
| Data type changes don't apply | Copilot may report a change but not apply it; verify in the Node panel and fix manually if needed. |
| Copilot stops responding | Wait 1–2 minutes, then start a new conversation. A fresh context often helps. If it persists, check your connection or try again later. |
| Results don't match expectations | Rephrase your instructions with more specificity. Include Node Type preferences ("Create Stage tables" or "Create Views only"), naming conventions ("Prefix all Nodes with STG_"), and target Subgraph ("Put all Nodes in the Inventory Subgraph"). |
| Need to start over | Open the version control modal (Git icon), discard all changes, and create a new checkpoint before trying again. |

---

## What's Next?

- [Coalesce Copilot](/docs/coalesce-ai/copilot) for core capabilities and workflow
- [Copilot Best Practices and Limitations](/docs/coalesce-ai/copilot/copilot-best-practices) for validation and safety
- [AI Policy](https://coalesce.io/company/ai-policy/) for Coalesce's AI usage guidelines

---

[Workspace permissions]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Storage Locations]: /docs/get-started/coalesce-fundamentals/environments/

---

## Operationalize ML In Your Data Pipelines with Snowflake Cortex

## Overview

This entry-level Hands-On Lab exercise is designed to help you master the basics of Coalesce. In this lab, you’ll explore the Coalesce interface, learn how to easily transform and model your data with our core capabilities, understand how to deploy and refresh version-controlled data pipelines, and build a ML Forecast node that forecasts sales values for a fiction food truck company.

### **What You’ll Need**

- A Snowflake account (either a [trial account][] or access to an existing account)
- A Coalesce account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account)
- Basic knowledge of SQL, database concepts, and objects
- The [Google Chrome browser][]

### **What You’ll Build**

- A Directed Acyclic Graph (DAG) representing a basic star schema in Snowflake

### **What You’ll Learn**

- How to navigate the Coalesce interface
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to join tables
- How to apply transformations to individual and multiple columns at once
- How to build out Dimension and Fact nodes
- How to make changes to your data and propagate changes across pipelines
- How to configure and build an ML Forecast node

By completing the steps we’ve outlined in this guide, you’ll have mastered the basics of Coalesce and can venture into our more advanced features.

---

## About Coalesce

Coalesce is the first cloud-native, visual data transformation platform built for Snowflake. Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

With Coalesce, you can:

- Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
- Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Snowflake functionality
- Analyze the impact of changes to pipelines with built-in data lineage down to the column level
- Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
- Ensure consistent data standards and governance across pipelines, with data never leaving your Snowflake instance

### How Is Coalesce Different?

Coalesce’s unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### Core Concepts in Coalesce

Coalesce currently only supports Snowflake as its target database, As you will be using a trial Coalesce account created with [Snowflake Marketplace][], your basic database settings will be configured automatically and you can instantly build code.

### **Organization**

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

### **Projects**

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Git repository and Snowflake account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

### **Workspaces vs. Environments**

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Snowflake schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, PreProd, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

## Lab Use Case

As the lead Data & Analytics manager for TastyBytes Food Trucks, you're responsible for building and managing data pipelines that deliver insights to the rest of the company. There customer-related questions that the business needs to answer that will help with inventory planning and marketing. Included in this, is building a machine learning forecast that will allow management to determine sales volume for each item on the menu.

In order to help your extended team answer these questions, you’ll need to build a customer data pipeline first.

---

## Before You Start

To complete this lab, please create free trial accounts with Snowflake and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

We recommend using Google Chrome as your browser for the best experience.

| Note: Not following these steps will cause delays and reduce your time spent in the Coalesce environment. |
| :---- |

### Step 1: Create a Snowflake Trial Account

1. Fill out the Snowflake trial account form [The Snowflake Trial Page](https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=Cj0KCQiAtvSdBhD0ARIsAPf8oNm6YH7UeRqFRkVeQQ-3Akuyx2Ijzy8Yi5Om-mWMjm6dY4IpR1eGvqAaAg3MEALw_wcB). Use an email address that is not associated with an existing Snowflake account.

2. When signing up for your Snowflake account, select the region that is physically closest to you and choose Enterprise as your Snowflake edition. Please note that the Snowflake edition, cloud provider, and region used when following this guide do not matter.

   ![image1](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image1.png)

3. After registering, you will receive an email from Snowflake with an activation link and URL for accessing your trial account. Finish setting up your account following the instructions in the email.

### Step 2: Create a Coalesce Trial Account from Snowflake Marketplace

1. Sign in to Snowsight. Make sure you are using the ACCOUNTADMIN role (or a role with privileges to create a database, warehouse, user, and role) and that your Snowflake email address is verified.
2. In the navigation menu, select **Marketplace > Snowflake Marketplace** and search for Coalesce.

      

3. Open the Coalesce listing and select Start free trial in the upper right. To see what the trial includes, review the Your free trial tab.
4. In the Connect to Coalesce dialog, review the information to be shared and the Snowflake objects that will be created, then select Connect to Coalesce.
5. You will be redirected to Coalesce to finish activating your trial account. Fill in your information to complete activation.

Congratulations! You’ve successfully created your Coalesce trial account.

### Step 3: Set Up The Dataset

1. We will be using a M Warehouse size within Snowflake for this lab. You can upgrade this within the admin tab of your Snowflake account.

   ![image6](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image6.png)

2. In your Snowflake account, click on the Worksheets Tab in the left-hand navigation bar.

   ![image7](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image7.png)

3. Within Worksheets, click the "+" button in the top-right corner of Snowsight and choose "SQL Worksheet.”

   ![image8](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image8.png)

4. Navigate to the [Cortex HOL Data Setup File](https://github.com/coalesceio/hands-on-labs/blob/main/cortex-hol/setup_guide.sql) that is hosted on GitHub.

5. Within GitHub, navigate to the right side and click "Copy raw contents". This will copy all of the required SQL into your clipboard.

   ![image9](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image9.png)

6. Paste the setup SQL from GitHub into your Snowflake Worksheet. Then click inside the worksheet and select All (*CMD + A for Mac or CTRL + A for Windows*) and Click "► Run".

7. After clicking "► Run" you will see queries begin to execute. These queries will run one after another with the entire worksheet taking around 5 minutes. Upon completion you will see a message that states your HOL data setup is now complete.

Once you’ve activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started. Click on the Launch button next to your Development Workspace to get started.

   ![image10](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image10.png)

### Step 4: Add the Cortex Package from the Coalesce Marketplace

You will need to add the ML Forecast node into your Coalesce workspace in order to complete this lab.

1. Launch your workspace within your Coalesce account

    ![image11](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image11.png)

2. Navigate to the build settings in the lower left hand corner of the left sidebar

   ![image12](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image12.png)

3. Select Packages from the Build Settings options

   ![image13](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image13.png)

4. Select Browse to Launch the Coalesce Marketplace

    ![image14](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image14.png)

5. Select Find out more within the Cortex package

   ![image15](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image15.png)

6. Copy the package ID from the Cortex page

   ![image16](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image16.png)

7. Back in Coalesce, select the Install button:

   ![image17](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image17.png)

8. Paste the Package ID into the corresponding input box:

   ![image18](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image18.png)

9. Give the package an Alias, which is the name of the package that will appear within the Build Interface of Coalesce. And finish by clicking Install.

   ![image19](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image19.png)

## Navigating the Coalesce User Interface

| About this lab: Screenshots (product images, sample code, environments) depict examples and results that may vary slightly from what you see when you complete the exercises. This lab exercise does not include Git (version control). Please note that if you continue developing in your Coalesce account after this lab, none of your work will be saved or committed to a repository unless you set up before developing.   |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

Let's get familiar with Coalesce by walking through the basic components of the user interface.

​​Once you've activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started.

   ![image20](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image20.png)
   ![image21](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image21.png)

1. Launch your Development Workspace by clicking the Launch button and navigate to the Build Interface of the Coalesce UI. This is where you can create, modify, and publish nodes that will transform your data.

   Nodes are visual representations of objects in Snowflake that can be arranged to create a Directed Acyclic Graph (DAG or Graph). A DAG is a conceptual representation of the nodes within your data pipeline and their relationships to and dependencies on each other.

2. In the Browser tab of the Build interface, you can visualize your node graph using the Graph, Node Grid, and Column Grid views. In the upper right hand corner, there is a person-shaped icon where you can manage your account and user settings.

   ![image22](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image22.png)

3. By clicking the question mark icon, you can access the Resource Center to find product guides, view announcements, request support, and share feedback.

   ![image24](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image24.png)

4. The Browser tab of the Build interface is where you’ll build your pipelines using nodes. You can visualize your graph of these nodes using the Graph, Node Grid, and Column Grid views. While this area is currently empty, we will build out nodes and our Graph in subsequent steps of this lab.

   ![image22](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image22.png)

5. Next to the Build page is the Deploy page. This is where you will push your pipeline to other environments such as Testing or Production.

   ![image25](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image25.png)

6. Next to the Deploy page is the Docs page. Documentation is an essential step in building a sustainable data architecture, but is sometimes put aside to focus on development work to meet deadlines. To help with this, Coalesce automatically generates and updates documentation as you work.

   ![image26](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image26.png)

## Configure Storage Locations and Mappings

Before you can begin transforming your data, you will need to configure storage locations. Storage locations represent a logical destination in Snowflake for your database objects such as views and tables.

1. To add storage locations, navigate to the left side of your screen and click on the Build settings cog.

   ![image27](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image27.png)

2. Click the “Add New Storage Locations” button and name it SALES for your raw customer data.

   ![image28](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image28.png)

3. Now map your storage location to their logical destinations in Snowflake. In the upper left corner next to your Workspace name, click on the pencil icon to open your Workspace settings. Click on Storage Mappings and map your SALES location to the CORTEX_HOL database and the RAW_POS schema. Select Save to ensure Coalesce stores your mappings.

   ![image29](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image29.png)

4. Go back to the build settings and click on Node Types and then click the toggle button next to View to enable View node types. Now you’re ready to start building your pipeline.

   ![image30](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image30.png)

## Adding Data Sources

Let’s start to build the foundation of your data pipeline by creating a Graph (DAG) and adding data in the form of Source nodes.

1. Start in the Build interface and click on Nodes in the left sidebar (if not already open). Click the + icon and select Add Sources.

   ![image31](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image31.png)

2. Select all of the tables from the SALES storage location except for the NEW_ORDERS table. You can click on the check box next to the storage location name to select all of the tables.

   ![image32](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image32.png)

3. You'll now see your graph populated with your Source nodes. Note that they are designated in red. Each node type in Coalesce has its own color associated with it, which helps with visual organization when viewing a Graph.

   ![image33](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image33.png)

4. Select Create All and Run All to create and populate the tables as objects in Snowflake.

## Creating Stage Nodes

Now that you’ve added your Source nodes, let’s prepare the data by adding business logic with Stage nodes. Stage nodes represent staging tables within Snowflake where transformations can be previewed and performed. Let's start by adding a standardized "stage layer" for all sources.

1. Press and hold the Shift key to multi-select all of your Source nodes from the node pane on the left side of your screen. Then right click and select Add Node > Stage from the drop down menu. You will now see that all of your Source nodes have corresponding Stage tables associated with them. If the 14M/38M data sources are being used, drop that suffix in the Node name in this case STG_ORDER_HEADER

   ![image34](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image34.png)

2. Now change the Graph to a Column Grid by using the top left dropdown menu. By using the Column Grid view, you can search and group by different column headers.

   ![image35](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image35.png)

3. Click on the Storage Location header and drag it to the sorting bar below the View As dropdown menu. You have now grouped your rows by Storage Location. Then expand all the columns from your SALES location.

   ![image36](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image36.png)

4. Change back to the Graph view from Column Grid to with the dropdown menu.

   ![image37](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image37.png)

5. Now let’s create the foundation for your data pipeline. In the upper right hand corner, press the Create All button to create your tables in Snowflake. Then click the Run All button to populate your nodes with data.

   ![image38](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image38.png)

## Exploring the Node Editor

Your Node Editor is used to edit the node object, the columns within it, and its relationship to other nodes. There are a few different components to the Node Editor.

1. Double click on your STG_CUSTOMER node to open up your Node Editor. The large middle section is your Mapping grid, where you can see the structure of your node along with column metadata like transformations, data types, and sources.

   ![image39](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image39.png)

2. On the right hand side of the Node Editor is the Config section, where you can view and set configurations based on the type of node you’re using.
      ![image40](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image40.png)

3. At the bottom of the Node Editor, press the arrow button to view the Data Preview pane.

   ![image41](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image41.png)

## Applying Single Column Transformations

Now let’s apply a single column transformation in your STG_CUSTOMER node by applying a consistent naming convention to our customer names. This will ensure that any customer names are standardized.

1. Select the FIRST_NAME column and click the blank Transform field in the Mapping grid. Enter the following transformation and then press Enter:

   ```SQL
   UPPER({{SRC}})
   ```

   ![image42](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image42.png)

2. Repeat this transformation for your LAST_NAME column:

   ```SQL
   UPPER({{SRC}})
   ```

   ![image43](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image43.png)

3. Click the Create and Run buttons to create and repopulate your node.

   ![image44](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image44.png)

## Joining Nodes

In order to prepare the data within your data pipeline, it needs to be processed more with an additional stage layer.

1. Return to the Browser tab and select your STG_ORDER_HEADER data with your STG_ORDER_DETAIL data by holding down the Shift key and clicking on both nodes.

   ![image45](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image45.png)

2. Right click on either node and select Join Nodes, and then select Stage. This will bring you to the node editor of your new Stage node.

   ![image46](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image46.png)

3. Select the Join tab in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement.

   ![image47](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image47.png)

4. Add column relationships in the last line of the existing Join statement using the code below - note that there is a filter included:

   ```SQL
      FROM {{ ref('WORK', 'STG_ORDER_HEADER') }} "STG_ORDER_HEADER"
      INNER JOIN {{ ref('WORK', 'STG_ORDER_DETAIL') }} "STG_ORDER_DETAIL"
      ON "STG_ORDER_HEADER"."ORDER_ID" = "STG_ORDER_DETAIL"."ORDER_ID"
      WHERE "ORDER_TS"::date >= '2022-08-01'
   ```

   ![image48](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image48.png)

5. Since both nodes contain a ORDER_ID and DISCOUNT_ID columns, we will need to rename or remove one of the columns to avoid getting a duplicate column name error. Let’s delete the ORDER_ID and DISCOUNT_ID from the STG_ORDER_DETAIL.

   ![image49](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image49.png)

6. You can rename the node by clicking on the pencil icon next to the node name and give the node a name of your choice. Rename the node to STG_ORDER_MASTER.

   ![image50](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image50.png)

7. To create your Stage Node, click the Create button in the lower half of the screen. Then click the Run button to populate your STG_ORDER_MASTER node and preview the contents of your node by clicking Fetch Data.

   ![image51](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image51.png)

## Creating Dimension Nodes

Now let’s experiment with creating Dimension nodes. These nodes are generally descriptive in nature and can be used to track particular aspects of data over time (such as time or location). Coalesce currently supports Type 1 and Type 2 slowly changing dimensions, which we will explore in this section.

1. Select STG_CUSTOMER and right click, create Dimension node. Because the demographics of your customer base changes over time, creating a Dimension node on top of this data will allow you to capture customer changes over time.

   ![image52](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image52.png)

2. Within the Option configuration section, under Business Key, check the box next to CUSTOMER_ID and press the > button to select it as your business key.

   ![image53](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image53.png)

3. To create a Type 2 dimension, select the FIRST_NAME, LAST_NAME, EMAIL, PHONE_NUMBER columns under Change Tracking. This will allow you to track changes to these columns over time.

   ![image54](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image54.png)

4. In the lower right of your screen, click the Create button to create your Dimension node and then the Run button to populate it with data.

   ![image55](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image55.png)

5. Now let’s create a Type 1 dimension which will help you focus on particular customer details. Navigate back to the Browser tab and select your STG_MENU node. Right click and select Add Node > Dimension from the dropdown menu.

   ![image56](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image56.png)

6. In your DIM_MENU node, navigate to the right and expand the Options drop down in the Config section. Under Business Key, check the box next to MENU_ITEM_ID and press the > button to select it as your business key.

   ![image57](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image57.png)

7. In the lower right of your screen, click the Create button to create your Dimension node and then the Run button to populate it with data.

   ![image58](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image58.png)

## Creating A Fact Node

Fact nodes represent Coalesce's implementation of a Kimball Fact Table, which consists of the measurements, metrics, or facts of a business process and is typically located at the center of a star or Snowflake schema surrounded by dimension tables.

Let’s create a Fact table to allow us to query the metrics surrounding our customer orders.

1. In your Browser tab, select the STG_ORDER_MASTER node. Then right click and select Add Node > Fact.

   ![image59](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image59.png)

2. Navigate to the right and expand the Options drop down in the Config section. Under Business Key, check the boxes next to ORDER_DETAIL_ID. Then press the > button to select them as your business key.

   ![image60](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image60.png)

3. Click the Create and Run buttons to run and populate your FCT_ORDER_MASTER node.

   ![image61](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image61.png)

## Using Cortex Nodes for Processing Data

Now that we have our fact and dimension nodes, we are going to process the data from the STG_REVIEWS table, allowing us to create a sentiment score for the reviews from each of our customers.

1. Right click on the STG_REVIEWS and select Add Node -> CortexML -> Cortex Functions.

   ![image74](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image74.png)

2. Within the LLM_REVIEWS node, open the Cortex Package dropdown in the Config settings of the node.

   ![image75](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image75.png)

3. We are going to create a sentiment score based on the reviews left by our customers. These reviews are found in the REVIEW_TEXT column. Duplicate the REVIEW_TEXT column to keep the original text, as the sentiment score we will create, will be in a new column.

   ![image76](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image76.png)

4. Call the new column REVIEW_SENTIMENT. You can do this by double clicking on the new column name.

   ![image77](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image77.png)

5. Select the Sentiment toggle on the right side of the node in the Cortex Package configuration options. Then, select the REVIEW_SENTIMENT column we just created.

   ![image78](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image78.png)

6. And that’s it. Coalesce will handle the rest. Select Create and Run to create the object and populate it with data, which will include the sentiment score.

   ![image79](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image79.png)

7. Next, we’ll process this data a bit more, so we can see the average review and sentiment for our customers. Right click on the LLM_REIVEWS node and select Add Node -> Stage.

   ![image80](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image80.png)

8. Rename the node to STG_CUSTOMER_SENTIMENT.

   ![image81](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image81.png)

9. Next, remove the following columns from the node:
   - REVIEW_ID
   - REVIEW_TEXT
   - REVIEW_DATE

   ![image82](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image82.png)

10. We want to know the average sentiment and star rating for each of our customers, as many of our customers have left multiple reviews. We can use the Coalesce bulk editing transformation functionality for this. Select the STAR_RATING and REVIEW_SENTIMENT columns and right click on either one and select Bulk Edit.

    ![image83](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image83.png)

11. In the attributions section, select Transform, as this is the operation we want to apply in bulk.

    ![image84](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image84.png)

12. In the transform SQL editor, copy and paste the following snippet of code.

      ```SQL
      AVG({{SRC}})
      ```

      ![image85](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image85.png)

13. Select the Preview button to view the changes and then select Update to apply the changes to your node.

      ![image86](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image86.png)

14. Finally, since we are using aggregate functions in SQL, we need to use a GROUP BY for our non-aggregated column. In the Join SQL Editor, start a new line and supply the following SQL:
    GROUP BY ALL

    ![image87](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image87.png)

15. Now Create and Run the node.
16. Navigate back to the Browser and now we can join the DIM_CUSTOMER node to the STG_CUSTOMER_SENTIMENT node we just created. Select both nodes and right click on either node and select Join Nodes -> View.

    ![image88](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image88.png)

17. Rename the node to V_CUSTOMER.
18. Navigate to the Join table to configure the join. In this case, we need to join on CUSTOMER_ID. Use the code below to configure the join:

      ```SQL
      FROM {{ ref('WORK', 'STG_CUSTOMER_SENTIMENT') }} "STG_CUSTOMER_SENTIMENT"

      LEFT JOIN {{ ref('WORK', 'DIM_CUSTOMERS') }} "DIM_CUSTOMERS"

      ON "STG_CUSTOMER_SENTIMENT"."CUSTOMER_ID" = "DIM_CUSTOMERS"."CUSTOMER_ID"
      ```

    ![image89](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image89.png)

19. Back in the mapping grid, remove one of the CUSTOMER_ID columns, as the table can’t contain two of the same named column.
20. Select Create when ready to create your view. Congratulations. You now have a view that users can query to learn about the sentiment of your customers.

## Creating ML Processing Node

1. To finish preparing your data for ML forecasting, we’ll perform one more join. Navigate back to the Browser and select the FCT_ORDER_MASTER node and the DIM_MENU node by holding down the shift key and clicking on both nodes. Right click on either node and select Join Nodes, and then select Stage. This will bring you to the node editor of your new Stage node.

   ![image91](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image91.png)

2. Select the Join tab in the Node Editor. This is where you can define the logic to join data from multiple nodes, with the help of a generated Join statement. Add column relationships in the last line of the existing Join statement using the code below:

      ```SQL
      FROM {{ ref('WORK', 'DIM_MENU') }} "DIM_MENU"
      INNER JOIN {{ ref('WORK', 'FCT_ORDER_MASTER') }} "FCT_ORDER_MASTER"
      ON "DIM_MENU"."MENU_ITEM_ID" = "FCT_ORDER_MASTER"."MENU_ITEM_ID"
      ```

   ![image92](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image92.png)

3. You can rename the node by clicking on the pencil icon next to the node name and give the node a name of your choice. Rename the node STG_ORDER_MASTER_ITEMS.

   ![image93](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image93.png)

4. Back in the mapping grid, delete one of the duplicate `MENU_ITEM_ID` columns, as well as all the `SYSTEM_` columns and then create and run the node.

   ![image94](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image94.png)

## Transforming Data in a Node

Now we will prepare the data for our ML forecast by selecting the columns we want to work with and applying single column transformations to that data.

The ML Forecast node we will use later in the lab will rely on three columns to produce a forecast:

- Series column
- Timestamp column
- Target column

We will structure our STG_ORDER_MASTER_ITEMS node so that it only contains the columns we will need for our ML Forecast Node.

- MENU_ITEM_NAME -> Series column
- ORDER_AMOUNT -> Target column
- ORDER_TS -> Timestamp column

1. To structure the node with the columns that we need, the first thing we will do is select the columns we want to work with. Select the ORDER_AMOUNT, ORDER_TS, and MENU_ITEM_NAME columns. Using the column drag selector on the left next to each column name, drag the three selected columns to the top of the mapping grid.

   ![image96](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image96.png)

2. Using the shift key, select all of the columns below the ORDER_AMOUNT, ORDER_TS, and MENU_ITEM_NAME columns. Right click on them and delete them.

   ![image97](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image97.png)

3. With the columns we need for our forecast in this node, we will now apply some column transformations. Currently the Timestamp column records values down to the second interval. For this lab, we want to predict order values at the day level. Let’s add this transformation.
   Select the ORDER_TS column and click into the transform field in the mapping grid. You can cast a timestamp as a date, removing the seconds, minutes, and hours, by using the transformation below:

      ```SQL
      {{SRC}}::date
      ```

   ![image98](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image98.png)

4. Next, we need to aggregate our sales at the day level. We can use a sum function for this. You can use the transformation here to sum the ORDER_AMOUNT column.

      ```SQL
      sum({{SRC}})
      ```

   ![image99](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image99.png)

5. We will leave the MENU_ITEM_NAME column as it is. However, because we have included an aggregate function, we will need to use a GROUP BY in the node. Navigate to the JOIN tab so we can finish preparing this node.

   ![image100](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image100.png)

6. Because we are using an aggregate function, we will add a GROUP BY statement. Additionally, the data set is relatively large, so we will narrow down the items we are working with by providing a filter. Within the Join tab, add a new line in the editor by hitting the enter key. Next, type GROUP BY within the editor, followed by the two non-aggregated columns, MENU_ITEM_NAME and ORDER_TS. You can copy the code for this below:

      ```SQL
      FROM {{ ref('WORK', 'DIM_MENU') }} "DIM_MENU"
      INNER JOIN {{ ref('WORK', 'FCT_ORDER_MASTER') }} "FCT_ORDER_MASTER"
      ON "DIM_MENU"."MENU_ITEM_ID" = "FCT_ORDER_MASTER"."MENU_ITEM_ID"
      WHERE "ITEM_CATEGORY" = 'Beverage' 
      GROUP BY ALL
      ```

   ![image102](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image102.png)

7. Create and Run the node to populate it with data for the next section of the lab guide.

## Adding an ML Forecast Node

The next step in building out this data pipeline for getting your ML Forecast Node operational, we will add in the ML Forecast Node to the pipeline.

1. Navigate back to the Browser to see your data pipeline. Select the STG_ORDER_MASTER_ITEMS node that we have been working with and right click on it. Select Add Node -> CortexML -> Forecast.

   ![image103](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image103.png)

2. Within the node, there will be three additional columns that have been added, FORECAST, LOWER_BOUND, and UPPER_BOUND. These are generated automatically by the ML Forecast node and will be used to forecast the values of the configured node.

   Within the config section, select the Forecast Model Input dropdown. This is where we will configure the node to forecast the values we want to predict. Since there are multiple menu items that we will be predicting order amounts for, we will leave the Multi-Series Forecast toggle to on.

   ![image104](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image104.png)

3. For the Model Instance Name, you can name the model any meaningful name you want, but for the sake of this lab, we will call this model `ORDER_ITEM_FORECAST`.

   ![image105](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image105.png)

4. For the Series Column input, we will use the MENU_ITEM_NAME column. This Series Column is the column that is used as the series for predicting values for.

   ![image106](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image106.png)

5. For the Timestamp Column, we will use the ORDER_TS column.

   ![image107](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image107.png)

6. For the Target Column, we will use the ORDER_AMOUNT column. The Target Column is the column that contains the values we want to predict.

   ![image108](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image108.png)

7. In this node type, the exogenous variables toggle is turned on. Exogenous Variables are additional features or columns that the model can use to help refine and predict the output for the model. Our node doesn’t contain any additional columns that are not being used by the required parameters, so we will toggle this off.

   We are presented with a Days to Forecast input that defaults to 30 days. We can update this value to any value of our choosing, but for the sake of this lab guide, will leave the value at 30.

   ![image109](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image109.png)

8. You have now successfully configured the ML Forecast node. Create and Run the node to populate it within Snowflake.

   ![image110](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image110.png)

## Creating an Anomaly Detection Model

Now that you have created your sales forecast, you can view both your historical and new data to look for anomalies. The final step of this pipeline will be adding the Anomaly Detection node.

1. Now we need our new orders data to compare our historical data too. Just like at the beginning of the lab, left click on the plus (+) button in the upper left corner ofthe screen and select Add Sources. Add the New Orders source from the modal.

   ![image156](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image156.png)

2. Next, add a Stage node to the data source so we can keep the consistency of our staging layer.

   ![image157](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image157.png)

3. In order to look for anomalies in our forecast data, we need to train our anomaly detection model on historical data. Open the STG_ORDER_MASTER_ITEMS node, which contains all of our historical orders data.

   ![image138](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image138.png)

4. Create a new column called LABEL and add a data type of BOOLEAN. This will be our training set where we tell the anomaly detection model what should be considered an anomaly.

   ![image139](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image139.png)

5. In the transformation field of the LABEL column, add the following transformation:

      ```SQL
      case when sum("FCT_ORDER_MASTER"."ORDER_AMOUNT") > 5800000 then TRUE else false END
      ```

   ![image140](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image140.png)

6. Create and Run the node to populate it with the new column and data. Fetch the data to ensure the logic in your transformation is working.

   ![image141](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image141.png)

7. Back in the Browser, right click on the STG_ORDER_MASTER_ITEMS node and select Add Node -> CortexML -> Anomaly Detection.

    ![image142](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image142.png)

8. Open the Anomaly Model Input settings in the configuration on the right side of the screen. We'll need to configure these parameters to build our anomaly detection.

    ![image143](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image143.png)

9. First, give the model instance a name, in this case we'll call this **AD_MODEL**.

    ![image144](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image144.png)

10. Since we have multiple menu items we are looking at anomaly detection for, toggle the Multi-Series toggle to on.

    ![image145](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image145.png)

11. For the Series Column, select the ITEM_NAME column.

    ![image146](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image146.png)

12. For the Timestamp Column select the ORDER_TS column.

    ![image147](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image147.png)

13. For the Target Column select the ORDER_AMOUNT column.

    ![image148](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image148.png)

14. Finally, toggle the Supervised Data toggle to on, becasue we will be training our model on the data we labled in the Stage node we were previously working in.

    ![image149](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image149.png)

15. For the Labeled Column, select the LABEL column.

    ![image150](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image150.png)

16. Now we need to add in the dataset that we want to evaluate for anomallies. Select the Multi Source toggle to turn it on so we can bring in our forecast data to evaluate for anomalies.

    ![image151](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image151.png)

17. In the Multi Source pane next to the mapping grid, select the plus + button to add the forecast data to our model.

    ![image152](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image152.png)

18. In the node navigation on the left side of your screen, drag and drop the STG_NEW_ORDERS node into the table column mapping.

    ![image153](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image153.png)

19. Coalesce will map the existing column to the historical dataset. Navigate to the join tab so you can create the dependency for the new data source. You'll see that the SQL editor is blank.

    ![image154](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image154.png)

20. Select the Generate Join dropdown to establish the dependency by selecting Copy to Editor.

    ![image155](operationalize-ml-in-your-data-pipelines-with-snowflake-cortex/assets/image155.png)

21. Now, Create and Run the node. It may take a minute or two to run depending on your warehouse size.

## Conclusion and Next Steps

Congratulations on completing your lab. You've mastered the basics of Coalesce and are now equipped to venture into our more advanced features. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

### What We’ve Covered

- How to navigate the Coalesce interface
- Configure storage locations and mappings
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to join tables
- How to apply transformations to individual and multiple columns at once
- How to configure and understand the ML Forecast node

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our [documentation](https://docs.coalesce.io/docs) and [resources](https://coalesce.io/resources/). For more technical guides on advanced Coalesce and Snowflake features, check out Snowflake Quickstart guides covering [Dynamic Tables](https://quickstarts.snowflake.com/guide/building_dynamic_tables_in_snowflake_with_coalesce/index.html?index=..%2F..index#1) and [Cortex ML functions](https://quickstarts.snowflake.com/guide/ml_forecasting_ad/index.html?index=..%2F..index#0).

### Additional Coalesce Resources

- [Getting Started](https://coalesce.io/get-started/)
- [Documentation](https://docs.coalesce.io/docs) & [Quickstart Guide](https://docs.coalesce.io/docs/quick-start)
- [Video Tutorials](https://fast.wistia.com/embed/channel/foemj32jtv)
- [Help Center](mailto:support@coalesce.io)

Reach out to our sales team at [coalesce.io](https://coalesce.io/contact-us/) or by emailing [sales@coalesce.io](mailto:sales@coalesce.io) to learn more.

[trial account]: https://signup.snowflake.com/
[Google Chrome browser]: https://www.google.com/chrome/
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake Marketplace]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce

---

## Snowflake Quickstart

## Before You Start

- Sign up for a Coalesce [account](https://coalesce.io/start-free/).
- Make sure you're using Google's Chrome browser, as other browsers are not officially supported.
- Have your Snowflake login available. Don’t have a Snowflake account yet or not ready to connect your company’s Snowflake to Coalesce? Sign up for a [free trial](https://signup.snowflake.com/) and use Snowflake’s provided sample data.

In this example, you’ll learn to connect your database, set up a staging node with a data transformation, and create a dimension and fact table

## Video Overview

The following video provides a brief overview of what this guide covers.

<YouTube id="MVQRH2qsizg?si=_pL_enXR9t2hINOm" />

## Interface

When you first sign in on a new account, you'll be presented with the **Projects Dashboard**, that includes a default [**Project**][] and **Development Workspace**.  Click the **Launch** button to open the workspace to continue. You can ignore the version control warning  since it won't be used in this guide.

In the Development Workspace is the [**Build** interface][] and  [**Deploy** interface][].

The Build interface is where you'll create nodes, build node graphs, and transform your data. The [**Deploy** interface][] is used to push your pipeline to other environments such as QA and production.

:::tip[The Problem Scanner]

For new accounts, the [Problem Scanner][] will show a few action items. You can disregard them for this guide..

:::

## Connect to Snowflake

1. Click on **Build Settings**, which is represented by a cogwheel icon.

2. Go to **Development Workspaces** and edit your current **Workspace** by clicking the pencil icon to the right of it.

3. On the **Settings** page, enter your Snowflake Account.

   1. Obtain your Snowflake URL, by opening the [account selector][] in Snowflake.

    

4. Go to **User Credentials** and fill out the form with your Snowflake login.

5. Click **Test Connection** to ensure your credentials work as expected.

6. Click **Save**.

You've now connected Coalesce to your Snowflake instance.

## Configure Storage Locations and Workspace

A storage location is a logical name you provide to represent a database and schema (in Snowflake) and you will need them to make use of the **Workspace** you configured in [Connect to Snowflake][].  Depending on your setup you may need to create a storage location in Snowflake.

1. Go to **Build Settings** >**Storage Mappings**. You'll have a **SAMPLE** and **WORK** locations. The Snowflake Sample Data will be mapped to SAMPLE and the WORK is where you'll write new tables.
2. Set the **SAMPLE** database to use the Snowflake sample data, and use the schema **TPCH_SF1**.
3. Set the **WORK** to a second storage location and schema in that storage location.

:::info[Snowflake Storage Locations]
If you don't have two storage locations, you'll need to create them. You can do this from Snowsight > Databases or using [SnowQL](https://docs.snowflake.com/en/sql-reference/sql/create-database). You can also get the sample data from [Snowflake](https://docs.snowflake.com/en/user-guide/sample-data-using).

:::

## The Node Graph

Now it's time to add **Sources** to the graph. The [Node Graph][] is where you'll configure **Nodes** that will transform your data. Below is an example of a graph with several nodes. Each box is considered a Node.

### Add Graph Sources

1. Expand **Nodes** from the sidebar.
2. Click on the **+** next to Search > **Add Sources**.
3. Choose your source tables and click **Add Sources**.
4. You'll now see your graph populated with some **Source Nodes**.

## Make a Stage Node to Transform Your Data

Now that you have **Source Nodes** on your graph, it's time to add a **Stage Node**.  [Stage Nodes][] are intermediate nodes in the graph where you can preview your changes.

1. Add one or more **Stage Nodes** by right clicking your `Nation` **Source Node**>**Add Node**>**Stage Node**.  You can select multiple **Source Nodes** by Shift+clicking them and then add multiple **Stage Nodes** simultaneously.
2. Double click on the **Stage Node** or right-click and select **Edit** to open up the [**Node Editor**][] for the Nation Stage Node you created.
3. Open **Node Properties** on the right and ensure the **Storage Location** is set to the WORK you configured [earlier][]. You'll be writing to this location in Snowflake.
4. Click **Create** to create a table in Snowflake. You'll see the status in the lower window.
5. Click **Run** to populate the table. You'll see the status in the lower window. You haven't transformed the data yet.
6. Edit the **Transform** field in the **Mapping** grid by double clicking in the transform field of the `N_NAME` column. Try a simple transform like `LOWER()` and the name of your column, or you can use the syntax `LOWER("NATION"."N_NAME")` to edit the Snowflake sample data.
7. Click **Run** again to transform the data. You'll see that the nation names are in lowercase.

:::info[On Transforms]

Any [Snowflake SQL transform][] can be used to transform your data.

:::

Congratulations, you've connected your database and applied a basic transformation to your data. Feel free to continue experimenting with some of the other node types below.

## Create a Dimension Table

 In this guide we will be making a Type 2 Dimension to track historical data of a column. A [slowly changing dimension(Type 2)][] is a industry standard for tracking historical data by creating multiple records for a given natural key.

By default, Coalesce creates a Type 1 Dimension, you'll add a Type 2 Dimension.

1. Right click on the `CUSTOMER` node and create a new **Stage Node**.
2. In the Node editor for `STG_CUSTOMER`, click **Create** and then **Run**.
3. Return to the main graph and create a Dimension node from the `STG_CUSTOMER` node.  

    

4. Go into the new `DIM_CUSTOMER` Node editor.
5. Open up **Options** and select `C_CUSTKEY` as a business key by selecting it and clicking the right arrow.
6. In the same **Options** area, go to **Change Tracking** and select `C_ADDRESS` and `C_PHONE`, then click the right arrow to add them.
7. Now **Create** and **Run** the `DIM_CUSTOMER` node. Congratulations. You've created a Type 2 Dimension table.

:::info[Type 1 vs Type 2]

In the **Dimension Node**, if no **Change Tracking** columns are selected, the node will act as a Type 1 Dimension. If **Change Tracking** columns are selected, it will act as a Type 2.

:::

## Create a Fact Table

Now let's create a [fact table][]. You're joining the Orders (`STG_ORDERS`) with the Dimension table on the `DIM_CUSTOMER_KEY` and `ORDERS` Customer key. Then you'll create a Fact table of the customers and orders. Because you're specifying the `O_ORDERKEY`, it will merge instead of inserting the data.

1. Create a new **Stage Node** from the `ORDERS` **Source Node**.
2. Open the new `STG_ORDERS` node and delete all the columns except for `O_ORDERKEY`, `O_CUSTKEY`, and `O_TOTALPRICE`. You can drag and drop rows to group the items for deletion.
3. Select the `DIM_CUSTOMER` node on the left side, then select `DIM_CUSTOMER_KEY` and drag it into your `STG_ORDERS` mapping grid.

    

4. Go to Join, next to Mapping in the `STG_ORDERS` Node Editor.
5. Delete any existing text.
6. Click **Generate Join** and then **Copy to Editor**.
7. Replace the `/_COLUMN_/` text with `O_CUSTKEY`.  It should look similar to the following:

    ```sql
    FROM {{ ref('SAMPLE', 'ORDERS') }} "ORDERS"
      LEFT JOIN {{ ref('WORK', 'DIM_CUSTOMER') }} "DIM_CUSTOMER"
      ON "ORDERS"."O_CUSTKEY" = "DIM_CUSTOMER"."C_CUSTKEY"
    ```

8. **Create** and **Run** the `STG_ORDERS` node.
9. Create a **Fact Node** from `STG_ORDERS`.
10. Open the new `FCT_ORDERS` Node Editor.
11. Open **Options** > Business Key > add the `O_ORDERKEY`
12. **Create** and **Run** the `FCT_ORDERS` node.

Congratulations. You have now made a fact table. You can also run this query in Snowflake. Make sure to adjust your schema(`MY_SCHEMA`) and database(`MY_DB`)  to your environment.

```sql
select DIM.C_NAME CUSTOMER_NAME,
sum(FCT.O_TOTALPRICE) TOTAL_PRICE
from "MY_DB"."MY_SCHEMA"."FCT_ORDERS" FCT
inner join "MY_DB"."MY_SCHEMA"."DIM_CUSTOMER" DIM
on FCT.DIM_CUSTOMER_KEY = DIM.DIM_CUSTOMER_KEY
group by DIM.C_NAME;
```

[**Build** interface]: /docs/build-your-pipeline/the-build-interface/
[**Deploy** interface]: /docs/deploy-and-refresh/overview-of-the-deploy-interface
[Problem Scanner]: /docs/build-your-pipeline/the-build-interface/problem-scanner
[account selector]: https://docs.snowflake.com/en/user-guide/admin-account-identifier#label-account-name
[Connect to Snowflake]: /docs/guides/snowflake-quickstart#connect-to-snowflake
[Node Graph]: /docs/build-your-pipeline/the-build-interface/
[Stage Nodes]: /docs/get-started/coalesce-fundamentals/nodes
[**Node Editor**]: /docs/build-your-pipeline/the-build-interface/node-editor
[earlier]: /docs/guides/snowflake-quickstart#configure-storage-locations-and-workspace
[Snowflake SQL transform]: https://docs.snowflake.com/en/sql-reference-functions.html
[slowly changing dimension(Type 2)]: https://www.ibm.com/docs/en/iiw/8.10.1?topic=techniques-slowly-changing-facts
[fact table]: /docs/get-started/coalesce-fundamentals/nodes
[**Project**]: /docs/setup-your-project/create-your-project

---

## Using Stored Procedures in Coalesce

## Overview

A Stored Procedure is a reusable block of SQL code that can be stored and executed. Stored procedures are used to perform specific tasks as part of a data pipeline, and they can be created, deployed, and called through a custom node type.

This guide details the steps to accomplish this in Coalesce, as well as provides the code for the custom Node Type used in this technique.

<Wistia id="uzzioe8s2g" />

## Creating a Stored Procedure

This custom Node Type takes a user-defined piece of SQL and executes it during Coalesce's Create and  Deploy process. The SQL provided will be the create SQL for the Stored Procedure.

1. Create a custom Node Type by going to **Build Settings > Node Types** and clicking **Create Node Type**. Update the Node name to something that will help you remember the functionality.
2. Add in the code into the **Node Definition** and the **Create Template**. The Run Template is blank. Using the `deployStrategy: advanced` will give you a Current State and Desired State. You can ignore these.

    <Tabs>
      <TabItem value="sp-node-definition" label="Node Definition" default>

      ```yaml
      capitalized: CREATE SQL
      short: SQL
      plural: SQLs
      tagColor: '#2EB67D'

      deployStrategy: advanced

      config:
      - groupName: 'Options'
        items:
        - type: textBox
          displayName: SQL
          attributeName: SQL1
          syntax: sql 
          isRequired: true

      systemColumns:

      - displayName: SQL_SEQ
        transform: ''
        dataType: NUMBER IDENTITY
        placement: beginning
        attributeName: isSurrogateKey

      - displayName: SYSTEM_START_DATE
        transform: CAST(CURRENT_TIMESTAMP AS TIMESTAMP)
        dataType: TIMESTAMP
        placement: end
        attributeName: isSystemStartDate
      ```

      </TabItem>
      <TabItem value="sp-create-template" label="Create Template">

      ```sql
      {% if desiredState == currentState %} {{ stage('Nothing to do.') }} SELECT 1 = 0{% elif desiredState != undefined %} {{
      stage('Create SQL1') }} {{ desiredState.config.SQL1 }} {% elif currentState != undefined and desiredState == undefined
      %} {# Stored Procedure Name #} {% set targetObjectDatabase = ref_no_link(currentState.node.location.name,
      currentState.node.name).split('.')[0] %} {% set targetObjectSchema = ref_no_link(currentState.node.location.name,
      currentState.node.name).split('.')[1] %} {% set fullyQualifiedTargetObjectName =
      ref_no_link(currentState.node.location.name, currentState.node.name) %} {{ stage('Drop SQL1') }} DROP PROCEDURE IF
      EXISTS {{ fullyQualifiedTargetObjectName }} (VARCHAR){% else %} {{ stage('Unknown State') }} SELECT 1 = 1{% endif %}
      ```

      </TabItem>
    </Tabs>

    <figure>
      
      <figcaption>Add the code to the Node Definition and Create Template</figcaption>
    </figure>

3. You need to update the Workspace macros. Go to **Build Settings > Workspace Macros**. Add the following code.

    ```sql
    {#-- The below block of code initialises variables in case of node typess using advance deployment strategy #}

    {% if desiredState %}
        {% set columns = desiredState.columns %}
        {% set storageLocations = desiredState.storageLocations %}
        {% set config = desiredState.config %}
        {% set sources = desiredState.sources %}
        {% set node = desiredState.node %}
        {% set parameters = desiredState.parameters %}
    {% endif %}

    {#-- The below block of code initialises variables in case of node typess using advance deployment strategy #}

    {% if desiredState %}
        {% set columns = desiredState.columns %}
        {% set storageLocations = desiredState.storageLocations %}
        {% set config = desiredState.config %}
        {% set sources = desiredState.sources %}
        {% set node = desiredState.node %}
        {% set parameters = desiredState.parameters %}
        {% set this = desiredState.this %}
    {% endif %}

    {#-- Macro to return node name #}
    {#-- Input parameters   - None #}
    {#-- Return            - Order by clause #}

    {%- macro node_name() -%}
    {{ ref_no_link(node.location.name, node.name) }} 
    {%- endmacro -%}
    ```

    <figure>
      
      <figcaption>Add to Workspace Macros</figcaption>
    </figure>

4. In the Mapping Grid, click the plus sign ‘+’ to **Create New Node > Stored Procedure Node** you created.

    <figure>
      
      <figcaption>Add the Store Procedure Node. In this example it's `SQL_NODE`.</figcaption>
    </figure>

5. In **Node Properties**, update the **Storage Location**. The Storage Location is where the Stored Procedure will be created in Snowflake.
6. In **Options > SQL**, you'll add the Data Definition Language (DDL) for the Stored Procedure. In this example, we are returning a message string.

    ```sql
    CREATE OR REPLACE PROCEDURE {{ node_name() }}("MESSAGE" STRING)
    RETURNS STRING
    LANGUAGE SQL
    EXECUTE AS OWNER
    AS '
    begin
      return message;
    end;
    ';

    -- Can also use {{this}}

    CREATE OR REPLACE PROCEDURE {{this}} ("MESSAGE" STRING)
    RETURNS STRING
    LANGUAGE SQL
    EXECUTE AS OWNER
    AS '
    begin
      return message;
    end;
    ';
    ```

7. Click **Create**, to create the Node, and check the results.

<figure>
  
  <figcaption>Update the Storage Location and SQL.</figcaption>
</figure>

:::info[Check Your Storage Location]

If you’re encountering errors when trying to create your Node, check the Storage Location for the stored procedure.

:::

## Calling the Stored Procedure

You can use a stored procedure in one of the following ways:

* As a piece of pre-SQL or post-SQL within a related Node.
  * `call my_sp('param_one', 'param_two', 'param_three');`
* As an independent Node. You would need to add a Run Template to the custom Node Type.

In this example, the Stored Procedure Node is called in the Pre-SQL.

```sql
call {{ref('WORK', 'SP_SIMPLE')}} ('hi')
```

<figure>
  
  <figcaption>Stored Procedure runs as part of the Pre-SQL</figcaption>
</figure>

## Deploy the Stored Procedure

1. Commit your Stored Procedure.
2. Choose the Environment for deployment.
3. Initiate the deployment from the chosen branch in Git.
4. The deployment process will automatically create the Stored Procedure in the selected Environment.

:::info[Stored Procedure Deployment]

Please be aware that the stored procedure will be included in every build, even when using advanced deployment options.

:::

---

## Trace Metrics and Dashboards in Catalog

<!-- vale Coalesce.FirstPerson = NO -->

## What You Will Accomplish

Your operations VP sees **Fare Revenue Per Mile** drop on the **NY Taxi Overview** dashboard and asks whether the number is trustworthy or the definition changed. You do not own the pipeline, but you need an answer before the leadership review: what the metric officially means, which dashboard shows it, and which warehouse columns actually drive the calculation.

This guide walks that investigation in Catalog using a realistic discovery path. You will use Catalog to connect the report layer to governed definitions and source tables so you can explain or audit the metric with evidence.

By the end of this walkthrough you should be able to tell your stakeholder:

- The official definition of **Fare Revenue Per Mile** and any caveats your data team documented.
- Which dashboards in your BI tools surface that metric and who owns them.
- Which warehouse table and columns feed those dashboards, so engineering or analytics can validate the logic or refresh timing.
- Whether Knowledge, **Mentioned in**, **Pinned Assets**, and lineage in Catalog agree, or where documentation is missing.

## Before You Begin

1. Make sure you have access to [Catalog][].
2. Confirm **Dashboards** and **Data** are populated. Empty lists mean an admin must finish warehouse and [Data visualization integrations][] setup first.

## Step 1: Confirm the Business Definition in Knowledge

Start with the metric name because executives usually cite the KPI label, not a table name.

1. In the left navigation, click **Knowledge**. Catalog opens the Knowledge hub at `/terms`.

   

2. Open **Metrics Glossary** from the Knowledge hub.

   

3. Click the **Subpages & Map** tab. The label can include a count, for example **24 Subpages & Map** or **0 Subpages & Map**. In this example, look at **Fare Revenue Per Mile**.

4. On the **Read me** tab, review the steward-written description, **Technical Definition**, formula, and field notes for **Fare Revenue Per Mile**.

   <figure>
   
   <figcaption>Fare Revenue Per Mile on the Read me tab: Technical Definition with `SUM(fare_revenue) / SUM(trip_distance)` and **Mentioned in** NY Taxi Overview in Details</figcaption>
   </figure>

5. In the **Details** panel, review:

   - **Owners**: Who to ping if the definition looks stale.
   - **Tags**: How stewards classify the metric.
   - **Mentioned in**: Other assets that reference the metric, for example **NY Taxi Overview**. Use these for cross-checks; they do not replace the formula on **Read me**. Stewards link the canonical dashboard from the metric side when the KPI is documented in Knowledge.

**Checkpoint:** You have a written definition and at least one linked dashboard. If **Read me** is empty and nothing is linked, record that gap. Your answer to the VP can be "definition not documented in Catalog yet" rather than guessing.

## Step 2: Open the Dashboard That Reports the Metric

1. Click **Dashboards**.

   

2. You can either select the integration and folder for your metric or find it in the search bar.
3. Open the dashboard your stakeholder named. In this example it is **NY Taxi Overview**. Use the list search box if the folder tree is large.

   

4. On the dashboard **Read me** tab, confirm the narrative matches what you read in Knowledge. For **NY Taxi Overview**, look for these sections:

   - **Audience:** **Target Audience** lists who should use the dashboard, for example analysts, operations managers, and executive leadership, and how often they are expected to open it.
   - **Metric and tiles:** **Important Notes / Business Rules** restates the **Fare Revenue Per Mile** formula (`SUM(fare_revenue) / SUM(trip_distance)`), and **Dashboard Content** > **Reports / Visualizations** lists a **Fare Revenue per Mile** tile with its objective. **Details** > **Mentioned in** links back to the Knowledge metric for a cross-check.

5. Use **Mentioned in** to jump back to **Fare Revenue Per Mile** or related Knowledge pages if stewards linked them from the dashboard side.

6. Scroll to **Pinned Assets** and **Data Refresh** on **Read me** when present. **Data Refresh** describes how often trip data is updated; if the schedule is not documented, note that gap before you answer the VP about whether stale data could explain the drop. In this example, stewards pinned **FCT_YELLOW_CAB_TRIPS** and **Fare Revenue Per Mile**, which gives you a shortcut to the warehouse table before you open lineage.

   

7. Check **Popularity** and **Owners** in **Details**. High popularity supports that this is the "official" view teams actually use, not an abandoned draft.

**Checkpoint:** You can name the dashboard, its owner, and whether its documentation aligns with the Knowledge definition. Misalignment here is a common root cause of "the number changed but the business meaning did not."

## Step 3: Trace Lineage to Warehouse Tables

Connect the visible chart to physical data. Catalog shows upstream tables on the dashboard **Lineage** tab when BI extraction has populated dependency links.

1. On **NY Taxi Overview**, open the **Lineage** tab.

2. Expand **Parents** first, then open **Sources** and **Children** when you need the full upstream or downstream chain. Each section answers a different question about how this dashboard connects to warehouse data:

   - **Sources** travels to the first ancestral assets in the chain. You see the earliest upstream tables Catalog found, often grouped by schema layers such as **NY_TAXI** > **GOLD**, **SILVER**, and **SYNQ**. Use **Sources** when you need the full upstream path beyond the tables Sigma queries directly.
   - **Parents** lists assets that directly feed the dashboard. For **NY Taxi Overview**, expect **FCT_YELLOW_CAB_TRIPS**, **DIM_VENDOR**, and **DIM_PAYMENT_TYPE** under **NY_TAXI** > **GOLD**. The fact table holds trip fare and distance fields that implement **Fare Revenue Per Mile**; the dimension tables add vendor and payment context to other tiles. Check **FAILED** or **PASSED** badges here when a stakeholder asks whether the dashboard number is trustworthy.
   - **Children** lists assets built from this dashboard. Dashboards are usually leaf assets in Catalog, so **NY Taxi Overview** often shows no children. An empty **Children** list confirms nothing else in Catalog depends on this dashboard.

   

3. Optional: click **Open Dashboard Lineage** or **Open Field Lineage** when you need the full graph view or column-level upstream paths.

**Checkpoint:** You can name at least one warehouse table on the lineage path from dashboard to source when lineage is populated. In this demo, that table is **FCT_YELLOW_CAB_TRIPS** at **Data** > **Snowflake Demo** > **NY_TAXI** > **GOLD**, reached from dashboard **Lineage** > **Parents**. If lineage is empty, your checkpoint is a recorded gap plus the dashboard and Knowledge links you already collected.

## Step 4: Find the Column That Drives the Metric

Tables can expose dozens of fields. Narrow to the columns that implement the KPI.

1. From dashboard **Lineage** > **Parents**, click **FCT_YELLOW_CAB_TRIPS** at **NY_TAXI > GOLD**, then open the **Columns** tab.

   

2. In **Search Column name**, filter for terms from the Knowledge formula. Search one term at a time. For example, type `FARE` to locate **FARE_AMOUNT**, then `TRIP` for **TRIP_DISTANCE**. Both map to the metric definition `SUM(fare_revenue) / SUM(trip_distance)`.

   

3. Confirm the columns you found feed the dashboard your stakeholder asked about. Scroll to **Used in Dashboards** at the bottom of the **Columns** tab. In this example, **NY Taxi Overview** is listed there, which ties **FARE_AMOUNT** and **TRIP_DISTANCE** on **FCT_YELLOW_CAB_TRIPS** to the view the VP is questioning. That is the handoff data engineering needs: concrete `table.column` names on the table the dashboard reads.

4. For column-level lineage, click **Open Field Lineage** on the **Lineage** tab.

5. Open the table **Read me** tab for table-level context and any **Pinned Assets** that point back to **Fare Revenue Per Mile** or sibling metrics.

**Checkpoint:** You can state concrete columns to validate in SQL or with the pipeline owner, for example `FCT_YELLOW_CAB_TRIPS.FARE_AMOUNT` and `FCT_YELLOW_CAB_TRIPS.TRIP_DISTANCE`. That is the artifact data engineering needs, not only "the taxi dashboard looks wrong."

## Step 5: Synthesize Your Answer

Bring the checkpoints together into a short narrative you can send or present:

| Question from your stakeholder | Where you found it in Catalog |
| :----------------------------- | :---------------------------- |
| What is Fare Revenue Per Mile supposed to mean? | Knowledge metric page > **Read me** |
| Which dashboard are they looking at? | **Dashboards** detail page or Knowledge **Mentioned in** |
| Who maintains that dashboard? | Dashboard **Details** > **Owners** |
| Which table holds the source data? | Dashboard **Lineage** > **Parents** |
| Which columns should we audit? | Table **Columns**, **Open Field Lineage**, or **Used in Dashboards** |
| Is documentation complete? | Empty **Read me**, missing links, or broken lineage |

Example reply you can adapt:

> **Fare Revenue Per Mile** is defined in Knowledge as `SUM(fare_revenue) / SUM(trip_distance)` with field notes on what counts in fare revenue. The VP is looking at **NY Taxi Overview**, owned by the names listed under **Details** > **Owners**. The dashboard reads **FCT_YELLOW_CAB_TRIPS**; audit **FARE_AMOUNT** and **TRIP_DISTANCE** there. Knowledge, dashboard **Read me**, **Pinned Assets**, and **Parents** lineage agree in our workspace. **Data Refresh** is documented or missing, so note whether stale data could explain the drop before you commit to an answer.

## Other Ways to Start the Same Investigation

This example began with **Knowledge** because the VP cited the Fare Revenue Per Mile metric name. The same outcome is reachable from other entry points:

| If you already know… | Start here |
| :------------------- | :--------- |
| Dashboard title only | **Dashboards** |
| Table or column name | **Data** browse or [Advanced Search in this guide][advanced-search-in-guide] |
| Only a vague question | [AI Assistant section][ai-assistant-section] |

### Search When You Have a Fragment of a Name

Use these paths when you know part of an asset name but not the full Catalog URL. Each subsection picks up the main investigation at the step shown in the table in this section.

#### From Home

1. Open **Home** in the left navigation, or go to `/search`.
2. Type a metric, dashboard, or table name in the search field. The placeholder reads "Search anything in the Coalesce Catalog".
3. Click **Search** to run the query. Matches stay on **Home**; open an asset from the list to continue the investigation at Step 2 or Step 3 in the table above. Do not use **Search** when you need the filtered `/results` page; use [Advanced Search in this guide][advanced-search-in-guide] instead.

   

#### Advanced Search

Use this when you need filters for asset type, owner, certification, tags, and related options rather than only top matches on **Home**. See [Advanced Search][] for how each filter behaves.

1. On **Home**, type a metric, dashboard, or table name in the search field, same as [From Home][from-home] step 2.
2. Press **Enter** (not **Search**) to open the **Search** results page at `/results` with a **Filters** panel on the right.

   

3. Adjust **Filters** for **Dashboards**, **Knowledge**, **Tables**, **is Certified**, **Owner**, **Tags**, and other options your workspace exposes.
4. Open the best match and continue from Step 2 or Step 3 in the table in this section.

#### Browse Data

Use this when you know the warehouse object path but not the exact Catalog URL.

1. In the left navigation, click **Data**.
2. Expand the warehouse, database, and schema, for example **Snowflake Demo** > **NY_TAXI** > **GOLD**, then open the table your team uses for the metric.
3. Continue at Step 4, **Find the Column That Drives the Metric**.

When you are already on a table or dashboard page, the top search field is scoped to data assets. Its placeholder shows **⌘ + P** on Mac or **Ctrl + P** on Windows for that in-context search.

For a full map of discovery methods, see [Data discovery][].

## Optional: Use AI Assistant

Use this path when you have only a vague question and do not yet know the metric, dashboard, or table name. You can open **AI Assistant** from the left navigation, or click **Ask** on **Home** when AI is enabled. After you get cited assets, open each link and continue the investigation at Step 3 or Step 4 in the table above.

1. In the left navigation, click **AI Assistant**, or open [AI Assistant in Catalog][].
2. Ask: `What is our Fare Revenue Per Mile definition?` and `Which tables feed the NY Taxi Overview dashboard?`
3. Open each cited link and confirm **Read me**, **Lineage**, and **Pinned Assets** match what you would expect from Steps 1 to 4.

   

## What's Next?

- Document or refresh **Fare Revenue Per Mile** and related metrics in [Knowledge][] with [Pinned assets][] to the canonical table and dashboard.
- Deepen lineage troubleshooting in [Lineage in Catalog][].
- Browse BI assets in [Dashboards in Catalog][].
- Onboard teammates with the [5-Minute Quick Start Guide][].

[Catalog]: https://app.castordoc.com/search
[Data visualization integrations]: /docs/catalog/integrations/data-viz
[AI Assistant in Catalog]: https://app.castordoc.com/ai/search
[Knowledge]: /docs/catalog/assets/knowledge
[Lineage in Catalog]: /docs/catalog/use-cases/discover/lineage
[Pinned assets]: /docs/catalog/document-your-data/pinned-assets
[Advanced Search]: /docs/catalog/use-cases/discover/advanced-search
[Data discovery]: /docs/catalog/use-cases/discover/
[Dashboards in Catalog]: /docs/catalog/assets/dashboards
[5-Minute Quick Start Guide]: /docs/catalog/use-cases/five-minute-quick-start-guide
[ai-assistant-section]: #optional-use-ai-assistant
[advanced-search-in-guide]: #advanced-search
[from-home]: #from-home

---

## Using Coalesce Marketplace to Build First Class Data Pipelines

## Overview

This Hands-On Lab exercise is designed to help you master building data pipelines using Coalesce Marketplace. In this lab, you’ll explore the Coalesce interface, learn how to easily transform and model your data with a variety of packages from Coalesce Marketplace, understand how to build reporting pipelines, and play with real-time functionality.

### **What You’ll Need**

- A Snowflake account (either a [trial account][] or access to an existing account)
- A Coalesce account (either a free trial started from our [Snowflake Marketplace listing][], or access to an existing account)
- Basic knowledge of SQL, database concepts, and objects
- The [Google Chrome browser][]

### **What You’ll Build**

- A Directed Acyclic Graph (DAG) representing a basic star schema in Snowflake

### **What You’ll Learn**

- How to navigate the Coalesce interface
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to join tables
- How to apply transformations to individual and multiple columns at once
- How to build out Dimension and Fact nodes
- How to make changes to your data and propagate changes across pipelines
- How to configure and build an ML Forecast node

By completing the steps we’ve outlined in this guide, you’ll have mastered the basics of Coalesce and can venture into our more advanced features.

---

## About Coalesce

Coalesce is the first cloud-native, visual data transformation platform built for Snowflake. Coalesce enables data teams to develop and manage data pipelines in a sustainable way at enterprise scale and collaboratively transform data without the traditional headaches of manual, code-only approaches.

### What Can You Do With Coalesce?

With Coalesce, you can:

- Develop data pipelines and transform data as efficiently as possible by coding as you like and automating the rest, with the help of an easy-to-learn visual interface
- Work more productively with customizable templates for frequently used transformations, auto-generated and standardized SQL, and full support for Snowflake functionality
- Analyze the impact of changes to pipelines with built-in data lineage down to the column level
- Build the foundation for predictable DataOps through automated CI/CD workflows and full Git integration
- Ensure consistent data standards and governance across pipelines, with data never leaving your Snowflake instance

### How Is Coalesce Different?

Coalesce’s unique architecture is built on the concept of column-aware metadata, meaning that the platform collects, manages, and uses column- and table-level information to help users design and deploy data warehouses more effectively. This architectural difference gives data teams the best that legacy ETL and code-first solutions have to offer in terms of flexibility, scalability, and efficiency.

Data teams can define data warehouses with column-level understanding, standardize transformations with data patterns (templates) and model data at the column level.

Coalesce also uses column metadata to track past, current, and desired deployment states of data warehouses over time. This provides unparalleled visibility and control of change management workflows, allowing data teams to build and review plans before deploying changes to data warehouses.

### **Organization**

A Coalesce organization is a single instance of the UI, set up specifically for a single prospect or customer. It is set up by a Coalesce administrator and is accessed via a username and password. By default, an organization will contain a single Project and a single user with administrative rights to create further users.

### **Projects**

Projects provide a way of completely segregating elements of a build, including the source and target locations of data, the individual pipelines and ultimately the Git repository where the code is committed. Therefore teams of users can work completely independently from other teams who are working in a different Coalesce Project.

Each Project requires access to a Git repository and Snowflake account to be fully functional. A Project will default to containing a single Workspace, but will ultimately contain several when code is branched.

### **Workspaces vs. Environments**

A Coalesce Workspace is an area where data pipelines are developed that point to a single Git branch and a development set of Snowflake schemas. One or more users can access a single Workspace. Typically there are several Workspaces within a Project, each with a specific purpose (such as building different features). Workspaces can be duplicated (branched) or merged together.

A Coalesce Environment is a target area where code and job definitions are deployed to. Examples of an environment would include QA, PreProd, and Production.

It isn’t possible to directly develop code in an Environment, only deploy to there from a particular Workspace (branch). Job definitions in environments can only be run via the CLI or API (not the UI). Environments are shared across an entire project, therefore the definitions are accessible from all workspaces. Environments should always point to different target schemas (and ideally different databases), than any Workspaces.

## Lab Use Case

As the lead Data & Analytics manager for Coalesce Restaurant Inc., you're responsible for building and managing data pipelines that deliver insights to the rest of the company. There are customer-related questions that the business needs to answer that will help with inventory planning and marketing. Included in this, is using Large Language Models that will allow management to determine customer sentiment.

In order to help your extended team answer these questions, you’ll need to build a customer data pipeline first.

---

## Before You Start

To complete this lab, please create free trial accounts with Snowflake and Coalesce by following the steps below. You have the option of setting up Git-based version control for your lab, but this is not required to perform the following exercises. Please note that none of your work will be committed to a repository unless you set Git up before developing.

We recommend using Google Chrome as your browser for the best experience.

| Note: Not following these steps will cause delays and reduce your time spent in the Coalesce environment. |
| :---- |

### Step 1: Create a Snowflake Trial Account

1. Fill out the [Snowflake trial account form][]. Use an email address that is not associated with an existing Snowflake account.

2. When signing up for your Snowflake account, select the region that is physically closest to you and choose Enterprise as your Snowflake edition. Please note that the Snowflake edition, cloud provider, and region used when following this guide do not matter.

    ![image1](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image1.png)

3. After registering, you will receive an email from Snowflake with an activation link and URL for accessing your trial account. Finish setting up your account following the instructions in the email.

### Step 2: Create a Coalesce Trial Account from Snowflake Marketplace

1. Sign in to Snowsight. Make sure you are using the ACCOUNTADMIN role (or a role with privileges to create a database, warehouse, user, and role) and that your Snowflake email address is verified.
2. In the navigation menu, select **Marketplace > Snowflake Marketplace** and search for Coalesce.

      

3. Open the Coalesce listing and select Start free trial in the upper right. To see what the trial includes, review the Your free trial tab.
4. In the Connect to Coalesce dialog, review the information to be shared and the Snowflake objects that will be created, then select Connect to Coalesce.
5. You will be redirected to Coalesce to finish activating your trial account. Fill in your information to complete activation.

Congratulations. You’ve successfully created your Coalesce trial account.

### Step 3: Set Up The Dataset

1. Within Worksheets, click the "+" button in the top-right corner of Snowsight and choose "SQL Worksheet.”

    ![image8](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image8.png)

2. Navigate to the [Coalesce Marketplace HOL Data Setup File][] that is hosted on GitHub.

3. Within GitHub, navigate to the right side and click "Copy raw contents". This will copy all of the required SQL into your clipboard.

    ![image9](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image9.png)

4. Paste the setup SQL from GitHub into your Snowflake Worksheet. Then click inside the worksheet and select All (*CMD + A for Mac or CTRL + A for Windows*) and Click "► Run".

5. After clicking "► Run" you will see queries begin to execute. These queries will run one after another within the entire worksheet taking around 60 seconds.

Once you’ve activated your Coalesce trial account and logged in, you will land in your Projects dashboard. Projects are a useful way of organizing your development work by a specific purpose or team goal, similar to how folders help you organize documents in Google Drive.

Your trial account includes a default Project to help you get started. Click on the Launch button next to your Development Workspace to get started.
    ![image10](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image10.png)

## Installing Packages From Coalesce Marketplace

You will need to add packages from Coalesce Marketplace into your workspace in order to complete this lab.

1. Within your workspace, navigate to the build settings in the lower left hand corner of the left sidebar

    ![image12](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image12.png)

2. Select Packages from the Build Settings options

    ![image13](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image13.png)

3. Select Browse to Launch the Coalesce Marketplace

    ![image14](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image14.png)

4. You'll see a variety of packages available to install. For this lab, we'll be installing each of the packages listed here:

    - External Data Package
    - Dynamic Tables
    - Cortex
    - Incremental Loading

    The process for installing each of these packages is the same, so once you do it once, you'll be able to follow the same flow for each consecutive package. Let's start with the External Data Package.

5. Select the **Find out more** button on the External Data Package. This will open the details of the package

    ![image15](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image15.png)

6. Copy the package ID from the External Data Package page

   ![image16](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image16.png)

7. Back in Coalesce, select the Install button:

   ![image17](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image17.png)

8. Paste the Package ID into the corresponding input box:

   ![image18](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image18.png)

9. Give the package an Alias, which is the name of the package that will appear within the Build Interface of Coalesce. We'll be using the following aliases for each package in this guide:

    - External Data Package: EDP
    - Dynamic Tables: Dynamic-Tables
    - Cortex: Cortex
    - Incremental Loading: Incremental

    Finish by clicking Install.

    ![image19](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image19.png)

10. Continue to follow this process until all four packages have been installed.

## Configure Storage Locations and Mappings

Your Coalesce trial account will come with two Storage Locations out of the box. We need to add one more to accomodate for some of the work we will be doing. In the setup of this guide, you will have run a SQL script in Snowflake that, among other things, created a stage and external volume. These two items will need to be mapped to a storage location in Coalesce so our packages can work with these objects.

1. To add storage locations, navigate to the left side of your screen and click on the Build settings cog.

    ![image27](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image27.png)

2. Click the “Add New Storage Locations” button and name it STORE representing the data coming from your restaurant.

    ![image28](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image28.png)

3. Now map your storage location to their logical destinations in Snowflake. In the upper left corner next to your Workspace name, click on the pencil icon to open your Workspace settings. Click on Storage Mappings and map your STORE location to the **MARKETPLACE_HOL** database and the **STORE** schema. Select Save to ensure Coalesce stores your mappings.

    ![image29](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image29.png)

4. Go back to the build settings and click on Node Types and then click the toggle button next to View to enable View node types. Now you’re ready to start building your pipeline.

    ![image30](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image30.png)

## Adding Data Sources

1. In the Browser within the Build Interface, select the + button in the upper left corner of the screen

    ![image31](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image31.png)

2. Select Add Sources. This will open the data sources modal which will display all of the obects available within each storage location you have configured.

    ![image32](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image32.png)

3. Within the STORE Storage Location that you configured in the previous step, you will see four tables available. Select all four of the tables and select **Add 4 Sources**

    ![image33](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image33.png)

4. There is an additional data source that is a JSON file that exists in an AWS S3 Bucket. To obtain this data source, we'll use our first Coalesce Marketplace Package. Select the + button again in the upper left corner and navigate to Create New Node -> EDP -> Copy Into.

    ![image34](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image34.png)

5. This will add a Copy Into node into your graph and immediately place you in the node. Rename the node to **CUSTOMER_REVIEWS**.

    ![image35](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image35.png)

6. On the right side of the node, open the Source Data dropdown within the Config Settings of the node.

    ![image36](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image36.png)

7. Toggle the Internal or External Stage toggle to on.

    ![image37](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image37.png)

8. Now you'll need to confiugre each of the parameters listed. For the Coalesce Storage Location of Stage, you will list the **STORE** storage location, as this is the database and schema that each object from the setup script was created within.

    ![image38](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image38.png)

9. For the Stage Name, you will use the name of the stage created in the setup script: **store_data**

    ![image39](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image39.png)

10. For the Path or Subfolder, you will pass through a path called **json** which is the folder containing the json data we wish to load.

    ![image40](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image40.png)

11. For the File Name(s), include the file name, which is named **customer_reviews.json**.

    ```JS
    customer_reviews.json
    ```

    ![image41](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image41.png)

12. You do not need to include anything for the file pattern. Open the **File Format** dropdown next.

13. You will keep the File Format Definition as **File Format Name**. For the Coalesce Storage Location, you will again use the **STORE** storage location.

    ![image42](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image42.png)

14. For the File Format Name, use the file format created in the setup script called **JSONFORMAT**

    ![image43](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image43.png)

15. As already discussed, the File Type is JSON, so select that from the File Type dropdown.

    ![image44](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image44.png)

16. Once complete, Create and Run the node.

    ![image45](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image45.png)

## Creating Stage Nodes

Now that you have your data sources added into Coalesce, we can begin processing all of the data by building a preparation layer for the rest of our data pipeline. Specifically, we'll use some stage nodes to accomplish this for several of our tables.

1. Because Coalesce uses nodes which represent templates or standards of objects, you can easily apply the same standard over and over again. Using the shift key, select the **CUSTOMERS**, **MENU**, and **CUSTOMER_REVIEWS** nodes and right click on either of them and select Add Node -> Stage.

    ![image46](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image46.png)

2. Coalesce will automatically create stage nodes for each of the data sources. Double click on the **STG_CUSTOMER_REVIEWS** node to open it.

    ![image47](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image47.png)

3. You'll see that there is a VARIANT column containing all of the JSON data we loaded int the previous node. Coalesce provides a simple and powerful solution for parsing JSON and XML strings. Right click on the **SRC** column and select Derive Mappings -> From JSON.

    ![image48](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image48.png)

4. Coalesce will split out each of the key value pairs into their own columns with the associated data type. With the name columns available, we no onger need the original columns from the Copy Into node. Select the **SRC** column, hold down the shift key, and select the **SCAN_TIME** column. Right click on either of these columnes and select Delete Columns.

    ![image49](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image49.png)

5. You can apply data transformations at any point in Coalesce. Let's apply some column level transformations to these remaning columns. You may have noted that all of the columns names in other nodes are all upper case. To ensure consistency, let's do that to these columns as well. Again, using the shift key, select all of the columns, and right click on any column and select Bulk Edit.

    ![image50](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image50.png)

6. The bulk editor will open and all of the attributes available to be edited will be displayed. In this case, we'll choose **Column Name**.

    ![image51](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image51.png)

7. The SQL editor will be displayed, which allows you to write any SQL that will be applied to all of the columns. In this case, we'll use an identifier that allows Coalesce to resolve the SQL statement to each column name, while applying an UPPER function.

    ```SQL
    {{ column.name | upper }}
    ```

    ![image52](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image52.png)

8. With your SQL written in the SQL editor, select Preview to view the changes that Coalesce will make in bulk to your columns. Select **Update** to apply the changes.

    ![image53](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image53.png)

9. With your updates made, Create and Run the node to apply the changes to your object.

    ![image54](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image54.png)

10. You can also apply single column level transformations. Back in the Browser, double click into the **STG_CUSTOMERS** node.

    ![image55](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image55.png)

11. Let's create a new column and apply a transformation that allows us to determine the email domain of each of our customers. Double click on the **Column Name** at the bottom of the mapping grid. Call the column **EMAIL_DOMAIN**.

    ![image56](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image56.png)

12. Next, you'll need to supply a data type, in this case, use the ```VARCHAR(50)``` data type and input this into the mapping grid.

    ![image57](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image57.png)

13. Now let's transform this column. To obtain the email domain, you can use the following SQL to the Transform column to apply the transformation.

    ```SQL
    SPLIT_PART("CUSTOMERS"."EMAIL", '@', 2)
    ```

    ![image58](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image58.png)

14. Create and Run all of the Stage Nodes to build and populate them with data in Snowflake.

## Using Dynamic Table Nodes

For the rest of the staging layer, you will use Dynamic Tables to transform your data while leveraging real-time pipeline updates to your data. Let's begin.

1. Using the shift key, select the **ORDERS** and **ORDER_DETAIL** tables, right click on either node and select Add Node -> Dynamic-Tables -> Dynamic Table Work Node. Coalesce will automatically add a Dynamic Table Work node to each data source.

    ![image59](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image59.png)

2. Next, when using the Dynamic Table package, you need to use a parameter to configure the Dynamic Table warehouse. To do this, navigate to your workspace settings by clicking the gear icon in the upper left corner.

    ![image60](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image60.png)

3. Select Parameters, and copy and paste the following code into the Parameters editor. Select Save once complete.

    ![image61](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image61.png)

    ```JS
    {
    "targetDynamicTableWarehouse": "DEV ENVIRONMENT"
    }
    ```

4. Navigate back to the Browser and double click on the **DT_WRK_ORDERS** table.

    ![image62](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image62.png)

5. Select the Dynamic Table Options dropdown on the right side of the node.

    ![image63](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image63.png)

6. For the Warehouse on which to execute Dynamic Table, input the warehouse that is available in your Snowflake account, or was created when running the setup script called **compute_wh**.

    ![image64](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image64.png)

7. Next, turn on the Downstream toggle to activate the ability for the table to automatically refresh based on the schedule of a downstream table refresh.

    ![image65](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image65.png)

8. Finally, in the General Options dropdown, toggle on the Distinct toggle to ensure only distinct order records are procesed.

    ![image66](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image66.png)

9. Select Create to create the Dynamic Table.

    ![image67](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image67.png)

10. We'll configure the **DT_WRK_ORDER_DETAILS** node the same way.

    ![image68](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image68.png)

11. Additionally, for the **DISCOUNT_AMOUNT** column, we'll apply some logic to simplify our discount policy. Use the following code to apply CASE WHEN logic to the column.

    ![image69](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image69.png)

    ```SQL
    CASE WHEN "ORDER_DETAIL"."DISCOUNT_AMOUNT" > 50 THEN 10 ELSE 1 END
    ```

12. Select Create to create the Dynamic Table.

## Joining Nodes Together

Now that you have built your processing layer, it's time to join some of these objects together to prepare data for further analysis.

1. Using the shift key, select the STG_CUSTOMERS and STG_CUSTOMER_REVIEWS nodes and right click on either node and select Join Nodes -> Stage.

    ![image71](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image71.png)

2. This will create a new stage node which you will be placed within. To configure the join, navigate to the Join tab in the upper left corner of the mapping grid.

    ![image72](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image72.png)

3. Coalesce will automatically infer as much of the join as possible, but you will need to provide the join condition. In this case, you will be joining on CUSTOMER_ID from both nodes. You can either manually confiugre this, or copy and paste the code provided

    ![image73](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image73.png)

    ```SQL
    FROM {{ ref('WORK', 'STG_CUSTOMER_REVIEWS') }} "STG_CUSTOMER_REVIEWS"
    INNER JOIN {{ ref('WORK', 'STG_CUSTOMERS') }} "STG_CUSTOMERS"
    ON "STG_CUSTOMER_REVIEWS"."customer_id" = "STG_CUSTOMERS"."CUSTOMER_ID"
    ```

4. Now that you have provided the join condition, rename the node to STG_CUSTOMER_MASTER.

    ![image74](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image74.png)

5. Navigate back to the mapping grid and delete one of the CUSTOMER_ID columns, as an object can't contain duplicate column names.

    ![image75](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image75.png)

6. Create and Run the node to create and populate the object.

    ![image76](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image76.png)

7. Next, let's join the Dynamic Table nodes together. We want this to be a real-time pipeline, so we'll use another Dynamic Table Work Node to join our nodes together. Select the DT_WRK_ORDERS and DT_WRK_ORDER_DETAIL nodes and right click on either node and select Join Nodes -> Dynamic-Tables -> Dynamic Table Work.

    ![image77](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image77.png)

8. You'll be placed inside of the new Dynamic Table node. Just as we did with the previous stage node, we can configure the join of this node exactly the same way. Navigate to the Join tab and, this time, we'll use ORDER_ID as the join condition. Which you can provide manually, or copy and paste the following code.

    ![image78](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image78.png)

    ```SQL
    FROM {{ ref('WORK', 'DT_WRK_ORDERS') }} "DT_WRK_ORDERS"
    INNER JOIN {{ ref('WORK', 'DT_WRK_ORDER_DETAIL') }} "DT_WRK_ORDER_DETAIL"
    ON "DT_WRK_ORDERS"."ORDER_ID" = "DT_WRK_ORDER_DETAIL"."ORDER_ID"
    ```

9. Rename the node to DT_WRK_ORDER_MASTER.

    ![image79](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image79.png)

10. Navigate back to the mapping grid and remove one of each of the ORDER_ID and ORDER_DETAIL_ID columns.

11. In the Dynamic Table Options dropdown, supply the warehouse you want this Dynamic Table to use. Again, we'll use **compute_wh**.

    ![image81](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image81.png)

12. For this Dynamic Table, leave the Downstream toggle off. Instead, we'll schedule this Dynamic Table to update ever 5 minutes. In the Lag Specification parameter, input 5 as the Time Value. For the Time Period, choose Minutes from the dropdown.

    ![image82](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image82.png)

13. Select Create to create the Dynamic Table object. You've now configured a pipeline of order data updating every 5 minutes.

    ![image83](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image83.png)

## Processing Real Time Data

You now have an order master table, producing data that is cleaned and unified. However, there is still more that can be done to allow your data team to process this data, especially large amounts of this data, in an efficient manner.

1. Currently, you have a pipeline of Dynamic Tables updating every 5 minutes. You can use a view node to query this data as it is updating, so your entire team has insight into this data. To do so, right click on the DT_WRK_ORDER_MASTER node and select Add Node -> View. Create the Veiw.

    ![image84](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image84.png)

2. In an environment where data is updating in real-time, transactional tables containing data like orders data can become quite large and it is unrealistic to processes all of the data in the table every time it updates. In these situations, you can use incremental data loading. Let's learn how to do this by incrementally processing the data from your Dynamic Table pipeline, which only contains the last 30 days worth of data.

    Right click on the DT_WRK_ORDER_MASTER node and select Add Node -> Incremental -> Incremental Load.

    ![image85](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image85.png)

3. Incremental Load Nodes require you to have your data persisted down stream so that the node can incrementally load data into a table that has persisted data. Select the Create button in the Incremental Node you just created.

    ![image86](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image86.png)

4. Next, you need to create a table to persist the data from the incremental node. Right Click on the INC_ORDER_MASTER node in the Browser and select Add Node -> Persistant Stage.

    ![image87](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image87.png)

5. Using the options dropdown on the right side of the node, you use a business key to identify each order. In this case, it is a composite key of ORDER_ID and ORDER_DETAIL_ID.

    ![image88](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image88.png)

6. Once configured, Create and Run the node to buid the object.

    ![image89](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image89.png)

7. With the persistant table created, you can finish configuring the incremental node you just created. Back in the Browser, double click on the INC_ORDER_MASTER tabe you created.

    ![image90](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image90.png)

8. Open the configuration options of the node on the right side of the screen and toggle on Filter data based on Persistent table. This will allow you to configure the node to incrementally filter and process data.

    ![image91](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image91.png)

9. Pass through the Persistent Table storage location to the incremental node, which in this case, is called **WORK**.

    ![image92](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image92.png)

10. Then, pass the persistent table name through as the table name parameter, called **PSTG_ORDER_MASTER**.

    ![image93](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image93.png)

11. Finally, you'll need to supply the data column used for incremental loading in the selector dropdown. In this case, choose **ORDER_TS**

    ![image94](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image94.png)

12. Once complete, navigate to the Join tab of the node and delete the line of code in the SQL editor.

    ![image95](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image95.png)

13. In the upper right corner of the SQL editor, you'll see the Generate Join tool. Select Generate Join and you'll see severa lines of SQL that Coalesce has automatically generated. This is the exact SQL needed to incrementally process your data based on the parameters that were just supplied to the node.

    ![image96](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image96.png)

14. Select **Copy to Editor** to copy the code into the SQL editor for you to reconfigure the dependency of the node as well as supply the incremental logic for the node.

    ![image97](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image97.png)

15. Once this is complete, Run the node. Now you have a view that is processing data incrementally and loading that incremental data into a persistent table.

    ![image98](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image98.png)

## Creating a type 2 slowly changing dimension

So far, you've spent your time building pipelines with some of the more exciting packages from Coalesce Marketplace. But in the real world, not every use case needs you to use functionality like this. What happens when you need to create objects from standard requests, like creating Type 2 Slowly changing dimensions? Coalesce supports this functionality out of the box, without you having to write a single line of code.

1. In the Browser, right click on the STG_MENU node and select Add Node -> Dimension.

    ![image99](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image99.png)

2. Select the options dropdown within the configuration on the right side of the screen.

    ![image100](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image100.png)

3. You are required to supply a business key for a dimension node to work. In this case, we'll use a composite key of MENU_ID and MENU_ITEM_ID.

    ![image101](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image101.png)

4. Next, to configure a Type 2 Slowly Changing Diemnsion (SCD), select the columns you want to have changes tracked on. For this lab, choose the ITEM_NAME column to track changes on.

    ![image102](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image102.png)

5. Now Create and Run the node. Once you Run the node, view the code that was automatically generated for you in the Results panel.

    ![image103](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image103.png)

## Using Cortex Functions

You've processed your orders data and have built some nodes around your customer data, but now it's time to circle back to the STG_CUSTOMER_MASTER table to gain some more insights our of this data. Primarily, we want to generate a sentiment score for each of our customers, based on the reviews they have left us. Let's learn how to do this.

1. In the Browser, right click on the STG_CUSTOMER_MASTER node and select Add Node -> Cortex -> Cortex Functions.

    ![image104](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image104.png)

2. In the Cortex Package options on the right side of the node, select the dropdown.

    ![image105](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image105.png)

3. Toggle on the SENTIMENT toggle.

    ![image106](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image106.png)

4. All Coalesce requires from you is to supply the column you want to generate a sentiment score from. In the mapping grid, select the **REVIEW_TEXT** column, right click on it, and select Duplicate Column.

    ![image107](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image107.png)

5. Double Click on the new column name and rename it to **REVIEW_SENTIMENT**.

    ![image108](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image108.png)

6. With a new column available to use for our sentiment score, we can keep the original text. In the Column Name dropdown in the SENTIMENT toggle, select the **REVIEW_SENTIMENT** column.

    ![image109](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image109.png)

7. Create and Run the node to view the results and see how easy Coalesce makes it to work with Cortex functions in Snowflake.

    ![image110](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image110.png)

## Creating aggregate FCT node

Now that you have an object that is creating a sentiment score about your customers reviews, the next thing you will do is create a Fact table that will store the measures about customers: in this case average review and average sentiment. Let's learn how to do this.

1. If it's not still open, double click into the LLM_CUSTOMER_MASTER node you created in the previous section.

    ![image111](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image111.png)

2. Holding the `command` or `windows` key, select the **CUSTOMER_ID**, **REVIEW_SENTIMENT**, and **STAR_RATING** columns in the mapping grid.

    ![image112](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image112.png)

3. Right click on either of the highlight columns and select Add Node -> Fact. Coalesce will automatically create a new node that contains just these three columns.

    ![image113](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image113.png)

4. We want to apply aggregate functions to this data to determine average sentiment score and star rating, as some customers have reviewed multiple menu items from our restaurant. You can use the same function to apply this transformation. For the **REVIEW_SENTIMENT** column, double click into the **Transform** column and supply the SQL statement here:

    ![image114](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image114.png)

    ```SQL
    AVG({{SRC}})
    ```

5. Apply the same transformation to the **STAR_RATING** column

    ![image115](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image115.png)

    ```SQL
    AVG({{SRC}})
    ```

6. Since you are using aggregate functions in an object, you will need to supply a **GROUP BY**. Navigate to the Join tab, and add a new line to the SQL editor. Add the SQL code here:

    ![image116](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image116.png)

    ```SQL:
    GROUP BY ALL 
    ```

7. Create and Run the node to build the object and populate it with data.

    ![image117](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image117.png)

## Reporting on our data

So far, you've used packages from Coalesce Marketplace to create a pipeline that lerverages different functionality that is fully managed by the nodes you've added to the pipeline. You've also created various objects that can be joined together to provide a holistic view of your data. Let's create a reporting object that brings together your order data, menu data, and customer data.

1. Holding down the shift key, in the browser, select the **PSTG_ORDER_MASTER**, **FCT_CUSTOMER_MASTER**, and **DIM_MENU** nodes (**MAKE SURE TO SELECT THE NODES IN THE ORDER LISTED HERE**). Right click on any of the selected nodes and select Join Nodes -> View.

    ![image118](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image118.png)

2. Coalesce will place you in the node. Rename the node **V_REPORT_MASTER**.

    ![image119](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image119.png)

3. Next, navigate to the join tab and configure manually, or use the following SQL to complete the join condition:

    ![image120](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image120.png)

    ```SQL
    FROM {{ ref('WORK', 'PSTG_ORDER_MASTER') }} "PSTG_ORDER_MASTER"
    INNER JOIN {{ ref('WORK', 'FCT_CUSTOMER_MASTER') }} "FCT_CUSTOMER_MASTER"
    ON "PSTG_ORDER_MASTER"."CUSTOMER_ID" = "FCT_CUSTOMER_MASTER"."CUSTOMER_ID"
    LEFT JOIN {{ ref('WORK', 'DIM_MENU') }} "DIM_MENU"
    ON "PSTG_ORDER_MASTER"."MENU_ITEM_ID"= "DIM_MENU"."MENU_ITEM_ID"
    ```

4. Next, select the Column Name header to alphanumerically sort the column names in the node. Using the shift key, select all of the ```SYSTEM_``` columns and delete them from the node.

    ![image121](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image121.png)

5. Next, you will need to remove any duplicate columns names from the node as an object cannot contain duplicate column names. Remove one of the columns for each of the following column names:
    - CUSTOMER_ID
    - ITEM_NAME
    - MENU_ITEM_ID

    ![image122](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image122.png)

6. Select the Create button to create the view.

    ![image123](using-coalesce-marketplace-to-build-first-class-data-pipelines/assets/image123.png)

## Conclusion and Next Steps

Congratulations on completing your lab. You've mastered the basics of Coalesce and are now equipped to venture into our more advanced features. Be sure to reference this exercise if you ever need a refresher.

We encourage you to continue working with Coalesce by using it with your own data and use cases and by using some of the more advanced capabilities not covered in this lab.

### What We’ve Covered

- How to navigate the Coalesce interface
- Configure storage locations and mappings
- How to add data sources to your graph
- How to prepare your data for transformations with Stage nodes
- How to join tables
- How to apply transformations to individual and multiple columns at once
- How to configure and understand the ML Forecast node

Continue with your free trial by loading your own sample or production data and exploring more of Coalesce’s capabilities with our [documentation](https://docs.coalesce.io/docs) and [resources](https://coalesce.io/resources/). For more technical guides on advanced Coalesce and Snowflake features, check out Snowflake Quickstart guides covering [Dynamic Tables](https://quickstarts.snowflake.com/guide/building_dynamic_tables_in_snowflake_with_coalesce/index.html?index=..%2F..index#1) and [Cortex ML functions](https://quickstarts.snowflake.com/guide/ml_forecasting_ad/index.html?index=..%2F..index#0).

### Additional Coalesce Resources

- [Getting Started][]
- [Documentation][] and [Quickstart Guide][]
- [Video Tutorials][]
- [Help Center][]

Reach out to our sales team at [coalesce.io](https://coalesce.io/contact-us/) or by emailing [sales@coalesce.io](mailto:sales@coalesce.io) to learn more.

[trial account]: https://signup.snowflake.com/
[Google Chrome browser]: https://www.google.com/chrome/
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce
[Snowflake trial account form]: https://signup.snowflake.com/?utm_source=google&utm_medium=paidsearch&utm_campaign=na-us-en-brand-trial-exact&utm_content=go-eta-evg-ss-free-trial&utm_term=c-g-snowflake%20trial-e&_bt=579123129595&_bk=snowflake%20trial&_bm=e&_bn=g&_bg=136172947348&gclsrc=aw.ds&gclid=Cj0KCQiAtvSdBhD0ARIsAPf8oNm6YH7UeRqFRkVeQQ-3Akuyx2Ijzy8Yi5Om-mWMjm6dY4IpR1eGvqAaAg3MEALw_wcB
[Coalesce Marketplace HOL Data Setup File]: https://github.com/coalesceio/hands-on-labs/blob/main/marketplace-hol/hol_setup.sql
[Getting Started]: https://coalesce.io/get-started/
[Documentation]: https://docs.coalesce.io/docs
[Quickstart Guide]: https://docs.coalesce.io/docs/quick-start
[Video Tutorials]: https://fast.wistia.com/embed/channel/foemj32jtv
[Help Center]: mailto:support@coalesce.io

---

## Using Coalesce to Infer File Schemas in Snowflake

## Overview

Loading files into Snowflake can be challenging when dealing with numerous columns, unknown schemas, or when you want to avoid writing extensive DDL. The Inferschema node in Coalesce offers a solution to simplify this process, making your life significantly easier.

### Resources

- [External Data Package - Inferschema][]
- [Sign up for Coalesce][]

[External Data Package - Inferschema]: /docs/marketplace/package/coalesce_snowflake_external-data-package
[Sign up for Coalesce]: https://coalesce.io/get-started/

<YouTube id="pPf5MJyfFL4" />

---

## Incremental Loading with Coalesce

## Overview

Incremental data loading is a technique used in data warehousing to efficiently update large datasets by only processing and loading new or changed data since the last update, instead of reloading the entire dataset each time.

For example, you're an e-commerce company and only want to load order changes, not all orders every time. You can use the Incremental Node to only update records that have shipped.

Coalesce makes this process easy by offering the Incremental Node Package. By using our Package, you can quickly get started building your pipeline with minimal code and configuration. The package contains the following Node Types:

* **Incremental Load** - Performs updates by loading only new or changed records. It minimizes data duplication and optimizes resource use by focusing only on changes since the last load.
* **Looped Load** - Runs multiple loads in a loop, which allows the system to handle large datasets in smaller batches. Each loop processes a subset of data, reducing memory usage and improving performance for larger tables.
* **Run View** - Creates a view that tracks the most recent runs. The view is refreshed with each execution, allowing for easy identification of the latest updates or changes without reloading the entire dataset.

These Node Types make sure you can follow best practices such as:

* **Change detection** - Identifying new, modified, or deleted records in the source system since the last data load.
* **Efficiency** - Reducing processing time and resource usage by only handling changes instead of the full dataset.
* **Timestamp or version tracking** - Using timestamps or version numbers to determine which records have changed.
* **Maintaining historical data** - Preserving historical information while updating with new data.

You can learn more about [Incremental Processing Strategies][] in our blog post.

:::info[High Water Mark]

Coalesce uses High Water Mark as a fast and efficient way to process data

:::

## Before You Begin

1. If you don’t have a Coalesce account, you can start a free trial from our [Snowflake Marketplace listing][].
2. This demo uses the [Snowflake sample data sets][].
3. Set up your [Storage Locations and Storage Mappings][].
4. Add your [source data][].

## Install the Incremental Loading Package

You're going to install the [Incremental Loading Package][].

Click on Node Types to see the three Node Types included in the package:

* Incremental Load
* Loop Load
* Run View

## Add Your Nodes

1. Right click on the `LINEITEM`source node and select **Add Node > Select the Incremental Node Package**.

    

2. Open the **Node Properties**.
3. Take a look at the Storage Location. Make sure it’s not the same location as your source data.

    

    :::info[Best Practice]

    You should create a separate Storage Location for your Incremental data when using this in a production environment.

    :::

4. Click **Create**, to create the Incremental Node. You'll configure the Node in a later step.

    :::info[Viewing SQL]

    You can always see the SQL that is executed by viewing the results.

    

    :::

5. Go back to the DAG and **right click** on the Incremental Node and select **Add Node > Persistent Stage.**
6. Open the **Node Properties** and make sure it’s not the same as the source Node. Make note of the Storage Location name.
7. **Create** and **Run** the Persistent Stage Node.

:::info[Best Practice]

It’s best practice to add a Persistent Stage table to hold the historical data.

:::

At this point your DAG should look similar to the image. The information in the Source Node changes, the Incremental Node only checks the changed information and the Persistent Stage holds the historical data.

## Configure the Incremental Node

1. Open the Incremental Node and go to the **Options.**
2. Leave it as a view, toggle on the **Filter data based on Persistent Table**.
3. Enter the **Storage Location** of the Persistent Table. In this example, it’s `WORK`.
4. Enter the name of the Persistent Table. In this example, it’s `PSTG_LINEITEM`.
5. Change the **Incremental Load Column** to `L_SHIPDATE`. This is the field that the table will use to determine if the data has changed.

    

6. Go to the **Join** tab, click **Generate Join**, and **Copy to Editor**.
7. The query is looking for new or updated records in `LINEITEM` that have a ship date more recent than anything currently in `PSTG_LINEITEM`.

    ```sql
    FROM{{ ref('SAMPLE', 'LINEITEM') }} "LINEITEM"
    WHERE "LINEITEM"."L_SHIPDATE" > 
    (SELECT COALESCE(MAX("L_SHIPDATE"), '1900-01-01') 
                FROM {{ ref_no_link('WORK', 'PSTG_LINEITEM') }} )
    ```

    

:::tip[Congratulations]

You've gone through the steps of settings up an Incremental Node that updates the data in a Persistent Table. The process is quick and easy.

:::

## Add Incremental Loading to an Existing Pipeline (Optional)

1. Add the Incremental Node from the source Node you want to track.
2. Either create or change an existing Stage Node to a Persistent Node Type.
3. If you're updating an existing Node. You’ll need to do a few more things.
    1. Update the Persistent Node Type to reference the Incremental Node. You can do this by updating the Join to use the Incremental Node as the Source and changing the source in the columns.

## Conclusion

Incremental loading is a vital strategy in data warehousing and ETL processes. It enhances efficiency by processing only new or updated data since the last load, significantly reducing load times and data transfer volumes. This approach ensures that target systems stay current with the latest information while minimizing resource usage.

### Resources

* [Incremental loading strategies][]
* [Incremental Processing Strategies][]
* [Incremental Loading Package][]

[Incremental loading strategies]: /docs/build-your-pipeline/incremental-loading-strategies
[Incremental Processing Strategies]: https://coalesce.io/product-technology/incremental-processing-strategies/
[Snowflake sample data sets]: https://docs.snowflake.com/en/user-guide/sample-data
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments/
[source data]: /docs/setup-your-project/add-a-data-source
[Incremental Loading Package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading/
[Snowflake Marketplace listing]: https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce

---

## Using Snowflake Managed Iceberg Tables with Coalesce

## Overview

Learn how to work with data in Iceberg Table format so your organization can work with a single copy of data in Snowflake.

### Resources

- [Iceberg Tables Package][]
- [Sign up for Coalesce][]

[Iceberg Tables Package]: /docs/marketplace/package/coalesce_snowflake_iceberg-tables
[Sign up for Coalesce]: https://coalesce.io/get-started/

<YouTube id="ByUNreNzbOk" />

---

## What is Transform?

Coalesce Transform is the metadata-driven workspace to design, build, and operate governed data pipelines on your cloud data platform. It combines a visual DAG with code, reusable templates, built-in testing and standards, column-level lineage, and instant change propagation so data teams ship reliable data products faster with less maintenance.

## Transform Overview

<YouTube id="u1549Y5272o" />

* **Ship faster with standards**: Use reusable templates and patterns to keep teams aligned while automating repetitive SQL work.
* **Govern as you build**: Enforce naming, contracts, tests, and approvals in the same place you transform—no bolt-ons.
* **See and trust lineage**: Column-level lineage and automatic propagation make impact analysis and updates instant.
* **Work across clouds**: A consistent Coalesce experience across your environments on leading cloud data platforms.
* **Scale without the drag**: Git-native workflows, APIs/CLI, and jobs built for thousands of tables and frequent change.
* **AI where it helps**: Generate SQL, refactor code, auto-document nodes/columns, and summarize changes/commits.

## Get Started

* **Quick Start Guides** - Build your first pipeline in minutes on [Snowflake](https://docs.coalesce.io/docs/guides/snowflake-quickstart?step=1) or [Databricks](https://docs.coalesce.io/docs/guides/databricks-quickstart?step=1).
* **[Build with AI in Transform](docs/coalesce-ai/)** - Turn on assistants for SQL, documentation, and change summaries.
* **[Connect Catalog](https://docs.castordoc.com/)** - Centralize documentation and expose trust signals to the business.

**New to Coalesce?** Start a free Coalesce trial from our [Snowflake Marketplace listing](https://app.snowflake.com/marketplace/listing/GZSTZ1868F5RH/coalesce-coalesce), or **[request a trial](https://coalesce.io/start-free/)** to explore Transform on other cloud platforms.

## **What’s Included**

### **Transformation Workbench**

Design data pipelines in a visual graph and code-based experience.

### **Standards, Templates & Patterns**

Apply metadata-driven templates (naming, conventions, macros) and ready-made node patterns so work is consistent across teams and domains.

### **Testing, Contracts & Approvals**

Define tests and data contracts inside the transformation flow. Surface approvals and policy checks before changes ship, so reliability goes out with every deploy.

### **Change Management & Propagation**

Detect upstream changes, propagate updates across dependent nodes, and run impact analysis at the column level—keeping docs and pipelines in sync by design.

### **Change Management & Impact Analysis**

Explore dynamic column-level lineage directly from your graph to understand dependencies, breakages, and blast radius before you change anything.

### **Environments & Storage Mappings**

Separate logical design from physical deployment. Use storage mappings per environment (`dev/test/prod`) to promote with confidence.

### **Jobs & Orchestration**

Schedule and monitor jobs (deploy, refresh, build) and integrate with external orchestrators as needed.

### **AI Assistants (In Transform)**

Use AI to generate or refactor SQL, draft node/column descriptions, and summarize changes/commit messages to reduce busywork and keep documentation fresh.

### **API & CLI**

Automate runs, query metadata, and wire Coalesce into CI/CD with a REST API and CLI.

### **Marketplace & Packages**

Jump-start projects with pre-built node types and solutions (for example, Dynamic Tables, platform base node types, accelerators).

## **Key Concepts in Transform**

* **Projects & Workspaces** - Organize work, connect Git, and manage access.
* **Graphs & Nodes** - Model data products with node types (Source, Stage, Dimension, Fact, View, Custom).
* **Policies & Templates** - Enforce standards with reusable patterns and metadata-driven rules.
* **Storage Mappings** - Bind logical nodes to physical schemas per environment.
* **Deploy, Refresh & Jobs** - Operate pipelines with predictable, auditable runs.

:::tip

Looking for documentation and discovery? Connect to **[Coalesce Catalog](https://docs.castordoc.com/)** to centralize docs and surface trust signals where analysts work.

:::

---

## Choosing the Right Node

Use this page as a quick reference for which Node Type to choose. Follow each scenario link for in-depth information. Use the **Related packages** column to find Coalesce Marketplace or built-in packages for each scenario.

## Reference

Use the **Scenario** links for the matching section in [Node types by scenario][] or the primary package doc. Use **Related packages** for every Marketplace listing or [Nodes and Node Types][] for built-in behavior.

| Scenario | Related packages |
| --- | --- |
| [Modeling Baseline With Built-In Node Types](/docs/guides/node-type-selection#modeling-baseline-with-built-in-node-types): Stage, Dimension, Fact, Persistent Stage, View | [Nodes and Node Types][] |
| [Base Node Types on Snowflake, Databricks, and BigQuery](/docs/guides/node-type-selection#base-node-types-on-snowflake-databricks-and-bigquery): Work, Persistent Stage, Dimension, Fact, View, Code; Snowflake adds Factless Fact and SQL Stage | [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [Advanced Deploy Merge and Tracking](/docs/guides/node-type-selection#advanced-deploy-merge-and-tracking): Dimension, Fact, Persistent Stage, Work, Factless Fact Advanced Deploy variants | [Snowflake Base Node Types Advanced Deploy][], [BigQuery Base Node Types Advanced Deploy][] |
| [Snowflake Change Feeds and Tasks](/docs/guides/node-type-selection#snowflake-change-feeds-and-tasks): Streams and Tasks Node Types | [Snowflake Streams and Tasks][] |
| [When You Land Each Batch in a Staging Layer](/docs/guides/node-type-selection#when-you-land-each-batch-in-a-staging-layer): Stage (built-in) or Work | [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [When You Keep Curated Rows Across Runs on a Business Key](/docs/guides/node-type-selection#when-you-keep-curated-rows-across-runs-on-a-business-key): Persistent Stage | [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [When You Load Only New or Changed Rows](/docs/guides/node-type-selection#when-you-load-only-new-or-changed-rows): Incremental Load; Looped Load, Run View, Grouped Incremental load, Test Passed Records, Test Failed Records, Code (Snowflake listing); Incremental Load, Code (Databricks listing) | [Incremental Loading package][], [Incremental Nodes][] |
| [When You Describe Business Entities and Slowly Changing Attributes](/docs/guides/node-type-selection#when-you-describe-business-entities-and-slowly-changing-attributes): Dimension | [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [When You Store Measures in Fact Tables](/docs/guides/node-type-selection#when-you-store-measures-in-fact-tables): Fact | [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [When You Record Events or Relationships Without Measures](/docs/guides/node-type-selection#when-you-record-events-or-relationships-without-measures): Factless Fact (Snowflake base), Factless Fact Advanced Deploy (Snowflake and BigQuery Advanced Deploy), or Fact or Persistent Stage | [Snowflake Base Node Types][], [Snowflake Base Node Types Advanced Deploy][], [Databricks Base Node Types][], [BigQuery Base Node Types][], [BigQuery Base Node Types Advanced Deploy][] |
| [When You Want a Simple View Object in the Graph](/docs/guides/node-type-selection#when-you-want-a-simple-view-object-in-the-graph): built-in View or package View Node Type (Snowflake, Databricks, BigQuery); Create As View on other types per listing | [Nodes and Node Types][], [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][] |
| [When You Own the Create and Run Templates](/docs/guides/node-type-selection#when-you-own-the-create-and-run-templates): Code | [Snowflake Base Node Types][], [Snowflake Base Node Types Advanced Deploy][], [Databricks Base Node Types][], [BigQuery Base Node Types][], [BigQuery Base Node Types Advanced Deploy][], [Incremental Loading package][], [Incremental Nodes][] |
| [When You Transform Entirely in Hand-Authored SQL (Snowflake)](/docs/guides/node-type-selection#when-you-transform-entirely-in-hand-authored-sql-snowflake): SQL Stage | [Snowflake Base Node Types][] |
| [When Several Sources Feed One Conformed Node](/docs/guides/node-type-selection#when-several-sources-feed-one-conformed-node): Multi Source or chained Work | [Multi Source Nodes][] |
| [Advanced Deploy Options by Node Family](/docs/guides/node-type-selection#advanced-deploy-options-by-node-family): Snowflake and BigQuery matrices on listings | [Snowflake Base Node Types Advanced Deploy][], [BigQuery Base Node Types Advanced Deploy][] |
| [Files, stages, pipes, and APIs on Snowflake](/docs/marketplace/package/coalesce_snowflake_external-data-package): External Data Node Types | [Snowflake External Data Package][] |
| [Batch files into Delta on Databricks](/docs/marketplace/package/coalesce_databricks_external-data-package): External data package | [Databricks External data package][] |
| [Dynamic tables on Snowflake](/docs/marketplace/package/coalesce_snowflake_dynamic-tables): Dynamic Table Node Types | [Snowflake Dynamic Tables package][] |
| [Iceberg tables on Snowflake](/docs/marketplace/package/coalesce_snowflake_iceberg-tables): Iceberg Node Types | [Snowflake Iceberg Tables package][] |
| [Materialized views on Snowflake](/docs/marketplace/package/coalesce_snowflake_materialized-view): Materialized View Node Types | [Snowflake Materialized View package][] |
| [Interactive tables on Snowflake](/docs/marketplace/package/coalesce_snowflake_interactive-table): Interactive Table Node Types | [Snowflake Interactive Table package][] |
| [Semantic layer on Snowflake](/docs/marketplace/package/coalesce_snowflake_semantic-view): Semantic View and Semantic Query | [Snowflake Semantic View package][] |
| [Lakeflow pipelines on Databricks](/docs/marketplace/package/coalesce_databricks_lakeflow-declarative-pipelines): Streaming and pipeline Node Types | [Databricks Lakeflow Declarative Pipelines package][] |
| [Materialized views on Databricks](/docs/marketplace/package/coalesce_databricks_materialized-view): Materialized View Node Types | [Databricks Materialized View package][] |
| [Near real-time stream with deferred MERGE on Snowflake](/docs/marketplace/package/coalesce_snowflake_deferred-merge): Deferred Merge streams | [Snowflake Deferred Merge package][] |
| [Calendar, PIVOT, UNPIVOT, and advanced SQL on Snowflake](/docs/marketplace/package/coalesce_snowflake_functional-node-types): Functional Node Types | [Snowflake Functional Node Types package][] |
| [Calendar, PIVOT, and UNPIVOT on Databricks](/docs/marketplace/package/coalesce_databricks_functional-node-types): Functional node types | [Databricks Functional node types package][] |
| [Create or alter instead of drop-recreate on Snowflake](/docs/marketplace/package/coalesce_snowflake_create-or-alter-package): Create or Alter Node Types | [Snowflake Create or Alter package][] |
| [Metrics and profiling on Snowflake](/docs/marketplace/package/coalesce_snowflake_data-quality): Data quality Node Types | [Snowflake Data quality package][] |
| [Great Expectations-style tests](/docs/marketplace/package/coalesce_snowflake_test-utility): Test Utility Node Types | [Snowflake Test Utility package][], [Databricks Test Utility package][] |
| [Masking and row access policies on Snowflake](/docs/marketplace/package/coalesce_snowflake_data-security-package): Data security views | [Snowflake Data security package][] |
| [Data Vault 2.0 methodology](/docs/marketplace/package/coalesce_snowflake_data-vault-by-scalefree): Hub, link, satellite, PIT, and related Node Types | [Data Vault by Scalefree for Snowflake][], [Data Vault by Scalefree for Databricks][] |
| [Cortex and related ML and AI on Snowflake](/docs/marketplace/package/coalesce_snowflake_cortex): Cortex Node Types | [Snowflake Cortex package][] |

:::info[Decisions before you pick a Node label]

- **What each row stands for:** One row in the target should match one business instance at the level you need (for example one customer, one order, or one shipment line).
- **History:** Overwrite in place (Type 1 style) or keep prior versions (Type 2 style or event-level facts).
- **Keys:** Business keys anchor merges on Dimensions and many Facts; Facts without a business key often use insert-only loads. Defaults are described in [Nodes and Node Types][].
- **Persistence:** Staging-style types often replace each batch; persistent types keep prior rows unless the template replaces them.

:::

[Node types by scenario]: /docs/guides/node-type-selection
[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Incremental Loading package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading
[Snowflake Base Node Types]: /docs/marketplace/package/coalesce_snowflake_base-node-types
[Snowflake Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy
[Snowflake Streams and Tasks]: /docs/marketplace/package/coalesce_snowflake_streams-and-tasks
[Databricks Base Node Types]: /docs/marketplace/package/coalesce_databricks_base-node-types
[Incremental Nodes]: /docs/marketplace/package/coalesce_databricks_incremental-nodes
[BigQuery Base Node Types]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types
[BigQuery Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types-advanced-deploy
[Multi Source Nodes]: /docs/build-your-pipeline/multisource-nodes
[Snowflake External Data Package]: /docs/marketplace/package/coalesce_snowflake_external-data-package
[Databricks External data package]: /docs/marketplace/package/coalesce_databricks_external-data-package
[Snowflake Dynamic Tables package]: /docs/marketplace/package/coalesce_snowflake_dynamic-tables
[Snowflake Iceberg Tables package]: /docs/marketplace/package/coalesce_snowflake_iceberg-tables
[Snowflake Materialized View package]: /docs/marketplace/package/coalesce_snowflake_materialized-view
[Snowflake Interactive Table package]: /docs/marketplace/package/coalesce_snowflake_interactive-table
[Snowflake Semantic View package]: /docs/marketplace/package/coalesce_snowflake_semantic-view
[Databricks Lakeflow Declarative Pipelines package]: /docs/marketplace/package/coalesce_databricks_lakeflow-declarative-pipelines
[Databricks Materialized View package]: /docs/marketplace/package/coalesce_databricks_materialized-view
[Snowflake Deferred Merge package]: /docs/marketplace/package/coalesce_snowflake_deferred-merge
[Snowflake Functional Node Types package]: /docs/marketplace/package/coalesce_snowflake_functional-node-types
[Databricks Functional node types package]: /docs/marketplace/package/coalesce_databricks_functional-node-types
[Snowflake Create or Alter package]: /docs/marketplace/package/coalesce_snowflake_create-or-alter-package
[Snowflake Data quality package]: /docs/marketplace/package/coalesce_snowflake_data-quality
[Snowflake Test Utility package]: /docs/marketplace/package/coalesce_snowflake_test-utility
[Databricks Test Utility package]: /docs/marketplace/package/coalesce_databricks_databricks-test-utility
[Snowflake Data security package]: /docs/marketplace/package/coalesce_snowflake_data-security-package
[Data Vault by Scalefree for Snowflake]: /docs/marketplace/package/coalesce_snowflake_data-vault-by-scalefree
[Data Vault by Scalefree for Databricks]: /docs/marketplace/package/coalesce_databricks_data-vault-by-scalefree-for-databricks
[Snowflake Cortex package]: /docs/marketplace/package/coalesce_snowflake_cortex

---

## Node Types by Scenario

Pick Node Types by modeling scenario, with platform notes and links to package listings. For the compact scenario-to-package table, start from [Choosing the Right Node][].

## Modeling Baseline With Built-In Node Types

Built-in Node types ship with Coalesce and cover staging, conformed modeling, persistent handoff, and simple views. Enable them under **Build Settings > Node Types**. Behavior, defaults, and how templates run: [Nodes and Node Types][].

## Base Node Types on Snowflake, Databricks, and BigQuery

Marketplace Base Node Types packages supply Work, Persistent Stage, Dimension, Fact, View, and Code templates per platform. Snowflake also includes Factless Fact and SQL Stage in the same package. Install the package for your engine, then use the listing as the source of truth for Node names and options: [Snowflake Base Node Types][], [Databricks Base Node Types][], [BigQuery Base Node Types][].

## Advanced Deploy Merge and Tracking

Advanced Deploy variants add merge, change tracking, last-modified comparison, and related options on Dimensions, Facts, Persistent Stages, Work, and Factless Facts when each listing supports them. Snowflake Advanced Deploy and BigQuery Advanced Deploy both ship these patterns. The BigQuery Advanced Deploy listing also includes Factless Fact Advanced Deploy where documented. Package references: [Snowflake Base Node Types Advanced Deploy][], [BigQuery Base Node Types Advanced Deploy][]. For a summary by Node family on this page, see [Advanced Deploy Options by Node Family](#advanced-deploy-options-by-node-family).

## Snowflake Change Feeds and Tasks

Use **Streams and Tasks** Node Types when you orchestrate change feeds and scheduled work in Snowflake. Node names and options: [Snowflake Streams and Tasks][].

## When You Land Each Batch in a Staging Layer

You need an intermediate table or view that usually holds only the current extract or one run's slice, not the long-lived conformed row set.

**Use:** built-in Stage on any data platform, or Work from Base Node Types on your data platform.

- **Snowflake:** [Snowflake Base Node Types][]. Truncate Before for full replace each run; truncation off appends and can duplicate keys. [Snowflake Base Node Types Advanced Deploy][] for Work Advanced Deploy with ASOF joins and the extra load options on the matrix.
- **Databricks:** [Databricks Base Node Types][], same truncate versus append choice, SQL you control per batch.
- **BigQuery:** [BigQuery Base Node Types][] with Truncate Before, partitions, and predicates for the intended slice. [BigQuery Base Node Types Advanced Deploy][] for Work Advanced Deploy when you need listing options.

## When You Keep Curated Rows Across Runs on a Business Key

You need a table that retains loaded rows, supports merge-style loads, and can track change history for selected columns when the package exposes those options.

**Use:** Persistent Stage from Base Node Types, or Persistent Stage Advanced Deploy for the full merge and tracking matrix on the listing.

- **Snowflake:** [Snowflake Base Node Types][]; [Snowflake Base Node Types Advanced Deploy][] for change tracking, last modified comparison, and related options.
- **Databricks:** [Databricks Base Node Types][] as the handoff between staging SQL and conformed Dimensions or Facts.
- **BigQuery:** [BigQuery Base Node Types][]; [BigQuery Base Node Types Advanced Deploy][] for last modified comparison and related options.

Pair with [When You Load Only New or Changed Rows](#when-you-load-only-new-or-changed-rows) when each Job should touch only new or changed sources.

## When You Load Only New or Changed Rows

You run on a schedule or trigger Jobs often, have a watermark or equivalent, and want each Job to touch only new or changed rows while a persistent table holds prior loads.

**Use:** Incremental Load and related types from Marketplace packages, wired to a Persistent Stage, Dimension, or Fact from Base Node Types.

- **Snowflake:** [Incremental Loading package][]. Incremental Load after source or Work; persistent table points at a Persistent Stage or other persistent target from [Snowflake Base Node Types][]. Looped Load for very large sources; Run View for run metadata; Grouped Incremental load, Test Passed Records, Test Failed Records, and **Code** on the same listing when those patterns apply. Tutorial: [Incremental loading tutorial][].
- **Databricks:** [Incremental Nodes][]. Incremental Load downstream of the source; persistent target from [Databricks Base Node Types][]; first-run versus incremental toggle per package docs. **Code** on the listing for custom templates alongside incremental patterns.
- **BigQuery:** Slice in Work or Persistent Stage SQL on [BigQuery Base Node Types][], high-water marks from the persistent table, [ref_no_link()][] when the SQL reference must not add a DAG edge. [BigQuery Base Node Types Advanced Deploy][] on persistent Dimension or Fact for documented merge and comparison options.

## When You Describe Business Entities and Slowly Changing Attributes

You model customers, products, dates, or other descriptive context, usually one row per entity. Dimension Nodes require a business key.

**Use:** Dimension from Base Node Types, or Dimension Advanced Deploy when you need merge, change tracking, and last modified options on the Advanced Deploy listing.

- **Type 1:** Attributes that should not version stay on one row. Do not select change tracking for those attributes; updates overwrite the current row.
- **Type 2:** Select change tracking on columns that should version when values change. Type 2 increases row count and merge work compared with Type 1.

### Orders and Statuses

If you only need current status on the order row, keep status non-tracked on an order-level Dimension or use a conformed status Dimension without Type 2 on the order. If you need full status history, use Type 2 on an order-keyed Dimension or a Fact table with one row per status change plus order, time, and status keys. Pick the shape your reports query.

### Package Listings for Dimensions

Dimension and Dimension Advanced Deploy on [Snowflake Base Node Types][] and [Snowflake Base Node Types Advanced Deploy][]. Dimension on [Databricks Base Node Types][]. Dimension and Dimension Advanced Deploy on [BigQuery Base Node Types][] and [BigQuery Base Node Types Advanced Deploy][]. Enable SCD Type 2 and per-column change tracking on the BigQuery Advanced Deploy listing where you need them. SCD edge cases: [SCD Type 2 change detection][].

## When You Store Measures in Fact Tables

You model numeric measures or additive facts where each row is one business event or line item.

**Use:** Fact from Base Node Types, or Fact Advanced Deploy for documented MERGE, INSERT, last modified comparison, and unmatched-row handling on the package matrix.

With a business key at that row level, templates usually MERGE; without one, many Fact templates INSERT. Confirm in your package version and generated SQL.

Order header versus line usually means different tables: header attributes in Dimensions with one row per order, line measures in Facts with one row per line. Degenerate dimensions on the Fact are fine when they are not conformed across many Facts.

### Package Listings for Facts

Fact and Fact Advanced Deploy on [Snowflake Base Node Types][] and [Snowflake Base Node Types Advanced Deploy][]. Fact on [Databricks Base Node Types][]. Fact on [BigQuery Base Node Types][] and [BigQuery Base Node Types Advanced Deploy][].

## When You Record Events or Relationships Without Measures

You track coverage, eligibility, milestones, or many-to-many relationships with keys and optionally time, without numeric measures.

**Use:** **Factless Fact** on Snowflake from the base package, or **Factless Fact Advanced Deploy** from Snowflake or BigQuery Advanced Deploy packages where the listing provides them. Otherwise use Fact or Persistent Stage from Base Node Types so each row is one event or relationship.

- **Snowflake:** [Snowflake Base Node Types][] for **Factless Fact**; [Snowflake Base Node Types Advanced Deploy][] for **Factless Fact Advanced Deploy**. This matches the same option family as other advanced facts on the matrix.
- **Databricks:** [Databricks Base Node Types][], with keys and merge or insert behavior that match rows per event or relationship, including bridge-style tables.
- **BigQuery:** [BigQuery Base Node Types][] with SQL and keys for relationship or event tables from the base package; [BigQuery Base Node Types Advanced Deploy][] for **Factless Fact Advanced Deploy** when you want the dedicated Factless Fact template on the listing.

## When You Want a Simple View Object in the Graph

You want a VIEW for clarity, security boundaries, or a thin semantic layer, with a dedicated Node in the DAG.

**Use:** Built-in **View** from [Nodes and Node Types][], or the **View** Node Type from Base Node Types in [Coalesce Marketplace][] for your data platform. Enable built-in View in **Build Settings > Node Types** if it is off. Dimension, Fact, and Work can also use **Create As View** on the listing when a view materialization fits those layers.

- **Snowflake:** **View** Node Type on [Snowflake Base Node Types][].
- **Databricks:** **View** Node Type on [Databricks Base Node Types][].
- **BigQuery:** **View** Node Type on [BigQuery Base Node Types][].

## When Several Sources Feed One Conformed Node

You combine extracts or shards before a single Dimension or Fact.

**Use:** Multi Source on Work, Persistent Stage, Fact, or other types your Base Node Types or Advanced Deploy package supports, or chain Work Nodes that UNION inputs.

Patterns and ordering: [Multi Source Nodes][]. Check MultiSource support on Snowflake and BigQuery package matrices for the Node Type you chose.

## Advanced Deploy Options by Node Family

This table summarizes [Snowflake Base Node Types Advanced Deploy][]. Base options for the same families are on [Snowflake Base Node Types][]. Change tracking and last modified comparison apply to Dimension and Persistent Stage in this matrix, not to Fact or Factless Fact. Business key applies to Dimension, Fact, and Persistent Stage. Exact flags and methods can change with package version.

| Node family | Change tracking | Last modified comparison | Business key | Typical methods | Use case |
| ----------- | --------------- | ------------------------ | ------------ | ---------------- | -------- |
| Dimension | Yes | Yes | Yes | MERGE, INSERT, UPDATE | Conformed customer or product rows on a natural key, optional Type 2 on selected attributes. |
| Fact | No | Yes | Yes | MERGE, INSERT | Order lines or shipments with MERGE when a stable business key exists, or insert-only when it does not. |
| Factless Fact | No | No | No | MERGE, INSERT | Coverage or eligibility without measures, for example promotions by store and week. |
| Persistent Stage | Yes | Yes | Yes | MERGE, INSERT | Handoff between ingest and conformed layers, rows retained across runs. |
| Work | No | No | No | INSERT | Per-batch landing or scratch results replaced or appended each run. |

On BigQuery, see [BigQuery Base Node Types][] and [BigQuery Base Node Types Advanced Deploy][] for similar matrices that cover Dimension, Fact, Persistent Stage, Work, and **Factless Fact Advanced Deploy**.

## When You Own the Create and Run Templates

You ship or extend package behavior with workspace-owned Jinja and SQL instead of only UI-driven options.

**Use:** **Code** Node Types from the package that matches your platform and layer. Listings include **Code** on [Snowflake Base Node Types][], [Snowflake Base Node Types Advanced Deploy][], [Databricks Base Node Types][], [BigQuery Base Node Types][], [BigQuery Base Node Types Advanced Deploy][], [Incremental Loading package][], and [Incremental Nodes][].

## When You Transform Entirely in Hand-Authored SQL (Snowflake)

You skip the column-mapping grid and keep transformation logic in a single SQL body.

**Use:** **SQL Stage** on [Snowflake Base Node Types][] for table materialization and stage options on the listing.

## Validate Changes in a Development Environment

Switching Node Types or moving between base and Advanced Deploy after objects exist can change materialization, templates, and options. Test Deploy and refresh in a development environment before production.

:::warning[Macros and the Node editor]

Invalid Workspace macros for your platform's SQL engine can prevent options from rendering or behaving as expected. Use supported functions so the Node UI and generated SQL stay aligned.

:::

Sample Node YAML and graphs in the public Coalesce GitHub organization illustrate SCD and persistent patterns. For example, browse repositories such as `Coalesce-Base-Node-Types` and `Coalesce-Base-Node-Types---Advanced-Deploy`.

## What's Next?

- [Nodes and Node Types][] - built-in types and how Create Template and Run Template work
- [Coalesce Marketplace][] and [Manage Packages][]
- [Snowflake Base Node Types][] and [Snowflake Base Node Types Advanced Deploy][]
- [Databricks Base Node Types][]
- [BigQuery Base Node Types][] and [BigQuery Base Node Types Advanced Deploy][]
- [Multi Source Nodes][]
- [SCD Type 2 change detection][]
- [ref() and ref_no_link()][], BigQuery graph references
- [Snowflake Streams and Tasks][]
- [Incremental loading strategies][], [Incremental loading tutorial][], [Incremental Loading package][], and [Incremental Nodes][] when you are building incremental loads

Package listings remain the source of truth for exact Node labels. For example, **Grouped Incremental load**, **Test Passed Records**, **Test Failed Records**, and other types on the incremental and functional packages.

---

[Choosing the Right Node]: /docs/marketplace/choosing-the-right-node
[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Incremental loading strategies]: /docs/build-your-pipeline/incremental-loading-strategies
[Incremental loading tutorial]: /docs/guides/using-incremental-nodes
[Incremental Loading package]: /docs/marketplace/package/coalesce_snowflake_incremental-loading
[Snowflake Base Node Types]: /docs/marketplace/package/coalesce_snowflake_base-node-types
[Snowflake Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy
[Snowflake Streams and Tasks]: /docs/marketplace/package/coalesce_snowflake_streams-and-tasks
[Databricks Base Node Types]: /docs/marketplace/package/coalesce_databricks_base-node-types
[Incremental Nodes]: /docs/marketplace/package/coalesce_databricks_incremental-nodes
[BigQuery Base Node Types]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types
[BigQuery Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_bigquery_bigquery-base-node-types-advanced-deploy
[Multi Source Nodes]: /docs/build-your-pipeline/multisource-nodes
[SCD Type 2 change detection]: /docs/build-your-pipeline/build-how-to/scd-type2-case-and-change-detection
[ref() and ref_no_link()]: /docs/reference/ref-functions
[ref_no_link()]: /docs/reference/ref-functions
[Coalesce Marketplace]: /docs/marketplace
[Manage Packages]: /docs/marketplace/manage-packages

---

## Package and Coalesce Marketplace FAQs

**Who can build a package?**

- At launch, we are excited to offer a limited selection of Coalesce certified Packages on the Node Marketplace. These Packages have been carefully crafted by our team to address common use cases and provide you with a head start in your data transformation journey.

**Are Packages secure?**

- You can ensure that our packages adhere to best practices and industry standards, saving you time and effort in the long run.

**What is the Coalesce Marketplace?**

- The Coalesce Marketplace is a centralized directory of templated collections of code (known as Packages).
- Packages are premade, ready-to-use collections of node types and macros that data teams can use for specific use case (for example, building a Data Vault).

---

## Learn How to Build Coalesce Packages



---

## Package Marketplace

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS CATALOG IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

    
        Package Marketplace
        Discover and install pre-built packages to accelerate your data pipeline development with Coalesce.
    

    <MarketplaceSearch />

---

## Install and Manage Packages

Coalesce Packages are custom Node Types that are accessed from the [Coalesce Marketplace][].

Coalesce Marketplace is a centralized library that provides data teams with templated collections of code, known as Packages. Every Package provides a curated selection of different Snowflake objects (known as node types in Coalesce), which are bundled together for a particular purpose, such as building out declarative data pipelines or implementing a Data Vault.

## Install Packages

You can install packages from our Coalesce Marketplace. Packages are installed on the Workspace level only.

## Use a Package

Once a package is installed, you can use them like any other Node. Create a new Node in the Mapping Grid or change the node type in the Node Editor.

## Manage Your Packages

Packages can come installed with multiple Macros and Node Types.

### Use Package Macros In Your Workspace

If your Package includes macros, you can use them in any Node. First, add them to your Workspace macros so they’re available throughout your Workspace.

1. Go to **Build Settings > Macros**.
2. Select the Workspace macros from the list.

Next, import the macros in your Workspace macros:

```sql
{% import "<package-alias>" as <friendly-name> with context %}
```

After importing, call the macro using its alias:

```sql
{{ <friendly-name>.<package-macro-name>(<parameter>, <parameter>) }}
```

#### Example Using Package Macros

If you installed the `@coalesce/test-utility` Package with the alias `testUtils` and want to use a macro called `expect_table_row_count_to_be_between` in a Node test:

1. In your Workspace macros, add:

    ```sql
    {% import "testUtils" as testUtils with context %}
    ```

    

2. Call the macro:

    ```sql
    {{ testUtils.expect_table_row_count_to_be_between('{{this}}', 1, 5) }}
    ```

    

### Workspace Macros

If you want to use Workspace macros in a package node, you can use them as normal. The import syntax isn’t needed.

### Edit the Package Configuration

The package configuration is a set of variables you can set values for to customize the package for your use. The available configuration options depend on the package. Not all packages have a configuration. For example, `timestamp_format` would let you set the format to HH:MM:SS.

1. Go to **Build Settings > Packages**.
2. For the package you want to edit, click **Edit Config**.

### Enable or Disable Package Nodes

Packages will be listed by the package alias created during installation.

Go to **Build Settings > Node Types**, to toggle the node.

## Duplicate or Copy Packages

You can duplicate packages by going to **Build Settings > Node Type** and selecting Duplicate next to the package you want to copy. Duplicating a package allows you to [customize][] the package Node Types.

## Upgrading Packages

1. Install the new package version. You'll follow the same steps as installing a package, just choose the version you want to install.
2. Copy the package configuration from the old package to the new package.
3. Bulk update the node types of any nodes you want to move.
4. Uninstall the old package version.

## Uninstall a Package

Packages can only be uninstalled if they aren’t in use. In use packages will return a list of locations the package is in use. You’ll need to remove those locations before you can delete them.

To delete a package:

1. Go to **Build Settings > Packages**.
2. Click **Uninstall** next to the package you want to delete.
3. This will remove the package from the current workspace.

Your packages are committed using the package alias you created during installation. The file will contain the package ID.

## Git and Packages

Your packages are committed using the package alias you created during installation. The file will contain the package ID.

To see an example of a Git Commit, review [What Gets Committed][].

[What Gets Committed]: /docs/git-integration/what-gets-committed
[Coalesce Marketplace]: /docs/marketplace
[customize]: /docs/build-your-pipeline/user-defined-nodes/

---

## BigQuery Base Node Types - Advanced Deploy

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/bigquery/bigquery-base-node-types-advanced-deploy</code> <CopyButton textToCopy='@coalesce/bigquery/bigquery-base-node-types-advanced-deploy' />
Supported Platform: 
Latest Version: 1.1.0 (March 30, 2026)

alter tableclusteringdata retentiondimensional modelingpartitioningprimary key

## Overview
Reusable templates to build and deploy BigQuery objects consistently. They support features like partitioning, clustering, table expiration, and rounding modes for performance and governance.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/bigquery/bigquery-base-node-types-advanced-deploy</code> <CopyButton textToCopy='@coalesce/bigquery/bigquery-base-node-types-advanced-deploy' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## BigQuery - Base Node Types Advanced Deploy – Brief Summary

- **Fact nodes**  
  Represent measurable business events such as sales, transactions, or usage. These are the core tables used for reporting, KPIs, and trend analysis.

- **Dimension nodes**  
  Provide business context for facts, like customer, product, time, or location. They help slice and analyze facts in meaningful ways.

- **Factless fact nodes**  
  Capture business events that do not have numeric measures, such as attendance, eligibility, or process milestones. Useful for compliance, tracking, and operational analysis.

- **Persistent nodes**  
  Store curated, reusable data that remains stable over time. They act as trusted reference layers, reduce reprocessing, and ensure consistency across reports and teams.

- **Work nodes**  
  Temporary or intermediate processing layers used during transformations. They support complex logic and performance optimization but are not intended for direct business consumption.

**Summary:**  
Together, these node types ensure data is accurate, reusable, scalable, and aligned with business reporting and decision-making needs.

----

## Prerequisites Checklist

Before the application can connect, ensure the following are in place:

- **API Enabled**
  - The BigQuery API is enabled in the Google Cloud Console.

- **Service Account Key**
  - A `.json` key file has been generated for the Service Account.
  - The key file path is correctly provided to the application.

- **IAM Permissions**

  - **roles/bigquery.jobUser** (Project level)
    - Allows the service account to run query jobs and pay for compute resources within the project.

  - **roles/bigquery.dataEditor** (Dataset level)
    - Granted only on the required dataset to limit access.
    - Provides permissions to read, create, and alter data within that dataset.
  
-----

## Nodetypes Config Matrix

| Category | Feature                         | Dim | Fact | Factless | Work | PStage |
|----------|----------------------------------|-----|------|----------|------|--------|
| Create   | Create As Table                  | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Create As View                   | ✅  | ✅   | ⬜       | ✅   | ⬜     |
| Create   | Primary Key                      | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Enable Partitioning              | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Enable Clustering                | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Table Expiration                 | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Default Rounding Mode (Optional) | ✅  | ✅   | ✅       | ✅   | ✅     |
| Create   | Labels                           | ⬜  | ⬜   | ⬜       | ⬜   | ⬜     |
| Load     | MultiSource                      | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Update Strategy                  | ✅  | ⬜   | ⬜       | ⬜   | ⬜     |
| Load     | Unmatched Record Strategy        | ✅  | ✅   | ⬜       | ⬜   | ⬜     |
| Load     | Business Key                     | ✅  | ✅   | ⬜       | ⬜   | ✅     |
| Load     | Last Modified Comparison         | ✅  | ✅   | ⬜       | ⬜   | ✅     |
| Load     | Change Tracking                  | ✅  | ⬜   | ⬜       | ⬜   | ✅     |
| Load     | Exclude Columns from Merge       | ✅  | ✅   | ⬜       | ⬜   | ✅     |
| Load     | Truncate Before                  | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Distinct                         | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Group By All                     | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Insert Zero Key Record           | ✅  | ⬜   | ⬜       | ⬜   | ⬜     |
| Load     | Methods                          | MERGEINSERT/UPDATE | MERGEINSERT | MERGE |INSERT | MERGEINSERT |
| Others   | Enable Tests                     | ✅  | ✅   | ✅       | ✅   | ✅     |
| Others   | Pre-SQL                          | ✅  | ✅   | ✅       | ✅   | ✅     |
| Others   | Post-SQL                         | ✅  | ✅   | ✅       | ✅   | ✅     |

---

## General Guidelines

* Modifying partitioning settings on a deployed table will cause the table to be **dropped and recreated** during the next deployment.
* Similar to partitioning, modifying clustering columns on an existing table will cause the table to be **dropped and recreated**.
* Columns that are part of clustering or partitioning cannot be **renamed** directly. If such changes are included, the ALTER statement will fail during deployment.
* Changes involving the drop and re-creation of table may not reflect accurately when executed in the **coalesce browser**.
* **NOT NULL** constraints cannot always be applied using ALTER TABLE

## Base Node Types Advanced Deploy

The Coalesce Base Node Types Package includes:

* [Work Advanced Deploy](#work-advanced-deploy)
* [Persistent Stage Advanced Deploy](#persistent-stage-advanced-deploy)
* [Dimension Advanced Deploy](#dimension-advanced-deploy)
* [Fact Advanced Deploy](#fact-advanced-deploy)
* [Factless Fact Advanced Deploy](#factless-fact-advanced-deploy)
* [Code](#code)

---

## Work Advanced Deploy
The Coalesce Work Node is a versatile node that allows you to develop and deploy a Work table/view in Google BigQuery.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Advance Deploy Node Configuration
* Node Properties
* Create Options
* Load Work Options
* Other Options

#### Work Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the work will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Work Advanced Deploy Options

You can create the node as:

* [Table](#work-advanced-deploy-create-as-table)
* [View](#work-advanced-deploy-create-as-view)

#### Work Advanced Deploy Create as Table

##### Work Advanced Deploy Create Options

| **Setting** | **Description** |
|---------|-------------|
| **Primary Key** | Toggle: True/False  Define primary key columns for documentation/metadata (Not enforced). For more info please refrer [documentation](https://docs.cloud.google.com/bigquery/docs/primary-foreign-keys#limitations) |
| **Enable Partitioning** | Toggle: True/False  **True**: Enables partitioning based on **Ingestion Time**, **Time-Unit Column**, or **Integer Range**.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/partitioned-tables#limitations).  *Note: Changing partitions drops and recreates the table.* |
| **Partition By** | **Dropdown**: Select the partitioning strategy. - **Ingestion Time**: Partitioning based on when data is loaded. - **Time-Unit Column**: Partitioning based on a specific DATE/TIMESTAMP column or expression. - **Integer Range**: Partitioning based on numeric ranges. |
| **Partition By Column** | **Column Selector**: Choose a specific column (DataType: DATE) to use for partitioning. *Used with "Time-Unit Column" strategy.* |
| **Time-Unit Expression** | **Text Box**: Provide a SQL expression for time partitioning. *Example*: `DATE_TRUNC(columnName, MONTH)` |
| **Integer Range Expression** | **Text Box**: Provide a SQL expression for integer range partitioning. *Example*: `RANGE_BUCKET(columnName, GENERATE_ARRAY(1, 100, 200))` |
| **Ingestion-time Expression** | **Text Box**: (Optional) Provide a custom expression for ingestion-time partitioning. *Example*: `DATE_TRUNC(_PARTITIONTIME, MONTH)` |
| **Partition Expiration Days** | **Text Box**: (Optional) Specify the number of days after which a partition should expire and be deleted. *Example*: `30` |
| **Enable Clustering** | **Toggle**: True/False  Enables or disables clustering for the table.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/clustered-tables#limitations) |
| **Cluster By** | **Tabular Input**: Select up to **4 columns** to cluster the table data. The order of columns determines the sort hierarchy. |
| **Table Expiration** | **Toggle**: True/False  Enables or disables the automatic expiration of the table. |
| **Expiration Type** | **Dropdown**: Select how the expiration is calculated. - **EXACT DATE/DATETIME**: The table will expire at a specific point in time. - **DAYS FROM NOW**: The table will expire after a set number of days from the deployment date. |
| **Expiration Value** | **Text Box**: Enter the value based on the selected Expiration Type. - For **EXACT DATE/DATETIME**, use format: `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (e.g., `2024-12-31`). - For **DAYS FROM NOW**, enter an integer (e.g., `30`). |
| **Default Rounding Mode** | **Dropdown**: (Optional) Specify the rounding behavior for numeric calculations. - `ROUND_HALF_AWAY_FROM_ZERO` - `ROUND_HALF_EVEN` |

##### Work Advanced Deploy Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Truncate Before** | Toggle: True/False**True**: Table is truncated before every load.**False**: Incremental load based on update strategy. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |

##### Work Advanced Deploy Other Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL / Post-SQL**| SQL to execute before or after the work load operation. |

##### Work Advanced Deploy Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Override SQL** | Toggle: True/False  Allows providing a custom SQL definition for the view. |
| **Multi Source** | Toggle: True/False**True**: Combines multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

### Work Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### Work Advanced Deploy Deployment

#### Work Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Work View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Work Advanced Deploy Redeployment

Once a Work table is initially deployed, subsequent configuration changes will result in either an in-place **`ALTER`** or a full **`DROP and RECREATE`** of the table, depending on the nature of the update (e.g., destructive changes like partitioning or clustering will trigger a recreation).

#### Altering the Work Tables

The following types of column or table modifications will result in an **`ALTER`** statement to update the table structure in the target environment, whether these changes are made individually or in combination:

*   **Primary Key Updates:** Adding/Updating/Modifying non-enforced primary key constraints.
*   **Table Metadata:** Rename or updating descriptions.
*   **Column Structure Changes:** 
    *   Adding new columns.
    *   Dropping existing columns.
    *   Renaming columns.
*   **Column Attribute Modifications:** Changing descriptions, data types or adjusting nullability constraints (e.g., `NULL` to `NOT NULL`).
*   **Configuration & Option Changes:**
    *   Updating **Table Expiration** or **Partition Expiration** settings.
    *   Adjusting the **Default Rounding Mode**.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **ALTER TABLE** | Alter table statement is executed to perform the alter operation |

#### Recreating the Work Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in recreating the work view.

#### Drop and Recreate Work View/Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Any materialization to Table** |  1. Drop `materialization`2. Create Work table |
| **Any materialization to View** |  1. Drop `materialization`2. Create Work view |

### Work Advanced Deploy Undeployment

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Work Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

-----

## Persistent Stage Advanced Deploy

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Advanced Deploy Node Configuration

The Persistent Stage node type has four configuration groups:

* Node Properties
* Create Options
* Load Persistent Stage Options
* Other Options

#### Persistent Stage Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the persistent stage will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Advanced Deploy Options

You can create the node as:

* [Table](#persistent-advanced-deploy-create-as-table)

#### Persistent Stage Advanced Deploy Create as Table

##### Persistent Stage Advance Deploy Create Options

| **Setting** | **Description** |
|---------|-------------|
| **Primary Key** | Toggle: True/False  Define primary key columns for documentation/metadata (Not enforced). For more info please refrer [documentation](https://docs.cloud.google.com/bigquery/docs/primary-foreign-keys#limitations) |
| **Enable Partitioning** | Toggle: True/False  **True**: Enables partitioning based on **Ingestion Time**, **Time-Unit Column**, or **Integer Range**.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/partitioned-tables#limitations).  *Note: Changing partitions drops and recreates the table.* |
| **Partition By** | **Dropdown**: Select the partitioning strategy. - **Ingestion Time**: Partitioning based on when data is loaded. - **Time-Unit Column**: Partitioning based on a specific DATE/TIMESTAMP column or expression. - **Integer Range**: Partitioning based on numeric ranges. |
| **Partition By Column** | **Column Selector**: Choose a specific column (DataType: DATE) to use for partitioning. *Used with "Time-Unit Column" strategy.* |
| **Time-Unit Expression** | **Text Box**: Provide a SQL expression for time partitioning. *Example*: `DATE_TRUNC(columnName, MONTH)` |
| **Integer Range Expression** | **Text Box**: Provide a SQL expression for integer range partitioning. *Example*: `RANGE_BUCKET(columnName, GENERATE_ARRAY(1, 100, 200))` |
| **Ingestion-time Expression** | **Text Box**: (Optional) Provide a custom expression for ingestion-time partitioning. *Example*: `DATE_TRUNC(_PARTITIONTIME, MONTH)` |
| **Partition Expiration Days** | **Text Box**: (Optional) Specify the number of days after which a partition should expire and be deleted. *Example*: `30` |
| **Enable Clustering** | **Toggle**: True/False  Enables or disables clustering for the table.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/clustered-tables#limitations) |
| **Cluster By** | **Tabular Input**: Select up to **4 columns** to cluster the table data. The order of columns determines the sort hierarchy. |
| **Table Expiration** | **Toggle**: True/False  Enables or disables the automatic expiration of the table. |
| **Expiration Type** | **Dropdown**: Select how the expiration is calculated. - **EXACT DATE/DATETIME**: The table will expire at a specific point in time. - **DAYS FROM NOW**: The table will expire after a set number of days from the deployment date. |
| **Expiration Value** | **Text Box**: Enter the value based on the selected Expiration Type. - For **EXACT DATE/DATETIME**, use format: `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (e.g., `2024-12-31`). - For **DAYS FROM NOW**, enter an integer (e.g., `30`). |
| **Default Rounding Mode** | **Dropdown**: (Optional) Specify the rounding behavior for numeric calculations. - `ROUND_HALF_AWAY_FROM_ZERO` - `ROUND_HALF_EVEN` |

##### Persistent Stage Advance Deploy Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Business key** | Required column for SCD Dimensions.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications. |
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. |
| **Enable SCD Type 2** | Toggle: True/False  **True**: Maintains historical versions of records using system start/end dates and version flags. |
| **Change Tracking Columns**(Visible when Last Modified Comparison is OFF) | **Checkbox List**: Provides a list of available target columns to define historical tracking behavior. - **SCD Type 2 (History):** Any column selected in this list will trigger the creation of a new record version when a change is detected. - **SCD Type 1 (Overwrite):** Columns that are **not** selected will follow SCD Type 1 logic, meaning changes to these columns will overwrite the existing current record without creating a new version. - **Default Logic:** If **no columns** are selected, the entire table is treated as **SCD Type 1**. |
| **Exclude Columns from Merge** | **Toggle**: True/False  Enables the ability to exclude specific columns from the `UPDATE` clause of the **`MERGE INTO`** statement. This is primarily used in **SCD Type 1** scenarios to ensure that certain columns remain unchanged after the initial record creation.  *Note: This option is only available when no Change Tracking columns are selected and Last Modified Comparison is disabled.* |
| **Exclude Merge List** | **Tabular Input**: A list of columns to be omitted from the update logic. - **Exclude Column Name**: Select the specific column(s) that should be ignored during the update phase of the merge. These columns will be populated during the initial **INSERT** but never modified during a **MERGE UPDATE**. |
| **Truncate Before** | Toggle: True/False**True**: Table is truncated before every load.**False**: Incremental load based on update strategy. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |

##### Persistent Stage Advance Deploy Other Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL / Post-SQL**| SQL to execute before or after the persistent stage load operation. |

#### Persistent Stage Advanced Deploy System Columns

These columns are automatically added to manage dimension logic:

| **Column** | **Description** |
|----------|-------------|
| **\&#x7B;\&#x7B;NODE_NAME\&#x7D;\&#x7D;_key** | The generated Surrogate Key for the dimension record. |
| **system_version** | Incremental version number for SCD Type 2 tracking. |
| **system_current_flag** | Indicates the active record ('Y'/'N'). |
| **system_start_date** | The timestamp when the record version became active. |
| **system_end_date** | The timestamp when the record version was superseded (Default: 2999-12-31). |
| **system_create_date** | Audit timestamp for when the row was first inserted. |
| **system_update_date** | Audit timestamp for the last modification. |

## BigQuery SCD Implementation Preferences

* **Strategy Selection:** Use the **MERGE** statement for creating Slowly Changing Dimensions (SCD Type 1 or 2).
* **Performance:** While traditional **INSERT/UPDATE** DML can be used for smaller batches, **MERGE** is recommended for superior execution performance on large datasets.
* **Optimization:** For tables exceeding **10 million rows**, ensure the table is **partitioned** and **clustered** to minimize the bytes scanned during the MERGE operation.

### Persistent Stage Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### Persistent Stage Advanced Deploy Deployment

When deployed for the first time into an environment the persistent stage node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Persistent Stage Advanced Deploy Redeployment

Once a Persistent Stage table is initially deployed, subsequent configuration changes will result in either an in-place **`ALTER`** or a full **`DROP and RECREATE`** of the table, depending on the nature of the update (e.g., destructive changes like partitioning or clustering will trigger a recreation).

#### Altering the Persistent Stage Tables

The following types of column or table modifications will result in an **`ALTER`** statement to update the table structure in the target environment, whether these changes are made individually or in combination:

*   **Primary Key Updates:** Adding/Updating/Modifying non-enforced primary key constraints.
*   **Table Metadata:** Rename or updating descriptions.
*   **Column Structure Changes:** 
    *   Adding new columns.
    *   Dropping existing columns.
    *   Renaming columns.
*   **Column Attribute Modifications:** Changing descriptions, data types or adjusting nullability constraints (e.g., `NULL` to `NOT NULL`).
*   **Configuration & Option Changes:**
    *   Updating **Table Expiration** or **Partition Expiration** settings.
    *   Adjusting the **Default Rounding Mode**.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **ALTER TABLE** | Alter table statement is executed to perform the alter operation |

#### Drop and Recreate Persistent Stage Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Any materialization to Table** |  1. Drop `materialization`2. Create Persistent table |

### Persistent Stage Advanced Deploy Undeployment

If a Persistent Stage Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table** | Removes the table from the environment |

-----

## Dimension Advanced Deploy

The Coalesce Dimension Advanced Deploy node is designed to manage the lifecycle of dimension tables, supporting both Slowly Changing Dimensions (SCD) Type 1 and Type 2. It provides advanced controls for partitioning, clustering, and automated "Zero Key" (Unknown member) record injection.

This node ensures historical integrity through system-managed versioning columns while offering flexible loading strategies like MERGE and INSERT/UPDATE.

### Dimension Advanced Deploy Node Configuration

The Dimension node type has four configuration groups:

* Node Properties
* Create Options
* Load Dimension Options
* Zero Key Record Options
* Other Options

#### Dimension Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Dimension will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Advanced Deploy Options

You can create the node as:

* [Table](#dimension-advanced-deploy-create-as-table)
* [View](#dimension-advanced-deploy-create-as-view)

#### Dimension Advanced Deploy Create as Table

#### Dimension Advanced Deploy Create Options

| **Setting** | **Description** |
|---------|-------------|
| **Primary Key** | Toggle: True/False  Define primary key columns for documentation/metadata (Not enforced). For more info please refrer [documentation](https://docs.cloud.google.com/bigquery/docs/primary-foreign-keys#limitations) |
| **Enable Partitioning** | Toggle: True/False  **True**: Enables partitioning based on **Ingestion Time**, **Time-Unit Column**, or **Integer Range**.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/partitioned-tables#limitations).  *Note: Changing partitions drops and recreates the table.* |
| **Partition By** | **Dropdown**: Select the partitioning strategy. - **Ingestion Time**: Partitioning based on when data is loaded. - **Time-Unit Column**: Partitioning based on a specific DATE/TIMESTAMP column or expression. - **Integer Range**: Partitioning based on numeric ranges. |
| **Partition By Column** | **Column Selector**: Choose a specific column (DataType: DATE) to use for partitioning. *Used with "Time-Unit Column" strategy.* |
| **Time-Unit Expression** | **Text Box**: Provide a SQL expression for time partitioning. *Example*: `DATE_TRUNC(columnName, MONTH)` |
| **Integer Range Expression** | **Text Box**: Provide a SQL expression for integer range partitioning. *Example*: `RANGE_BUCKET(columnName, GENERATE_ARRAY(1, 100, 200))` |
| **Ingestion-time Expression** | **Text Box**: (Optional) Provide a custom expression for ingestion-time partitioning. *Example*: `DATE_TRUNC(_PARTITIONTIME, MONTH)` |
| **Partition Expiration Days** | **Text Box**: (Optional) Specify the number of days after which a partition should expire and be deleted. *Example*: `30` |
| **Enable Clustering** | **Toggle**: True/False  Enables or disables clustering for the table.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/clustered-tables#limitations) |
| **Cluster By** | **Tabular Input**: Select up to **4 columns** to cluster the table data. The order of columns determines the sort hierarchy. |
| **Table Expiration** | **Toggle**: True/False  Enables or disables the automatic expiration of the table. |
| **Expiration Type** | **Dropdown**: Select how the expiration is calculated. - **EXACT DATE/DATETIME**: The table will expire at a specific point in time. - **DAYS FROM NOW**: The table will expire after a set number of days from the deployment date. |
| **Expiration Value** | **Text Box**: Enter the value based on the selected Expiration Type. - For **EXACT DATE/DATETIME**, use format: `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (e.g., `2024-12-31`). - For **DAYS FROM NOW**, enter an integer (e.g., `30`). |
| **Default Rounding Mode** | **Dropdown**: (Optional) Specify the rounding behavior for numeric calculations. - `ROUND_HALF_AWAY_FROM_ZERO` - `ROUND_HALF_EVEN` |

#### Dimension Advanced Deploy Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Update Strategy** | Choose the SQL pattern for loading data: - **MERGE**: Utilizes a single **`MERGE INTO`** statement to synchronize the source and target. - **INSERT/UPDATE**: Utilizes a multi-step transactional approach (e.g., **`BEGIN TRANSACTION`**...**`COMMIT`**) to execute separate update and insert operations. |
| **Unmatched Record Strategy** | Options for `NO DELETE`, `SOFT DELETE` (Flagging), or `HARD DELETE` (Physical removal) for records no longer in source. |
| **Business key** | Required column for SCD Dimensions.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications. |
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. |
| **Enable SCD Type 2** | Toggle: True/False  **True**: Maintains historical versions of records using system start/end dates and version flags. |
| **Change Tracking Columns**(Visible when Last Modified Comparison is OFF) | **Checkbox List**: Provides a list of available target columns to define historical tracking behavior. - **SCD Type 2 (History):** Any column selected in this list will trigger the creation of a new record version when a change is detected. - **SCD Type 1 (Overwrite):** Columns that are **not** selected will follow SCD Type 1 logic, meaning changes to these columns will overwrite the existing current record without creating a new version. - **Default Logic:** If **no columns** are selected, the entire table is treated as **SCD Type 1**. |
| **Exclude Columns from Merge** | **Toggle**: True/False  Enables the ability to exclude specific columns from the `UPDATE` clause of the **`MERGE INTO`** statement. This is primarily used in **SCD Type 1** scenarios to ensure that certain columns remain unchanged after the initial record creation.  *Note: This option is only available when no Change Tracking columns are selected and Last Modified Comparison is disabled.* |
| **Exclude Merge List** | **Tabular Input**: A list of columns to be omitted from the update logic. - **Exclude Column Name**: Select the specific column(s) that should be ignored during the update phase of the merge. These columns will be populated during the initial **INSERT** but never modified during a **MERGE UPDATE**. |
| **Insert Zero Key** | Toggle: True/False  Automatically inserts a placeholder record (e.g., ID 0, "UNKNOWN") to handle null foreign keys in fact tables. |
| **Default Surrogate Key Value** | The numeric or string value to be assigned to the Surrogate Key column for the zero key record. **Default**: `0` |
| **Default String Value** | The default value used for all string/text columns in the zero key record. **Default**: `UNKNOWN` |
| **Default Date Value** | The default value used for all Date columns (Format: DD-MM-YYYY). **Default**: `01-01-1900` |
| **Default Timestamp Value** | The default value used for all Timestamp columns (Format: YYYY-MM-DD HH24:MI:SS). **Default**: `1900-01-01 00:00:00` |
| **Default Boolean Value** | The default value used for all Boolean columns. **Options**: `TRUE`, `FALSE` |
| **Advanced Zero Key Record Options** | **Toggle**: True/False  When enabled, allows for granular control over specific columns via the Custom Zero Key Values table. |
| **Custom Zero Key Values** | **Tabular Input**: Allows you to override the global defaults for specific columns. - **Column**: Select a specific column from the node. - **Default Value**: Provide a specific value for that column for the zero key record. |
| **Truncate Before** | Toggle: True/False**True**: Table is truncated before every load.**False**: Incremental load based on update strategy. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |

#### Dimension Advanced Deploy Other Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL / Post-SQL**| SQL to execute before or after the dimension load operation. |

##### Dimension Advanced Deploy Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Override SQL** | Toggle: True/False  Allows providing a custom SQL definition for the view. |
| **Multi Source** | Toggle: True/False**True**: Combines multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Dimension Advanced Deploy System Columns

These columns are automatically added to manage dimension logic:

| **Column** | **Description** |
|----------|-------------|
| **\&#x7B;\&#x7B;NODE_NAME\&#x7D;\&#x7D;_key** | The generated Surrogate Key for the dimension record. |
| **system_version** | Incremental version number for SCD Type 2 tracking. |
| **system_current_flag** | Indicates the active record ('Y'/'N'). |
| **system_start_date** | The timestamp when the record version became active. |
| **system_end_date** | The timestamp when the record version was superseded (Default: 2999-12-31). |
| **system_create_date** | Audit timestamp for when the row was first inserted. |
| **system_update_date** | Audit timestamp for the last modification. |

## BigQuery SCD Implementation Preferences

* **Strategy Selection:** Use the **MERGE** statement for creating Slowly Changing Dimensions (SCD Type 1 or 2) with NO DELETE/SOFT DELETE/HARD DELETE
* **Performance:** While traditional **INSERT/UPDATE** DML can be used for smaller batches, **MERGE** is recommended for superior execution performance on large datasets.
* **Optimization:** For tables exceeding **10 million rows**, ensure the table is **partitioned** and **clustered** to minimize the bytes scanned during the MERGE operation.

### Dimension Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### Dimension Advanced Deploy Deployment

#### Dimension Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Dimension View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Dimension Advanced Deploy Redeployment

Once a Dimension table is initially deployed, subsequent configuration changes will result in either an in-place **`ALTER`** or a full **`DROP and RECREATE`** of the table, depending on the nature of the update (e.g., destructive changes like partitioning or clustering will trigger a recreation).

#### Altering the Dimension Tables

The following types of column or table modifications will result in an **`ALTER`** statement to update the table structure in the target environment, whether these changes are made individually or in combination:

*   **Primary Key Updates:** Adding/Updating/Modifying non-enforced primary key constraints.
*   **Table Metadata:** Rename or updating descriptions.
*   **Column Structure Changes:** 
    *   Adding new columns.
    *   Dropping existing columns.
    *   Renaming columns.
*   **Column Attribute Modifications:** Changing descriptions, data types or adjusting nullability constraints (e.g., `NULL` to `NOT NULL`).
*   **Configuration & Option Changes:**
    *   Updating **Table Expiration** or **Partition Expiration** settings.
    *   Adjusting the **Default Rounding Mode**.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **ALTER TABLE** | Alter table statement is executed to perform the alter operation |

#### Recreating the Dimension Views

The subsequent deployment of Dimension node of materialization type view with changes in view definition, adding table description or renaming view results in recreating the dimension view.

#### Drop and Recreate Dimension View/Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Any materialization to Table** |  1. Drop `materialization`2. Create Dimension table |
| **Any materialization to View** |  1. Drop `materialization`2. Create Dimension view |

### Dimension Advanced Deploy Undeployment

If a Dimension Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

-----

## Fact Advanced Deploy

The Coalesce Fact Advanced Deploy node is designed to manage the lifecycle of fact tables, supporting both Slowly Changing Fact (SCD) Type 1 and Type 2. It provides advanced controls for partitioning, clustering.

This node ensures historical integrity through system-managed versioning columns while offering flexible loading strategies like MERGE.

### Fact Advanced Deploy Node Configuration

The Fact node type has four configuration groups:

* Node Properties
* Create Options
* Load Fact Options
* Other Options

  

#### Fact Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Fact will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Fact Advanced Deploy Options

You can create the node as:

* [Table](#fact-advanced-deploy-create-as-table)
* [View](#fact-advanced-deploy-create-as-view)

#### Fact Advanced Deploy Create as Table

#### Fact Advanced Deploy Create Options

| **Setting** | **Description** |
|---------|-------------|
| **Primary Key** | Toggle: True/False  Define primary key columns for documentation/metadata (Not enforced). For more info please refrer [documentation](https://docs.cloud.google.com/bigquery/docs/primary-foreign-keys#limitations) |
| **Enable Partitioning** | Toggle: True/False  **True**: Enables partitioning based on **Ingestion Time**, **Time-Unit Column**, or **Integer Range**.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/partitioned-tables#limitations).  *Note: Changing partitions drops and recreates the table.* |
| **Partition By** | **Dropdown**: Select the partitioning strategy. - **Ingestion Time**: Partitioning based on when data is loaded. - **Time-Unit Column**: Partitioning based on a specific DATE/TIMESTAMP column or expression. - **Integer Range**: Partitioning based on numeric ranges. |
| **Partition By Column** | **Column Selector**: Choose a specific column (DataType: DATE) to use for partitioning. *Used with "Time-Unit Column" strategy.* |
| **Time-Unit Expression** | **Text Box**: Provide a SQL expression for time partitioning. *Example*: `DATE_TRUNC(columnName, MONTH)` |
| **Integer Range Expression** | **Text Box**: Provide a SQL expression for integer range partitioning. *Example*: `RANGE_BUCKET(columnName, GENERATE_ARRAY(1, 100, 200))` |
| **Ingestion-time Expression** | **Text Box**: (Optional) Provide a custom expression for ingestion-time partitioning. *Example*: `DATE_TRUNC(_PARTITIONTIME, MONTH)` |
| **Partition Expiration Days** | **Text Box**: (Optional) Specify the number of days after which a partition should expire and be deleted. *Example*: `30` |
| **Enable Clustering** | **Toggle**: True/False  Enables or disables clustering for the table.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/clustered-tables#limitations) |
| **Cluster By** | **Tabular Input**: Select up to **4 columns** to cluster the table data. The order of columns determines the sort hierarchy. |
| **Table Expiration** | **Toggle**: True/False  Enables or disables the automatic expiration of the table. |
| **Expiration Type** | **Dropdown**: Select how the expiration is calculated. - **EXACT DATE/DATETIME**: The table will expire at a specific point in time. - **DAYS FROM NOW**: The table will expire after a set number of days from the deployment date. |
| **Expiration Value** | **Text Box**: Enter the value based on the selected Expiration Type. - For **EXACT DATE/DATETIME**, use format: `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (e.g., `2024-12-31`). - For **DAYS FROM NOW**, enter an integer (e.g., `30`). |
| **Default Rounding Mode** | **Dropdown**: (Optional) Specify the rounding behavior for numeric calculations. - `ROUND_HALF_AWAY_FROM_ZERO` - `ROUND_HALF_EVEN` |

#### Fact Advanced Deploy Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Unmatched Record Strategy** | Options for `NO DELETE` or `HARD DELETE` (Physical removal) for records no longer in source. *Available only when atleast one Business Key is chosen*|
| **Business key** | Required column for SCD Fact.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications.*Available only when atleast one Business Key is chosen*|
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. *Available only when atleast one Business Key is chosen*|
| **Enable SCD Type 2** | Toggle: True/False  **True**: Maintains historical versions of records using system start/end dates and version flags. *Available only when atleast one Business Key is chosen*|
| **Exclude Columns from Merge** | **Toggle**: True/False  Enables the ability to exclude specific columns from the `UPDATE` clause of the **`MERGE INTO`** statement. This is primarily used in **SCD Type 1** scenarios to ensure that certain columns remain unchanged after the initial record creation. *Available only when atleast one Business Key is chosen*|
| **Exclude Merge List** | **Tabular Input**: A list of columns to be omitted from the update logic. - **Exclude Column Name**: Select the specific column(s) that should be ignored during the update phase of the merge. These columns will be populated during the initial **INSERT** but never modified during a **MERGE UPDATE**. *Available only when atleast one Business Key is chosen* |
| **Truncate Before** | Toggle: True/False**True**: Table is truncated before every load.**False**: Incremental load based on update strategy. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |

#### Fact Advanced Deploy Other Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL / Post-SQL**| SQL to execute before or after the Fact load operation. |

##### Fact Advanced Deploy Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Override SQL** | Toggle: True/False  Allows providing a custom SQL definition for the view. |
| **Multi Source** | Toggle: True/False**True**: Combines multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Fact Advanced Deploy System Columns

These columns are automatically added to manage fact logic:

| **Column** | **Description** |
|----------|-------------|
| **system_create_date** | Audit timestamp for when the row was first inserted. |
| **system_update_date** | Audit timestamp for the last modification. |

## BigQuery SCD Implementation Preferences

* **Optimization:** For tables exceeding **10 million rows**, ensure the table is **partitioned** and **clustered** to minimize the bytes scanned during the MERGE operation.

### Fact Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### Fact Advanced Deploy Deployment

#### Fact Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Fact View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Fact Advanced Deploy Redeployment

Once a Fact table is initially deployed, subsequent configuration changes will result in either an in-place **`ALTER`** or a full **`DROP and RECREATE`** of the table, depending on the nature of the update (e.g., destructive changes like partitioning or clustering will trigger a recreation).

#### Altering the Fact Tables

The following types of column or table modifications will result in an **`ALTER`** statement to update the table structure in the target environment, whether these changes are made individually or in combination:

*   **Primary Key Updates:** Adding/Updating/Modifying non-enforced primary key constraints.
*   **Table Metadata:** Rename or updating descriptions.
*   **Column Structure Changes:** 
    *   Adding new columns.
    *   Dropping existing columns.
    *   Renaming columns.
*   **Column Attribute Modifications:** Changing descriptions, data types or adjusting nullability constraints (e.g., `NULL` to `NOT NULL`).
*   **Configuration & Option Changes:**
    *   Updating **Table Expiration** or **Partition Expiration** settings.
    *   Adjusting the **Default Rounding Mode**.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **ALTER TABLE** | Alter table statement is executed to perform the alter operation |

#### Recreating the Fact Views

The subsequent deployment of Fact node of materialization type view with changes in view definition, adding table description or renaming view results in recreating the Fact view.

#### Drop and Recreate Fact View/Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Any materialization to Table** |  1. Drop `materialization`2. Create Fact table |
| **Any materialization to View** |  1. Drop `materialization`2. Create Fact view |

### Fact Advanced Deploy Undeployment

If a Fact Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

-----

## Factless Fact Advanced Deploy

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Google BigQuery.

A factless fact table is used to record events or situations that have no measures.

### Factless Fact Advanced Deploy Node Configuration

The Factless Fact node type has four configuration groups:

* Node Properties
* Create Options
* Load Fact Options
* Other Options

#### Factless Fact Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Fact will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Factless Fact Advanced Deploy Options

You can create the node as:

* [Table](#fact-advanced-deploy-create-as-table)

#### Factless Fact Advanced Deploy Create as Table

##### Factless Fact Advanced Deploy Create Options

| **Setting** | **Description** |
|---------|-------------|
| **Primary Key** | Toggle: True/False  Define primary key columns for documentation/metadata (Not enforced). For more info please refrer [documentation](https://docs.cloud.google.com/bigquery/docs/primary-foreign-keys#limitations) |
| **Enable Partitioning** | Toggle: True/False  **True**: Enables partitioning based on **Ingestion Time**, **Time-Unit Column**, or **Integer Range**.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/partitioned-tables#limitations).  *Note: Changing partitions drops and recreates the table.* |
| **Partition By** | **Dropdown**: Select the partitioning strategy. - **Ingestion Time**: Partitioning based on when data is loaded. - **Time-Unit Column**: Partitioning based on a specific DATE/TIMESTAMP column or expression. - **Integer Range**: Partitioning based on numeric ranges. |
| **Partition By Column** | **Column Selector**: Choose a specific column (DataType: DATE) to use for partitioning. *Used with "Time-Unit Column" strategy.* |
| **Time-Unit Expression** | **Text Box**: Provide a SQL expression for time partitioning. *Example*: `DATE_TRUNC(columnName, MONTH)` |
| **Integer Range Expression** | **Text Box**: Provide a SQL expression for integer range partitioning. *Example*: `RANGE_BUCKET(columnName, GENERATE_ARRAY(1, 100, 200))` |
| **Ingestion-time Expression** | **Text Box**: (Optional) Provide a custom expression for ingestion-time partitioning. *Example*: `DATE_TRUNC(_PARTITIONTIME, MONTH)` |
| **Partition Expiration Days** | **Text Box**: (Optional) Specify the number of days after which a partition should expire and be deleted. *Example*: `30` |
| **Enable Clustering** | **Toggle**: True/False  Enables or disables clustering for the table.For more info please refer [documentation](https://docs.cloud.google.com/bigquery/docs/clustered-tables#limitations) |
| **Cluster By** | **Tabular Input**: Select up to **4 columns** to cluster the table data. The order of columns determines the sort hierarchy. |
| **Table Expiration** | **Toggle**: True/False  Enables or disables the automatic expiration of the table. |
| **Expiration Type** | **Dropdown**: Select how the expiration is calculated. - **EXACT DATE/DATETIME**: The table will expire at a specific point in time. - **DAYS FROM NOW**: The table will expire after a set number of days from the deployment date. |
| **Expiration Value** | **Text Box**: Enter the value based on the selected Expiration Type. - For **EXACT DATE/DATETIME**, use format: `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` (e.g., `2024-12-31`). - For **DAYS FROM NOW**, enter an integer (e.g., `30`). |
| **Default Rounding Mode** | **Dropdown**: (Optional) Specify the rounding behavior for numeric calculations. - `ROUND_HALF_AWAY_FROM_ZERO` - `ROUND_HALF_EVEN` |

##### Factless Fact Advanced Deploy Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources using `UNION ALL` or `UNION DISTINCT`. |
| **Truncate Before** | Toggle: True/False**True**: Table is truncated before every load.**False**: Incremental load based on update strategy. |
| **Distinct** | Toggle: True/False**True**: Applies a DISTINCT clause to the data. |
| **Group By All** | Toggle: True/False**True**: Applies a GROUP BY ALL clause on columns. |

##### Factless Fact Advanced Deploy Other Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL / Post-SQL**| SQL to execute before or after the Fact load operation. |

#### Factless Fact Advanced Deploy System Columns

These columns are automatically added to manage fact logic:

| **Column** | **Description** |
|----------|-------------|
| **system_create_date** | Audit timestamp for when the row was first inserted. |
| **system_update_date** | Audit timestamp for the last modification. |

## BigQuery SCD Implementation Preferences

* **Optimization:** For tables exceeding **10 million rows**, ensure the table is **partitioned** and **clustered** to minimize the bytes scanned during the MERGE operation.

### Factless Fact Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### Factless Fact Advanced Deploy Deployment

#### Factless Fact Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Factless Fact Advanced Deploy Redeployment

Once a Factless Fact table is initially deployed, subsequent configuration changes will result in either an in-place **`ALTER`** or a full **`DROP and RECREATE`** of the table, depending on the nature of the update (e.g., destructive changes like partitioning or clustering will trigger a recreation).

#### Altering the Factless Fact Tables

The following types of column or table modifications will result in an **`ALTER`** statement to update the table structure in the target environment, whether these changes are made individually or in combination:

*   **Primary Key Updates:** Adding/Updating/Modifying non-enforced primary key constraints.
*   **Table Metadata:** Rename or updating descriptions.
*   **Column Structure Changes:** 
    *   Adding new columns.
    *   Dropping existing columns.
    *   Renaming columns.
*   **Column Attribute Modifications:** Changing descriptions, data types or adjusting nullability constraints (e.g., `NULL` to `NOT NULL`).
*   **Configuration & Option Changes:**
    *   Updating **Table Expiration** or **Partition Expiration** settings.
    *   Adjusting the **Default Rounding Mode**.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **ALTER TABLE** | Alter table statement is executed to perform the alter operation |

#### Drop and Recreate Factless Fact Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Any materialization to Table** |  1. Drop `materialization`2. Create Fact table |

### Factless Fact Advanced Deploy Undeployment

If a Fact Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table** | Removes the table from the environment |

---

## Code

### Work Advance Deploy Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/WorkAdvanceDeploy-170/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/WorkAdvanceDeploy-170/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/WorkAdvanceDeploy-170/run.sql.j2)

### Persistent Stage Advance Deploy Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/PersistentStageAdvancedDeploy-171/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/PersistentStage-160/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/PersistentStageAdvancedDeploy-171/run.sql.j2)

### Dimension Advance Deploy Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/run.sql.j2)

### Fact Advance Deploy Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/run.sql.j2)

### Factless Fact Advance Deploy Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-177/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-177/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-177/run.sql.j2)

[Macros](https://github.com/coalesceio/big-query-base-node-types-advanced-deploy/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.0 | March 30, 2026 | Moved UUID Generation logic to Node definition for the column visibility and other minor fixes |
| 1.0.0 | February 13, 2026 | Initial Version of Base Node Types - Advanced |

---

## BigQuery Base Node Types

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/bigquery/bigquery-base-node-types</code> <CopyButton textToCopy='@coalesce/bigquery/bigquery-base-node-types' />
Supported Platform: 
Latest Version: 1.1.0 (April 03, 2026)

data warehousedimensiondimension tabledimensional modelingeltfactfact tablepersistent stagestagingstar schematransformationwork

## Overview
Coalesce base Node Types to construct facts and dimensions.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/bigquery/bigquery-base-node-types</code> <CopyButton textToCopy='@coalesce/bigquery/bigquery-base-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Coalesce-BigQuery - Base Node Types

## Brief Summary
 
These Node Types transforms raw data into a Single Source of Truth, driving strategic value through four specialized architectural layers:
 
*   **Data Preparation & History (Work & Persistent Stage):**
    All data begins in the **Work** and **Persistent Stage** areas. Think of this as our quality control and storage hub. Here, raw data is cleaned and organized.
 
*   **Business Context & Events (Dimension):**
    To make sense of numbers, we need context. **Dimension** nodes provide the "who, what, where, and why" (e.g., Customer Details, Product Types, Store Locations). More importantly, we use these stages to keep a "historical memory" of the business, tracking how information—like a customer’s address or a product’s category—changes over time, ensuring we never lose sight of our past performance.
 
*   **Performance Metrics (Fact):**
    The **Fact** nodes are the heartbeat of our reporting. These store the quantitative "how much" of the business—such as total revenue, costs, and profit margins. By combining these facts with our Dimensions, leadership can see exactly how specific regions, products, or time periods are performing.
 
*   **Simplified Access (View):**
    Finally, **Views** act as a user-friendly window into this complex system. Instead of navigating technical tables, business users interact with Views that have been tailored for specific needs—providing secure, easy-to-read, and high-speed access to the exact data required for day-to-day decision-making.

---

## Nodetypes Config Matrix

| Category | Feature                         | Dim | Fact | View | Work | PStage |
|----------|----------------------------------|-----|------|----------|------|--------|
| Create   | Create As Table                  | ✅  | ✅   | ⬜       | ✅   | ✅     |
| Create   | Create As View                   | ✅  | ✅   | ⬜       | ✅   | ⬜     |
| Load     | MultiSource                      | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Business Key                     | ✅  | ✅   | ⬜       | ⬜   | ✅     |
| Load     | Last Modified Comparison         | ✅  | ✅   | ⬜       | ⬜   | ✅     |
| Load     | Change Tracking                  | ✅  | ⬜   | ⬜       | ⬜   | ✅     |
| Load     | Truncate Before                  | ✅  | ✅   | ⬜       | ✅   | ✅     |
| Load     | Distinct                         | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Group By All                     | ✅  | ✅   | ✅       | ✅   | ✅     |
| Load     | Methods                          | MERGE | MERGEINSERT | ⬜ |INSERT | MERGEINSERT |
| Others   | Enable Tests                     | ✅  | ✅   | ⬜       | ✅   | ✅     |
| Others   | Pre-SQL                          | ✅  | ✅   | ⬜       | ✅   | ✅     |
| Others   | Post-SQL                         | ✅  | ✅   | ⬜       | ✅   | ✅     |

---

The Coalesce Base Node Types Package includes:

* [Work](#work)
* [Persistent Stage](#persistent-stage)
* [Dimension](#dimension)
* [Fact](#fact)
* [View](#view)
* [Code](#code)

---

## Work

The Coalesce work node is a versatile node that allows you to develop and deploy a Work table/view in Google BigQuery.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Node Configuration

The Work node type has two configuration groups:

* [Node Properties](#work-node-properties)
* [Options](#work-options)

  

  #### Work Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Work Options

You can create the node as:

* [Table](#work-options-table)
* [View](#work-options-view)

##### Work Options Table

| **Property** | **Description** |
|---------|-------------|
| **Multi Source** |Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination-**False**: Single source node or multiple sources combined using a join |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be truncated before data load. **True**: Truncate table stage gets executed**False**: Table is appended with data load |
| **Enable tests** | Toggle: True/FalseDetermines if column/node data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Work Options View

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the `Create SQL` space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Work Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group By Clauses**
>
> You should specify group by clause in this space if you are not opting for the group by all provided in OPTIONS config.

### Work Deployment

#### Work Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Work View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Work Redeployment

After the WORK node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.

#### Altering the Work Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Metadata Update for the Work Tables

If any of the following change are detected, metadata update stage executes.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct or group by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Making metadata updates** | Refreshes metadata |

#### Recreating the Work Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Dropping existing view |
| **Create View** | Creates new view with updated definition |

### Work Undeployment

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Drops the existing Work Table from target environment |

If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Work view from target environment |

---

## Persistent Stage 

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Node Configuration

The Persistent node type has two configuration groups:

* [Node Properties](#persistent-stage-node-properties)
* [Options](#persistent-stage-options)

  

#### Persistent Stage Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the PStage will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Options

| **Option** | **Description** |
|---------|-------------|
| **Create As** | Table is the only option at this time |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2. |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications. |
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. |
| **Enable SCD Type 2** | Toggle: True/False  **True**: Maintains historical versions of records using system start/end dates and version flags. |
| **Change tracking** | Checkbox List: Provides a list of available target columns to define historical tracking behavior. - SCD Type 2 (History): Any column selected in this list will trigger the creation of a new record version when a change is detected.- SCD Type 1 (Overwrite): Columns that are not selected will follow SCD Type 1 logic, meaning changes to these columns will overwrite the existing current record without creating a new version.- Default Logic: If no columns are selected, the entire table is treated as SCD Type 1. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is appended with data load |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Persistent Stage Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group By Clause**
>
> You should specify group by clause in this space if you are not opting for the group by all provided in OPTIONS config.

### Persistent Stage Deployment

#### Persistent Stage Initial Deployment

When deployed for the first time into an environment the Persistent node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Table** | This will execute a `CREATE OR REPLACE` statement and create a table in the target environment |

#### Persistent Stage Redeployment

After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

#### Altering the Persistent Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Metadata Update for the Persistent Tables

If any of the following change are detected, metadata update stage executes.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct or group by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Making metadata updates** | Refreshes metadata |

### Persistent Stage Undeployment

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Drops table |
| **Drop View** | Drops view |

---

## Dimension

The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Google BigQuery.

A dimension table or dimension entity is a table or entity in a star, snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

### Dimension Node Configuration

The Dimension node type has two configuration groups:

* [Node Properties](#dimension-node-properties)
* [Options](#dimension-options)

  

#### Dimension Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Dimension will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Options

##### Dimension Table Options

| **Options** | **Description** |
|---------|-------------|
| **Create As** | Table or View |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications. |
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. |
| **Enable SCD Type 2** | Toggle: True/False  **True**: Maintains historical versions of records using system start/end dates and version flags. |
| **Change tracking** | Checkbox List: Provides a list of available target columns to define historical tracking behavior. - SCD Type 2 (History): Any column selected in this list will trigger the creation of a new record version when a change is detected.- SCD Type 1 (Overwrite): Columns that are not selected will follow SCD Type 1 logic, meaning changes to these columns will overwrite the existing current record without creating a new version.- Default Logic: If no columns are selected, the entire table is treated as SCD Type 1. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is appended with data load |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the `Create SQL` space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group By Clause**
>
> You should specify group by clause in this space if you are not opting for the group by all provided in OPTIONS config.

### Dimension Deployment

#### Dimension Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table will execute the Create Dimension Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Table** | This will execute a `CREATE OR REPLACE` statement and create a table in the target environment |
| **Create Dimension View** | This will execute a `CREATE OR REPLACE` statement and create a view in the target environment |

#### Dimension Redeployment

After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

#### Altering the Dimension Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Metadata Update for the Dimension Tables

If any of the following change are detected, metadata update stage executes.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct or group by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Making metadata updates** | Refreshes metadata |

#### Recreating the Dimension Views

Any of the following changes to views will result in deleting and recreating the Dimension view.

* View defintion
* Adding table description
* Renaming view results

### Dimension Undeployment

If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Target table in Google BigQuery is dropped |

If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension View in the target environment will be dropped.

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Dimension view from target environment. |

---

## Fact

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Google BigQuery.

A fact table or a fact entity is a table or entity in a star or snowflake schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

### Fact Node Configuration

The Fact node has two configuration groups:

* [Node Properties](#fact-node-properties)
* [Options](#fact-options)

#### Fact Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Fact will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Fact Options

| **Options** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for Fact table creation.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **Toggle**: True/False - **True**: Enables high-performance Change Data Capture (CDC) by comparing a specific source timestamp or numeric column to identify records that have changed since the last load. - **False**: Performs standard CDC by comparing data values across all designated Change Tracking columns to detect modifications. |
| **Treat NULL as Current Timestamp**(For TIMESTAMP Columns) | **Toggle**: True/False - **True**: Source records with a `NULL` value in the comparison column are assigned the current system timestamp. This ensures that records with missing modification metadata are treated as "new" and are updated in the target table. - **False**: `NULL` values are handled per standard SQL comparison rules, which may result in these records being ignored during incremental loads. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is appended with data load |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

#### Fact View

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the `Create SQL` space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Enable tests** | Toggle: True/FalseDetermines if node/columns data quality tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.

> 📘 **Specify Group By Clause**
>
> You should specify group by clause in this space if you are not opting for the group by all provided in OPTIONS config.

### Fact Deployment

#### Fact Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table will execute the Create Fact Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a `CREATE OR REPLACE` statement and create a table in the target environment |
| **Create Fact View** | This will execute a `CREATE OR REPLACE` statement and create a view in the target environment |

#### Fact Redeployment

After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Metadata Update for the Fact Tables

If any of the following change are detected, metadata update stage executes.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct or group by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Making metadata updates** | Refreshes metadata |

#### Recreating the Fact Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Fact Undeployment

If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Target table in Google BigQuery is dropped |

---

## View 

The Coalesce View UDN is a versatile node that allows you to develop and deploy a View in Google BigQuery.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.

### View Node Configuration

The View node type has two configuration groups:

* [Node Properties](#view-node-properties)
* [Options](#view-options)

#### View Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the View will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the `Create SQL` space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- `UNION DISTINCT`: Combines sources with duplicate elimination- `UNION ALL`: Combines sources without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: Group By All is invisible. `DISTINCT` data is chosen for processing**False**: Group By All is visible |
| **Group By All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

> 📘 **Specify Group By Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### View Deployment

#### View Initial Deployment

When deployed for the first time into an environment the View node will execute the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a View in the target environment |

#### View Redeployment

The subsequent deployment of View node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

#### View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes the view from the environment |

## Code

### Work Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Work-159/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Work-159/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Work-159/run.sql.j2)

### Persistent Stage Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/PersistentStage-160/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/PersistentStage-160/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/PersistentStage-160/run.sql.j2)

### Dimension Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Dimension-161/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Dimension-161/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Dimension-161/run.sql.j2)

### Fact Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Fact-162/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Fact-162/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/Fact-162/run.sql.j2)

### View Code

* [Node definition](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/View-168/definition.yml)
* [Create Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/View-168/create.sql.j2)
* [Run Template](https://github.com/coalesceio/big-query-base-node-types/blob/main/nodeTypes/View-168/run.sql.j2)

[Macros](https://github.com/coalesceio/big-query-base-node-types/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.0 | April 03, 2026 | Added Last Modified Comparison option to MERGE logic |
| 1.0.0 | January 28, 2026 | BigQuery Base node types - Initial Release |

---

## Base Node Types

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/base-node-types</code> <CopyButton textToCopy='@coalesce/databricks/base-node-types' />
Supported Platform: 
Latest Version: 1.1.1 (April 16, 2026)

delta lakedimension tabledimensional modelingfact tablelakehousestagingtransformation

## Overview
Coalesce base Node Types to construct facts and dimensions.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/base-node-types</code> <CopyButton textToCopy='@coalesce/databricks/base-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Databricks - Base Node Types  – Brief Summary

- **Fact nodes**  
  Represent measurable business events such as sales, transactions, or usage. These are the core tables used for reporting, KPIs, and trend analysis.

- **Dimension nodes**  
  Provide business context for facts, like customer, product, time, or location. They help slice and analyze facts in meaningful ways.

- **Persistent nodes**  
  Store curated, reusable data that remains stable over time. They act as trusted reference layers, reduce reprocessing, and ensure consistency across reports and teams.

- **Work nodes**  
  Temporary or intermediate processing layers used during transformations. They support complex logic and performance optimization but are not intended for direct business consumption.

**Summary:**  
Together, these node types ensure data is accurate, reusable, scalable, and aligned with business reporting and decision-making needs.

----
## Nodetypes Config Matrix

| Category | Feature                         | Dim | Fact | Work | PStage |
|----------|----------------------------------|-----|------|------|--------|
| Create   | Create As Table                  | ✅  | ✅   | ✅   | ✅     |
| Create   | Create As Transient Table        | ✅  | ✅   | ✅   | ✅     |
| Create   | Create As View                   | ✅  | ✅   | ✅   | ⬜     |
| Create   | Create with Override SQL         | ✅  | ✅   | ✅   | ⬜     |
| Load     | MultiSource                      | ✅  | ✅   | ✅   | ✅     |
| Load     | Insert Strategy                  | ✅  | ✅   | ✅   | ✅     |
| Load     | Unmatched Record Strategy        | ✅  | ⬜   | ⬜   | ⬜     |
| Load     | Business Key                     | ✅  | ✅   | ⬜   | ✅     |
| Load     | Last Modified Comparison         | ✅  | ✅   | ⬜   | ✅     |
| Load     | Change Tracking                  | ✅  | ⬜   | ⬜   | ✅     |
| Load     | Truncate Before                  | ✅  | ✅   | ✅   | ✅     |
| Load     | Distinct                         | ✅  | ✅   | ✅   | ✅     |
| Load     | Group By All                     | ✅  | ✅   | ✅   | ✅     |
| Load     | Order By                         | ✅  | ✅   | ✅   | ✅     |
| Load     | Insert Zero Key Record           | ✅  | ⬜   | ⬜   | ⬜     |
| Load     | Methods                          | MERGE | MERGEINSERT |INSERT | MERGEINSERT |
| Others   | Enable Tests                     | ✅  | ✅   | ✅   | ✅     |
| Others   | Pre-SQL                          | ✅  | ✅   | ✅   | ✅     |
| Others   | Post-SQL                         | ✅  | ✅   | ✅   | ✅     |

---

# Coalesce Databricks Base Node Types Package

The Coalesce Base Node Types Package includes:

* [Work](#work)
* [Persistent Stage](#persistent-stage)
* [Dimension](#dimension)
* [Fact](#fact)
* [View](#view)
* [Code](#code)

---

## Work

The Coalesce work node is a versatile node that allows you to develop and deploy a Work table/view in Databricks.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Node Configuration

The Work node type has two configuration groups:

* [Node Properties](#work-node-properties)
* [Options](#work-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Work Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Work Options

You can create the node as:

* [Table](#work-options-table)
* [View](#work-options-view)

##### Work Options Table

![Work_options_table1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/f5643f40-fb37-4182-a11b-577bdc3e8f8d)

| **Property** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**:Truncates before data load**False**: Appends data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Work Options View

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Work Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/2dc81bb8-2285-46e1-8b93-ce7082800fc5)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Work Deployment

#### Work Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Work View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Work Redeployment

After the WORK node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.

#### Altering the Work Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Work Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Work Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Work Undeployment

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Databricks is dropped |

If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Work view from target environment |

---

## Persistent Stage 

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Node Configuration

The Persistent node type has two configuration groups:

* [Node Properties](#persistent-stage-node-properties)
* [Options](#persistent-stage-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Persistent Stage Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Options

![pstage_options1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/3914feb3-1db7-452b-8590-7c657b99c0eb)

| **Option** | **Description** |
|---------|-------------|
| **Create As** | Table is the only option at this time |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)**| Records with NULL timestamp are updated in target|
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**:Truncates table before data load**False**: Appends data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Persistent Stage Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![pstage_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/84ce06c9-103c-4700-ad3d-2e8222995b31)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Persistent Stage Deployment

#### Persistent Stage Initial Deployment

When deployed for the first time into an environment the Persistent node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Persistent Stage Redeployment

After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

#### Altering the Persistent Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

### Recreating the Persistent Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

### Persistent Stage Undeployment

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Drops table |
| **Drop Table or View** | Removes the table |

---

## Dimension

The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Snowflake.

A dimension table or dimension entity is a table or entity in a star, Snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

### Dimension Node Configuration

The Dimension node type has two configuration groups:

* [Node Properties](#dimension-node-properties)
* [Options](#dimension-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Dimension Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Options

##### Dimension Table Options

![Dimension](https://github.com/user-attachments/assets/8f65ee5c-ce09-457e-b4de-9e6c94038795)

| **Options** | **Description** |
|---------|-------------|
| **Create As** | Table or View |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)**| Records with NULL timestamp are updated in target|
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 Dimension |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**:Truncate before data load **False**: Appends data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension View Options

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![Dimension_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/5c3df3b0-f56d-4276-a51f-22364206b3c3)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Dimension Deployment

#### Dimension Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table will execute the Create Dimension Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Dimension View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Dimension Redeployment

After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

#### Altering the Dimension Tables

A few types of column or table changes will result in an ALTER statement to modify the Dimension Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Dimension Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Dimension Views

Any of the following changes to views will result in deleting and recreating the Dimension view.

* View defintion
* Adding table description
* Renaming view results

### Dimension Undeployment

If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Databricks is dropped |

If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension View in the target environment will be dropped.

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Dimension view from target environment. |

## Fact

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Databricks.

A fact table or a fact entity is a table or entity in a star or Databricks schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

### Fact Node Configuration

The Fact node has two configuration groups:

* [Node Properties](#fact-node-properties)
* [Options](#fact-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Fact Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Fact Options

![fact_options](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/fe15f4d1-fccc-4522-a75b-71682a8ef493)

| **Options** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for Fact table creation |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)**| Records with NULL timestamp are updated in target|
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Truncates data before data load**False**: Append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

#### Fact View

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/e540d2d0-2623-4b99-a435-a26df5fe3306)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Fact Deployment

#### Fact Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table will execute the Create Fact Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Fact View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Fact Redeployment

After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Fact Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Fact Tables

If any changes like change in join clause, adding transformations, change in business key column, change in configs like adding distinct, group by or orderby, the Fact Table will be recreated by running a CREATE OR REPLACE statement.

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

* Join clause
* Adding transformation
* Change in business key
* Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Fact Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Fact Undeployment

If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Databricks is dropped |

## View 

The Coalesce View UDN is a versatile node that allows you to develop and deploy a View in Databricks.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.

### View Node Configuration

The View node type has two configuration groups:

* [Node Properties](#view-node-properties)
* [Options](#view-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### View Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Secure** | Toggle: True/False**True**: A secured view is created**False**: A normal view is created |

### View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### View Deployment

#### View Initial Deployment

When deployed for the first time into an environment the View node will execute the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a View in the target environment |

#### View Redeployment

The subsequent deployment of View node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

#### View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes the view from the environment |

## Code

### Work Code

* [Node definition](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Work-363/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Work-363/create.sql.js)
* [Run Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Work-363/run.sql.js)

### Persistent Stage Code

* [Node definition](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/PersistentStage-367/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/PersistentStage-367/create.sql.j2)
* [Run Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/PersistentStage-367/run.sql.j2)

  ### Dimenison Code

* [Node definition](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Dimension-365/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Dimension-365/create.sql.j2)
* [Run Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Dimension-365/run.sql.j2)
  
### Fact Code

* [Node definition](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Fact-366/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Fact-366/create.sql.j2)
* [Run Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/Fact-366/run.sql.j2)

### View Code

* [Node definition](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/View-452/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/View-452/create.sql.j2)
* [Run Template](https://github.com/coalesceio/databricks-nodes/blob/main/nodeTypes/View-452/run.sql.j2)

[Macros](https://github.com/coalesceio/databricks-nodes/tree/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.1 | April 16, 2026 | Unmatched strategy added to Pstage, Fact |
| 1.1.0 | April 09, 2026 | Last Modified Toggle,Soft Delete/Hard delete,Zero key insert |
| 1.0.0 | June 06, 2025 | Initial version of Databricks Base node types |

---

## Data Vault by Scalefree for Databricks

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/data-vault-by-scalefree-for-databricks</code> <CopyButton textToCopy='@coalesce/databricks/data-vault-by-scalefree-for-databricks' />
Supported Platform: 
Latest Version: 1.1.0 (August 01, 2025)

audit trailbusiness vaultdata vaultdata vault 2.0historizationhublinkraw vaultsatellite

## Overview
Data Vault for Coalesce powered by Scalefree.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/data-vault-by-scalefree-for-databricks</code> <CopyButton textToCopy='@coalesce/databricks/data-vault-by-scalefree-for-databricks' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
Welcome to datavault4coalesce!

This is the Databricks version of Datavault4Coalesce. If you are looking for the Snowflake version, visit here!

***

The following documentation sheds some light on the node types that have been developed by Scalefree to make your Coalesce-Experience more comfortable!
The documentation can be found by clicking the links in the sidebar on the right side. In the documentation, the macros and their parameters are explained and further exemplified.

> **_NOTE:_**  Coalesce has released a Hands-On Guide for using Datavault4Coalesce. Check it our [here](https://guides.coalesce.io/data-vault/index.html#0)!

### Included node types
- Staging Area 
- Hubs, Links & Satellites 
- PIT Tables
- Snapshot Control

With datavault4coalesce you will get a lot of awesome features, including:
### Features
- Ghost Records
- Multi Batch Processing
- Automatic PIT cleanup based on logarithmic snapshot logic
- Pattern based on years of experience in Data Vault Loading
- Virtual Load Enddate

### Requirements

To use the node types efficiently, there are a few prerequisites you need to provide:
- Coalesce Environment connected to a Databricks Instance
- Basic Data Vault Knowledge (If this is new to you, check the following [Article](https://www.scalefree.com/what-is-data-vault/))

### Installation

To have access to the Datavault4Coalesce node definitions, you have to write an email to support@coalesce.io and request that the package is added to your environment!

### Resources:
- Learn more about coalesce [in the docs](https://docs.coalesce.io/docs)
- Contact us via datavault4coalesce@scalefree.com
- Check out the [Scalefree-Blog](https://www.scalefree.com/blog/)
- Blog Articles:
   - [Datavault4Coalesce Announcement](https://www.scalefree.com/scalefree-newsletter/bring-your-data-vault-automation-to-the-next-level-with-datavault4coalesce/)
   - [PIT Tables](https://www.scalefree.com/scalefree-newsletter/point-in-time-tables-insurance/)
   - [Hash Keys in Data-Vault](https://www.scalefree.com/architecture/hash-keys-in-the-data-vault/)
- [Scalefree Github Repo](https://github.com/ScalefreeCOM/datavault4coalesce-databricks.git/)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.0 | August 01, 2025 | GA Release |

---

## Databricks-Test Utility

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/databricks-test-utility</code> <CopyButton textToCopy='@coalesce/databricks/databricks-test-utility' />
Supported Platform: 
Latest Version: 1.0.0 (November 19, 2025)

assertionscolumn testdata qualitydata testinggreat expectationsnode testnull checkrow counttestinguniquenessvalidation

## Overview
This is an extension package for Coalesce, inspired by the Great Expectations package for Python. The intent is to allow Coalesce users to deploy GE-like tests in their data warehouse.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/databricks-test-utility</code> <CopyButton textToCopy='@coalesce/databricks/databricks-test-utility' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
Test-Utility

        

## About

`Test-Utility` is an extension package for [**Coalesce**], inspired by the [Great Expectations package for Python](https://greatexpectations.io/). The intent is to allow Coalesce users to deploy GE-like tests in their data warehouse directly from Coalesce, vs having to add another integration with their data warehouse.

## How to use?

Step 1: Once you install this package from Coalesce marketplace then you will need to import this package to workspace macro.

Assuming the alias you gave the package is TestUtility you would add the below Jinja.  `testUtils` could be anything.  You will use this alias to reference test macros in the imported package.

```yaml
{% import "TestUtility" as testUtils with context %}
```

![image](https://github.com/user-attachments/assets/62d6075d-f1cb-4b90-818e-b2a1c79285dc)

Step 2: Open a Node for which you want to create a test case.

Step 3: Goto Testing Configuration.

Step 4: Click on 'New Test' button.

![image](https://github.com/user-attachments/assets/dcc56a4b-5f89-49e8-be24-ee0deaa6a099)

Step 5: You will see new test case added for the Node.

![image](https://github.com/user-attachments/assets/aab2130d-9bbe-43f0-91b5-b1ae592bf842)

Step 6: In the text field, call the macro using the package import alias followed by a dot and the test case name.
        
_For Example, I am trying to Run Test case, 'expect_table_row_count_to_be_between' from the avilable test below._
        
![image](https://github.com/user-attachments/assets/4c546eb2-7b66-4e41-b514-99da676993a6)

Step 7: Replace the input parameters in macro call as per requirment.

_In this test case 'group_by' and 'filterCondition' inputs are optional, so i am ignoring here._

![image](https://github.com/user-attachments/assets/febcfc74-68dd-4447-a28a-94fb6b350935)

Note - You can refer object name with all the avilable pattern in Coalesce.  For example, [ref_no_link()](https://docs.coalesce.io/docs/reference/ref-functions/#ref_no_link) may be used instead of [this](https://docs.coalesce.io/docs/reference/ref-functions/#this).

Step 8: You can execute this test case using 'Run' button.

## Example

Lets consider, i have one table with named, TEST_TABLE. 

1. I want to check if each column value to be in a given set.
   _I will refer test case expect_column_values_to_be_in_set for this scenario. And my test case systax will be_
```yaml
   {{ testUtils.expect_column_values_to_be_in_set( 'CONTACT_VIA', ['EMAIL','CALL','TEXT']) }}
```

2. I want to check if each column entries to be strings that match a given SQL like pattern.
   _I will refer test case expect_column_values_to_match_like_pattern for this scenario. And my test case systax will be_
```yaml
   {{ testUtils.expect_column_values_to_match_like_pattern('{{ 'EMAIL_ID', '%@%') }}
```

## Available Tests

### Table shape

- [expect_column_to_exist](#expect_column_to_exist)
- [expect_row_values_to_have_recent_data](#expect_row_values_to_have_recent_data)
- [expect_grouped_row_values_to_have_recent_data](#expect_grouped_row_values_to_have_recent_data)
- [expect_table_aggregation_to_equal_other_table](#expect_table_aggregation_to_equal_other_table)
- [expect_table_column_count_to_be_between](#expect_table_column_count_to_be_between)
- [expect_table_column_count_to_equal_other_table](#expect_table_column_count_to_equal_other_table)
- [expect_table_column_count_to_equal](#expect_table_column_count_to_equal)
- [expect_table_columns_to_not_contain_set](#expect_table_columns_to_not_contain_set)
- [expect_table_columns_to_contain_set](#expect_table_columns_to_contain_set)
- [expect_table_columns_to_match_ordered_list](#expect_table_columns_to_match_ordered_list)
- [expect_table_columns_to_match_set](#expect_table_columns_to_match_set)
- [expect_table_row_count_to_be_between](#expect_table_row_count_to_be_between)
- [expect_table_row_count_to_equal_other_table](#expect_table_row_count_to_equal_other_table)
- [expect_table_row_count_to_equal_other_table_times_factor](#expect_table_row_count_to_equal_other_table_times_factor)
- [expect_table_row_count_to_equal](#expect_table_row_count_to_equal)

### Missing values, unique values, and types

- [expect_column_values_to_be_of_type](#expect_column_values_to_be_of_type)
- [expect_column_values_to_be_in_type_list](#expect_column_values_to_be_in_type_list)
- [expect_column_values_to_have_consistent_casing](#expect_column_values_to_have_consistent_casing)

### Sets and ranges

- [expect_column_values_to_be_in_set](#expect_column_values_to_be_in_set)
- [expect_column_values_to_not_be_in_set](#expect_column_values_to_not_be_in_set)
- [expect_column_values_to_be_between](#expect_column_values_to_be_between)
- [expect_column_values_to_be_decreasing](#expect_column_values_to_be_decreasing)
- [expect_column_values_to_be_increasing](#expect_column_values_to_be_increasing)

### String matching

- [expect_column_value_lengths_to_be_between](#expect_column_value_lengths_to_be_between)
- [expect_column_value_lengths_to_equal](#expect_column_value_lengths_to_equal)
- [expect_column_values_to_match_like_pattern](#expect_column_values_to_match_like_pattern)
- [expect_column_values_to_match_like_pattern_list](#expect_column_values_to_match_like_pattern_list)
- [expect_column_values_to_match_regex](#expect_column_values_to_match_regex)
- [expect_column_values_to_match_regex_list](#expect_column_values_to_match_regex_list)
- [expect_column_values_to_not_match_like_pattern](#expect_column_values_to_not_match_like_pattern)
- [expect_column_values_to_not_match_like_pattern_list](#expect_column_values_to_not_match_like_pattern_list)
- [expect_column_values_to_not_match_regex](#expect_column_values_to_not_match_regex)
- [expect_column_values_to_not_match_regex_list](#expect_column_values_to_not_match_regex_list)

### Aggregate functions

- [expect_column_distinct_count_to_be_greater_than](#expect_column_distinct_count_to_be_greater_than)
- [expect_column_distinct_count_to_be_less_than](#expect_column_distinct_count_to_be_less_than)
- [expect_column_distinct_count_to_equal_other_table](#expect_column_distinct_count_to_equal_other_table)
- [expect_column_distinct_count_to_equal](#expect_column_distinct_count_to_equal)
- [expect_column_distinct_values_to_be_in_set](#expect_column_distinct_values_to_be_in_set)
- [expect_column_distinct_values_to_contain_set](#expect_column_distinct_values_to_contain_set)
- [expect_column_distinct_values_to_equal_set](#expect_column_distinct_values_to_equal_set)
- [expect_column_max_to_be_between](#expect_column_max_to_be_between)
- [expect_column_mean_to_be_between](#expect_column_mean_to_be_between)
- [expect_column_median_to_be_between](#expect_column_median_to_be_between)
- [expect_column_min_to_be_between](#expect_column_min_to_be_between)
- [expect_column_most_common_value_to_be_in_set](#expect_column_most_common_value_to_be_in_set)
- [expect_column_proportion_of_unique_values_to_be_between](#expect_column_proportion_of_unique_values_to_be_between)
- [expect_column_quantile_values_to_be_between](#expect_column_quantile_values_to_be_between)
- [expect_column_stdev_to_be_between](#expect_column_stdev_to_be_between)
- [expect_column_sum_to_be_between](#expect_column_sum_to_be_between)
- [expect_column_unique_value_count_to_be_between](#expect_column_unique_value_count_to_be_between)

### Multi-column

- [expect_column_pair_values_A_to_be_greater_than_B](#expect_column_pair_values_a_to_be_greater_than_b)
- [expect_column_pair_values_to_be_equal](#expect_column_pair_values_to_be_equal)
- [expect_column_pair_values_to_be_in_set](#expect_column_pair_values_to_be_in_set)
- [expect_compound_columns_to_be_unique](#expect_compound_columns_to_be_unique)
- [expect_multicolumn_sum_to_equal](#expect_multicolumn_sum_to_equal)
- [expect_select_column_values_to_be_unique_within_record](#expect_select_column_values_to_be_unique_within_record)

### Distributional functions

- [expect_column_values_to_be_within_n_moving_stdevs](#expect_column_values_to_be_within_n_moving_stdevs)
- [expect_column_values_to_be_within_n_stdevs](#expect_column_values_to_be_within_n_stdevs)

## Documentation

### [expect_column_to_exist]

Expect the specified column to exist.

*Applies to:* Column

```yaml
tests: {{ expect_column_to_exist( 'columnName') }}
```

### [expect_row_values_to_have_recent_data]

Expect the model to have rows that are at least as recent as the defined interval prior to the current timestamp. Optionally gives the possibility to apply filters on the results.

*Applies to:* Column

```yaml
tests: {{ expect_row_values_to_have_recent_data( 'columnName', 'datePart_', 'interval_', row_condition = None) }}

Inputs: datepart_ = 'day'
        interval_ = '1'
        row_condition = 'id is not null' #optional
```

### [expect_grouped_row_values_to_have_recent_data] 

Expect the model to have **grouped** rows that are at least as recent as the defined interval prior to the current timestamp.
Use this to test whether there is recent data for each grouped row defined by `group_by` (which is a list of columns) and a `timestamp_column`. Optionally gives the possibility to apply filters on the results.

*Applies to:* Object

```yaml
tests : {{ expect_grouped_row_values_to_have_recent_data(group_by,timestamp_column,datepart,interval,filterCondition=None ) }}

Inputs: group_by = ['group_id', 'other_group_id']
        timestamp_column = 'date_day'
        datepart = day
        interval = 1
        filterCondition = "id is not null" #optional
```

### [expect_table_aggregation_to_equal_other_table]

Expect an (optionally grouped) expression to match the same (or optionally other) expression in a different table.

*Applies to:* Object

Simple Test:

```yaml
tests: {{ expect_table_aggregation_to_equal_other_table(expression,compare_model,group_by=None) }}

Inputs: expression = sum(col_numeric_a)
        compare_model = ref("other_model")
        group_by = [idx] #optional
```

More complex Test:

```yaml
tests: {{ expect_table_aggregation_to_equal_other_table(
          expression,
          compare_model,compare_expression=None,
          group_by=None,compare_group_by=None,
          row_condition=None,compare_row_condition=None,
          tolerance=0.0,tolerance_percent=None ) }}

Inputs: expression = 'max("column_a")'
        compare_model = 'ref("other_model")'
        compare_expression = 'max("column_b")'
        group_by = ['date_column']
        compare_group_by = ['some_other_date_column']
        row_condition = 'some_flag=true'
        compare_row_condition = 'some_flag=false'

```

**Note**: You can also express a **tolerance** factor, either as an absolute tolerable difference, `tolerance`, or as a tolerable % difference `tolerance_percent` expressed as a decimal (i.e 0.05 for 5%).

### [expect_table_column_count_to_be_between]

Expect the number of columns in a model to be between two values.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_be_between( minValue , maxValue) }}
Inputs: minValue = 1
        maxValue = 4 
```

### [expect_table_column_count_to_equal_other_table]

Expect the number of columns in a model to match another model.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_equal_other_table( 'comparing_tableName') }}

```

### [expect_table_columns_to_not_contain_set]

Expect the columns in a model not to contain a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_not_contain_set( notExpectedColumnList) }}
Inputs: notExpectedColumnList = ["col_a", "col_b"]
```

### [expect_table_columns_to_contain_set]

Expect the columns in a model to contain a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_contain_set( ExpectedColumnList) }}
Inputs: ExpectedColumnList = ["col_a", "col_b"]

```

### [expect_table_column_count_to_equal]

Expect the number of columns in a model to be equal to `expected_number_of_columns`.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_equal( ExpectedCount ) }}
Inputs: ExpectedCount = '7'
```

### [expect_table_columns_to_match_ordered_list]

Expect the columns to exactly match a specified list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_match_ordered_list( column_list, transform="UPPER") }}

Inputs:   column_list = ["col_a", "col_b"]
          transform = upper # (Optional)
```

### [expect_table_columns_to_match_set]

Expect the columns in a model to match a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_match_set( ExpectedColumnList) }}

Inputs: column_list = ["col_a", "col_b"]
```

### [expect_table_row_count_to_be_between]

Expect the number of rows in a model to be between two values.
*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_be_between( minValue , maxValue ,group_by = None, filterCondition = None) }}

Inputs:   minValue = 1 
          maxValue = 4 
          group_by = ['group_id', 'other_group_id', ...] # (Optional)
          filterCondition: 'id is not null' # (Optional)
```

### [expect_table_row_count_to_equal_other_table]

Expect the number of rows in a model match another model.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal_other_table( comparing_tableName, group_by_t1 = None, group_by_t2 = None,  filterCondition_t1 = None, filterCondition_t2 = None) }}

Inputs:   comparing_tableName = {{ ref('target','other_model') }}
          group_by_t1 = ['col1', 'col2'] # (Optional)
          group_by_t2 = ['col1', 'col2'] # (Optional)
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_table_row_count_to_equal_other_table_times_factor]

Expect the number of rows in a model to match another model times a preconfigured factor.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal_other_table_times_factor( comparing_tableName,  group_by_t1 = None, group_by_t2 = None, filterCondition_t1 = None, filterCondition_t2 = None, factor = None) }}

Inputs:   comparing_tableName = {{ ref('target','other_model') }}
          factor = '13'
          group_by_t1 = ['col1', 'col2'] # (Optional)
          group_by_t2 = ['col1', 'col2'] # (Optional)
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_table_row_count_to_equal]

Expect the number of rows in a model to be equal to `expected_number_of_rows`.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal( numberOfRecordExpected, group_by = None, filterCondition = None) }}

Inputs:   numberOfRecordExpected = '4'
          group_by = ['group_id', 'other_group_id', ...] # (Optional)
          filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_of_type]

Expect a column to be of a specified data type.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_of_type( columnName, dataType) }}

Input: dataType = 'date'
```

### [expect_column_values_to_be_in_type_list]

Expect a column to be one of a specified type list.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_in_type_list( columnName, dataType) }}

Inputs: dataType = ['date', 'datetime']
```

### [expect_column_values_to_have_consistent_casing]

Expect a column to have consistent casing. By setting `display_inconsistent_columns` to true, the number of inconsistent values in the column will be displayed in the terminal whereas the inconsistent values themselves will be returned if the SQL compiled test is run.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_have_consistent_casing( column_name, display_inconsistent_columns=False) }}

Input: display_inconsistent_columns = false # (Optional)
```

### [expect_column_values_to_be_in_set]

Expect each column value to be in a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: expectedValueList = ['a','b','c']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_between]

Expect each column value to be between two values.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_between( columnName, minValue , maxValue, filterCondition = None ) }}

Inputs: minValue = '0'  
        maxValue = '10' 
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_be_in_set]

Expect each column value not to be in a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: expectedValueList = ['e','f','g']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_increasing]

Expect column values to be increasing.

If `strictly: True`, then this expectation is only satisfied if each consecutive value is strictly increasing – equal values are treated as failures.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_increasing(  column_name, sort_column=None, strictly=True, filterCondition=None, group_by=None, step=None) }}

Inputs: sort_column = date_day
        filterCondition = "id is not null" # (Optional)
        strictly = true # (Optional for comparison operator. Default is 'true', and it uses '>'. If set to 'false' it uses '>='.)
        group_by = [group_id, other_group_id, ...] # (Optional)
        step = 1 # (Optional. If set, it requires the difference between values to be exactly this step. Requires numeric columns.)
```

### [expect_column_values_to_be_decreasing]

Expect column values to be decreasing.

If `strictly=True`, then this expectation is only satisfied if each consecutive value is strictly decreasing – equal values are treated as failures.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_decreasing( column_name, sort_column=None, strictly=True, filterCondition=None, group_by=None, step=None) }}

Inputs: sort_column = col_numeric_a
        filterCondition = "id is not null" # (Optional)
        strictly = true # (Optional for comparison operator. Default is 'true' and it uses '<'. If set to 'false', it uses '<='.)
        group_by = [group_id, other_group_id, ...] # (Optional)
        step = 1 # (Optional. If set, it requires the difference between values to be exactly this step. Requires numeric columns.)
```

### [expect_column_value_lengths_to_be_between]

Expect column entries to be strings with length between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_value_lengths_to_be_between( columnName, minLength, maxLength, filterCondition = None) }}

Inputs: min_value = '1' 
        max_value = '4' 
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_value_lengths_to_equal]

Expect column entries to be strings with length equal to the provided value.

*Applies to:* Column

```yaml
tests: {{ expect_column_value_lengths_to_equal( columnName, rowsValueLength, filterCondition = None) }}

Inputs: rowsValueLength = '10'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_match_regex]

Expect column entries to be strings that match a given regular expression. Valid matches can be found anywhere in the string, for example "[at]+" will identify the following strings as expected: "cat", "hat", "aa", "a", and "t", and the following strings as unexpected: "fish", "dog".

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_regex( columnName, regex, isRaw = False, flags='', filterCondition=None ) }}

Inputs: regex = '[at]+'
        filterCondition = 'id is not null' # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_not_match_regex]

Expect column entries to be strings that do NOT match a given regular expression. The regex must not match any portion of the provided string. For example, "[at]+" would identify the following strings as expected: "fish”, "dog”, and the following as unexpected: "cat”, "hat”.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_regex( columnName, regex, isRaw = False, flags='', filterCondition=None ) }} 

Inputs: regex = "[at]+"
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_match_regex_list]

Expect the column entries to be strings that can be matched to either any of or all of a list of regular expressions. Matches can be anywhere in the string.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_regex_list( columnName, regexList, matchType='all', isRaw=False, flags='', filterCondition=None ) }}

Inputs: regex_list = ["@[^.]*", "&[^.]*"]
        matchType = 'any' # (Optional. Default is 'all', which applies an 'AND' for each regex. If 'any', it applies an 'OR' for each regex.)
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_not_match_regex_list]

Expect the column entries to be strings that do not match any of a list of regular expressions. Matches can be anywhere in the string.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_regex_list( columnName, regexList, matchType='all', isRaw=False, flags='', filterCondition=None ) }} 

Inputs: regex_list = ["@[^.]*", "&[^.]*"]
        matchType = 'any' # (Optional. Default is 'all', which applies an 'AND' for each regex. If 'any', it applies an 'OR' for each regex.)
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_match_like_pattern]

Expect column entries to be strings that match a given SQL `like` pattern.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_like_pattern( columnName, pattern, filterCondition = None) }}

Inputs: pattern = '%@%'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_match_like_pattern]

Expect column entries to be strings that do not match a given SQL `like` pattern.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_like_pattern( columnName, pattern, filterCondition = None) }}

Inputs: pattern = '%&%'
        row_condition = "id is not null" # (Optional)
```

### [expect_column_values_to_match_like_pattern_list]

Expect the column entries to be strings that match any of a list of SQL `like` patterns.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_like_pattern_list( columnName, patternList, matchType='any', filterCondition = None) }}

Inputs: patternList = ["%@%", "%&%"]
        matchType = 'any' # (Optional. Default is 'any', which applies an 'OR' for each pattern. If 'all', it applies an 'AND' for each regex.)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_match_like_pattern_list]

Expect the column entries to be strings that do not match any of a list of SQL `like` patterns.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_like_pattern_list( columnName, patternList, matchType='any', filterCondition = None) }}

Inputs: patternList = ["%@%", "%&%"]
        matchType = 'any' # (Optional. Default is 'any', which applies an 'OR' for each pattern. If 'all', it applies an 'AND' for each regex.)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_equal]

Expect the number of distinct column values to be equal to a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_equal( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_be_greater_than]

Expect the number of distinct column values to be greater than a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_be_greater_than( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_be_less_than]

Expect the number of distinct column values to be less than a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_be_less_than( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_be_in_set]

Expect the set of distinct column values to be contained by a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b','c','d']
        row_condition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_contain_set]

Expect the set of distinct column values to contain a given set.

In contrast to `expect_column_values_to_be_in_set` this ensures not that all column values are members of the given set but that values from the set must be present in the column.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_contain_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_equal_set]

Expect the set of distinct column values to equal a given set.

In contrast to `expect_column_distinct_values_to_contain_set` this ensures not only that a certain set of values are present in the column but that these and only these values are present.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_equal_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b','c']
        row_condition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_equal_other_table]

Expect the number of distinct column values to be equal to number of distinct values in another model.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_equal_other_table( columnName, otherTableName, otherColumnName, filterCondition_t1 = None, filterCondition_t2 = None) }}

Inputs:   columnName = 'col_1'
          otherTableName = ref("my_model_2")
          otherColumnName = col_2
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_column_mean_to_be_between]

Expect the column mean to be between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_mean_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0' 
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_median_to_be_between]

Expect the column median to be between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_median_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0'
        maxValue = '2'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_quantile_values_to_be_between]

Expect specific provided column quantiles to be between provided min_value and max_value values.

*Applies to:* Column

```yaml
tests: {{ expect_column_quantile_values_to_be_between( columnName, quantile, minValue , maxValue, filterCondition = None, group_by = None ) }}

Inputs: quantile = '.95'
        minValue = '0' 
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition: "id is not null" # (Optional)
```

### [expect_column_stdev_to_be_between]

Expect the column standard deviation to be between a min_value value and a max_value value. Uses sample standard deviation (normalized by N-1).

*Applies to:* Column

```yaml
tests: {{ expect_column_stdev_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0' 
        maxValue = '2'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_unique_value_count_to_be_between]

Expect the number of unique values to be between a min_value value and a max_value value.

*Applies to:* Column

```yaml
tests: {{ expect_column_unique_value_count_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '2'
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_proportion_of_unique_values_to_be_between]

Expect the proportion of unique values to be between a min_value value and a max_value value.

For example, in a column containing [1, 2, 2, 3, 3, 3, 4, 4, 4, 4], there are 4 unique values and 10 total values for a proportion of 0.4.

*Applies to:* Column

```yaml
tests: {{ expect_column_proportion_of_unique_values_to_be_between( columnName, minValue, maxValue,  filterCondition = None, group_by = None) }}

Inputs: minValue = '0'  
        maxValue = '.4' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_most_common_value_to_be_in_set]

Expect the most common value to be within the designated value set

*Applies to:* Column

```yaml
tests: {{ expect_column_most_common_value_to_be_in_set( column_name, value_set, top_n, quote_values=True, data_type="STRING", filterCondition=None) }}

Inputs: value_set: ['0.5']
        top_n: 1
        quote_values: true # (Optional. Default is 'true'.)
        data_type: "decimal" # (Optional. Default is 'decimal')
```

### [expect_column_max_to_be_between]

Expect the column max to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_max_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None) }}

Inputs: minValue = '1' 
        maxValue = '1' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_min_to_be_between]

Expect the column min to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_min_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None) }}

Inputs: minValue = 0 
        maxValue = 1 
        group_by = [group_id, other_group_id, ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_sum_to_be_between]

Expect the column to sum to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_sum_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = 1 
        maxValue = 2 
        group_by = [group_id, other_group_id, ...] # (Optional)
        row_condition = "id is not null" # (Optional)
```

### [expect_column_pair_values_A_to_be_greater_than_B]

Expect values in column A to be greater than column B.

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_A_to_be_greater_than_B( columnNameA, columnNameB, orEqualTo = FALSE, filterCondition = None) }}

Inputs: columnNameA = 'col_numeric_a'
        columnNameB = 'col_numeric_a'
        orEqualTo = True
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_pair_values_to_be_equal]

Expect the values in column A to be the same as column B.

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_to_be_equal( columnNameA, columnNameB, filterCondition = None) }}

Inputs: columnNameA = 'col_numeric_a'
        columnNameB = 'col_numeric_a'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_pair_values_to_be_in_set]

Expect paired values from columns A and B to belong to a set of valid pairs.

Note: value pairs are expressed as lists within lists

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_to_be_in_set( columnNameA, columnNameB, validPairs, filterCondition = None) }}

Inputs: column_A = 'col_numeric_a'
        column_B = 'col_numeric_b'
        value_pairs_set = [[0, 1], [1, 0], [0.5, 0.5], [0.5, 0.5]]
        filterCondition = "id is not null" # (Optional)
```

### [expect_select_column_values_to_be_unique_within_record]

Expect the values for each record to be unique across the columns listed. Note that records can be duplicated.

*Applies to:* Object

```yaml
tests: {{ expect_select_column_values_to_be_unique_within_record( column_list, filterCondition=None) }}

Inputs: column_list = ["col_string_a", "col_string_b"]
        filterCondition = "id is not null" # (Optional)
```

### [expect_multicolumn_sum_to_equal]

Expects that sum of all rows for a set of columns is equal to a specific value

*Applies to:* Object

```yaml
tests: {{ expect_multicolumn_sum_to_equal( column_list, sum_total, group_by=None, filterCondition=None) }}

Inputs: column_list = ["col_numeric_a", "col_numeric_b"]
        sum_total = 4
        group_by = [group_id, other_group_id, ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_compound_columns_to_be_unique]

Expect that the columns are unique together, e.g. a multi-column primary key.

*Applies to:* Object

```yaml
tests: {% macro expect_compound_columns_to_be_unique( columnNames, filterCondition = None) %}

Inputs: column_list = ["date_col", "col_string_b"]
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_within_n_moving_stdevs]

A simple anomaly test based on the assumption that differences between periods in a given time series follow a log-normal distribution.
Thus, we would expect the logged differences (vs N periods ago) in metric values to be within Z sigma away from a moving average.
By applying a list of columns in the `group_by` parameter, you can also test for deviations within a group.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_within_n_moving_stdevs(
     column_name, date_column_name, group_by=None, 
    period='day', lookback_periods=1, trend_periods=7, test_periods=14, 
    sigma_threshold=3, sigma_threshold_upper=None, sigma_threshold_lower=None, 
    take_diffs=true, take_logs=true
) }}

Inputs: date_column_name = date
        period = day # (Optional. Default is 'day')
        lookback_periods = 1 # (Optional. Default is 1)
        trend_periods = 7 # (Optional. Default is 7)
        test_periods = 14 # (Optional. Default is 14)
        sigma_threshold = 3 # (Optional. Default is 3)
        take_logs = true # (Optional. Default is 'true')
        sigma_threshold_upper = x # (Optional. Replace 'x' with a value. Default is 'None')
        sigma_threshold_lower = y # (Optional. Replace 'y' with a value. Default is 'None')
        take_diffs = true # (Optional. Default is 'true')
        group_by = [group_id] # (Optional. Default is 'None')
```

### [expect_column_values_to_be_within_n_stdevs]

Expects (optionally grouped & summed) metric values to be within Z sigma away from the column average

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_within_n_stdevs( column_name, group_by=None, sigma_threshold=3) }}

Inputs: group_by = ['group_id'] # (Optional. Default is 'None')
        sigma_threshold = '3' # (Optional. Default is 3)
```

### [test_missing_dates]

Check missing dates in given range.

*Applies to:* Column

```yaml
tests: {{ test_missing_dates(date_column, from, to) }}

Inputs: from = '2024-01-01' # (start of date range)
        to = '2024-01-01' # (end of date range)
```

### [test_missing_date_offset]

Check missing dates from current date to offset.

*Applies to:* Column

```yaml
tests: {{ test_missing_date_offset(date_column, from, to) }}

Inputs: from = '100' 
        to = '1' 
```

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.0 | November 19, 2025 | Macro only package to perform data quality tests |

---

## External data package

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/external-data-package</code> <CopyButton textToCopy='@coalesce/databricks/external-data-package' />
Supported Platform: 
Latest Version: 1.0.1 (January 30, 2026)

cloud storagecopy-intocsvdata ingestionexternal volumesfile loadingjsonparquetschema inferenceseed

## Overview
Node types for data loading from files

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/external-data-package</code> <CopyButton textToCopy='@coalesce/databricks/external-data-package' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# External Data Package

## Brief Summary
 
External Data Package in Databricks is used to load data from files in cloud storage or volumes into Delta tables in an incremental and scalable way.

It is mainly used for batch ingestion of files.

---

The Coalesce Databricks External Data Package includes:

* [CopyInto](#CopyInto)
* [Code](#code)
  
 CopyInto 

### CopyInto Node Configuration

The Copy-Into node type the following configurations available:

* [Node Properties](#copy-into-node-properties)
* [General Options](#copy-into-general-options)
* [Source Data](#copy-into-file-location)
* [File Format Options](#copy-into-file-format)
* [Copy Options](#copy-into-copy-options)

CopyInto - Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### Key points to use CopyInto Node

* CopyInto node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* The data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config or reload parameter
* The path or subfolder name inside stage where the file is located can be specified using config 'Path or subfolder'.Do not prefix or suffix '/' in path name.Example,one level 'SUBFOLDER',two levels 'SUBFOLDER/INNERFOLDER'.

### Use CopyInto node with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema
* This is mainly a test to make sure create will work
* Hit Run button to execute DML

![Resync](https://github.com/user-attachments/assets/de3c4ce2-370e-43d0-a010-fc6131cd8669)

If the above works, it should be deployable as is.  Deploy will simply take the columns and execute a create table.

  CopyInto - General Options 

![Dbx-Copy-Into](https://github.com/user-attachments/assets/60b01ca8-2b99-46ec-8e5c-1491266f2333)

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Select from the options to create as Table   -**Table** |
| **TruncateBefore** | True / False toggle that determines whether or not a table is to be truncated before reloading - **True**: Table is truncated and Copy-Into statement is executed to reload the data into target table- **False**: Data is loaded directly into target table and no truncate action takes place. |
| **InferSchema** | True / False toggle that determines whether or not to infer the columns of file before loading - **True**: The node is created with the inferred columns- **False**: No infer table step is executed |
      
##### Source data

| **Setting** | **Description** |
|---------|-------------|
| **Path (Ex:gs://bucket/base/path )** | A path can be an internal volume,external volume in databricks or an external location pointing to S3 bucket/gcp container .|
| **File Name(Ex:a.csv,b.csv)** | Specifies a files name to be loaded.**Note:** It is advised to add either the filename or file pattern file loads |
| **Path or subfolder** | Not mandatory.Specifies the path or subfolders inside the stage where the file is located.Ensure that '/' is not pre-fixed before or after the subfolder name|
| **File Pattern (Ex:'.*hea.*[.]csv')**| A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match.**Note:** It is advised to add either the filename or file pattern file loads |

 CopyInto - File Format Options 

| **Setting** | **Description** |
|-------------|-----------------|
|**File Format Values**|Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available.File types supported-CSV,JSON,XML,ORC,AVRO,PARQUET |
|**GENERIC(ALL FILE TYPES)**|**ignorecorruptFiles**-Whether to ignore corrupt files.**ignoremissingfiles**-Boolean whether to ignore missing files.|
|**CSV**|**Record delimiter**-Characters that separate records in an input file**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Parse Header(InferSchema-true)**-Boolean that specifies whether to use the first row headers in the data files to determine column names.**Encoding**- Specifies the character set of the source data when loading data into a table **Date format**- String that defines the format of date values in the data files to be loaded. **Timestamp format**- String that defines the format of timestamp values in the data files to be loaded.**Multi-line**-Whether the CSV records span multiple lines.|
|**JSON**|**Date format**- String that defines the format of date values in the data files to be loaded.**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded**Multi-line**-Whether the CSV records span multiple lines.**Encoding**- Specifies the character set of the source data when loading data into a table|
|**XML**|**Date format**- String that defines the format of date values in the data files to be loaded.
**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded
**Multi-line**-Whether the CSV records span multiple lines.**rowTag**-The row tag of the XML files to treat as a row. In the example XML `<books> <book></book>...</books>`, the appropriate value is book. This is a required option for xml.
**rootTag**-Root tag of the XML files. For example, in `<books> <book><book>...</books>`, the appropriate value is books.**Encoding**- Specifies the character set of the source data when loading data into a table|
|**EXCEL**|**Header Rows**- Number of header rows to skip/use for column names (default: 1)
**Data Address**- Sheet and cell range specification (e.g., 'Sheet1'!A1:E100)|

 CopyInto - Copy Options 

| **Setting** | **Description** |
|-------------|-----------------|
|**Force**|- Boolean, default false. If set to true, idempotency is disabled and files are loaded regardless of whether they've been loaded before.|
|**Mergeschema**|-Boolean, default false. If set to true, the schema can be evolved according to the incoming data.|

### CopyInto - System Columns

The set of columns which has source data and file metadata information.

* **LOAD_TIMESTAMP** - Current timestamp when the file gets loaded.
* **FILENAME** - Name of the file the current row belongs to. 
* **FILEPATH** - Full path of the file in storage
* **FILEBLOCKSTART** - Start offset of the file split being read
* **FILEBLOCKEND** - Length of the file split being read
* **FILESIZE** - Size of the file in bytes
* **FILE_LAST_MODIFIED** - Last modified timestamp of the file

### CopyInto Deployment

#### CopyInto Initial Deployment
When deployed for the first time into an environment the Copy-into node of materialization type table will execute the below stage:

| Deployment Behavior | Stages Executed |
|--|--|
| Initial Deployment |Create Table|

### CopyInto Redeployment

#### Altering the CopyInto Tables

There are few column or table changes like Change in table name,Dropping existing column, Alter Column data type,Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column | Edit table description**: Alter table statement is executed to perform the alter operation.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### CopyInto Undeployment

If the CopyInto node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop table**: Target table in Snowflake is dropped

### Code

### CopyInto

* [Node definition](https://github.com/coalesceio/Databricks-External-data-package/blob/main/nodeTypes/CopyInto-531/definition.yml)
* [Create Template](https://github.com/coalesceio/Databricks-External-data-package/blob/main/nodeTypes/CopyInto-531/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Databricks-External-data-package/blob/main/nodeTypes/CopyInto-531/run.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.1 | January 30, 2026 | Excel support added to Copy-Into databricks |
| 1.0.0 | September 11, 2025 | Copy-Into node type to load data from Databricks |

---

## Functional node types

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/functional-node-types</code> <CopyButton textToCopy='@coalesce/databricks/functional-node-types' />
Supported Platform: 
Latest Version: 1.2.0 (August 18, 2025)

calendar tabledate dimensionpivotunpivot

## Overview
A package of functional nodes specific to common types of transformations or data sets.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/functional-node-types</code> <CopyButton textToCopy='@coalesce/databricks/functional-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Functional Node Types

The Coalesce Functional Node Types Package includes:

* [Date Dimension](#date-dimension)
* [Unpivot](#Unpivot)
* [Pivot](#Pivot)
* [Code](#code)

---

## Date Dimension

The Coalesce Date Dimension Table provides a comprehensive breakdown of date-related attributes, enabling efficient handling of date operations across various
use cases. The table typically includes columns such as day, month, year, day of the week, week of the year, quarter, and flags like day is weekday
or weekend. Additional columns like fiscal year, fiscal quarter, holiday indicators can also be included, depending on the requirements.

### Date Dimension Node Configuration

The Date Dimension node type has two configuration groups:

* [Node Properties](#date-dimension-node-properties)
* [Options](#date-options)
* [Additional options](#additional-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/45d22ea5-32ca-49f5-a464-b266cb29b516)

#### Date Dimension Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

##### Date Options

| **Setting** | **Description** |
|---------|-------------|
| **Starting Date**| A date from where the date values should be added in the date table.Default is :DATEADD(DAY, -730, CURRENT_DATE)|
| **Number of Days To Generate** | Numeric value indicating how many days' records should be generated from the Starting Date. |
| **Generated Date Column Name** |Metadata column name used in the SQL generated for inserting records into the table. |

#### Additional Options

You can create the node as:

* [Table](#date-table-create-as-table)
* [View](#date-table-create-as-view)

##### Date Dimension Create as Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention if enabled |
| **Business key** | Required column for Type 1 Dimensions |
| **Default String Value** | If Insert Zero Key Record toggle is True then add a default value for columns with datatype string |
| **Default Surrogate Key Value** | If Insert Zero Key Record toggle is True then add a default value for surrogate key column|
| **Default Date Value (Date Format DD-MM-YYYY)** | If Insert Zero Key Record toggle is True then add a default value for date key column in the format DD-MM-YYYY|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Date Dimension Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Override Create SQL** | Toggle: True/False**True**: View is created by overriding the SQL**False**: Nodetype defined create view SQL will execute |

### Date Dimension Deployment

#### Date Dimension Initial Deployment

When deployed for the first time into an environment the Date node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Date Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Date View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Date Dimension Redeployment

After the Date node with materialization type table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Date Table/view or recreating the Date table/view.

#### Altering the Date Table 

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

#### Date Dimension Recreating the Views

The subsequent deployment of Date node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with updated definition |

#### Date Dimension Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table** |  Drop view  Create or Replace Date table/transient table |
| **Table to View** |  Drop table/transient table Create Date view |

> 📘 **Materialization Date Dimension**
>
> When the materialization type of Date node is changed from table to View and use Override Create SQL for view creation. This ensures that the following change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/0296abf8-0747-4ae8-8478-0782e5e2e545)

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Date Dimension Deploy Undeployment

If a Date Dimension Node of materialization type table/view are deleted from a Datespace, that Datespace is committed to Git and that commit deployed to a higher level environment then the DateTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Unpivot

The [Unpivot node](https://docs.snowflake.com/en/sql-reference/constructs/unpivot#examples) in Coalesce rotates a table by transforming columns into rows. 
UNPIVOT is not exactly the reverse of PIVOT because it cannot undo aggregations made by PIVOT.

This operator can be used to transform a wide table (e.g. empid, jan_sales, feb_sales, mar_sales) into a narrower table (e.g. empid, month, sales).

### Unpivot limitations

* It cannot reverse aggregations performed by PIVOT
* It requires that all columns have the same data type.In case if the columns from source have diffrent data types,ensure the data types are type casted in an upstream node before adding a UNPIVOT node.
* UNPIVOT cannot be used in dynamic tables or stored procedures
* Ensure that your data is structured and formatted correctly, as any inconsistencies may affect the unpivoting process. It's important to check for any missing values, duplicate entries, or data types that are not compatible with the unpivot function.

### Unpivot Node Configuration

Unpivot has three configuration groups: 

* [Node Properties](#unpivot-node-properties)
* [General Options](#unpivot-general-options)
* [Unpivot Options](#unpivot-options)

#### Unpivot Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Pivot Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 ![image](https://github.com/user-attachments/assets/edd67d5d-7216-429a-a292-2fe4980d1a9e)

#### Unpivot general Options

![image](https://github.com/user-attachments/assets/3e607c04-f5c8-40ed-8da8-018a5455f520)

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' |
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Unpivot Options

![image](https://github.com/user-attachments/assets/45d4e9ae-8da3-46cf-b6bf-37ff693c8fa9)

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Pivot table** | Toggle: True/False True,it is the first run and the pivot table structure is yet to be determined.False,when the pivot table is created and generated columns have been Re-synced in Coalesce|
| **Value-Coulmn** |Column that will hold the values from the unpivoted columns |
| **Name-column** | Column that will hold the names of the unpivoted columns  |
| **Column-list**| The names of the columns in the source table or subquery that will be rotated into a single pivot column|
| **Include NULLS**| Specifies whether to include or exclude rows with NULLs|

### Unpivot node Usage

* Add a Unpivot node on top of source node
* Add the Unpivot column list ,value column,name column in config
* When you choose the Unpivot and value dropdown,ensure that the textbox alongside the dropdown is entered with Column name.This textBox information is required once the Unpivot table structure is synced into Coalesce.
* The toggle 'Infer Structure of Unpivot Data' is required to be true when the node is created for the first time.
* The toggle 'Single value column' is set to false, if you want a multi-dimensional Unpivot
* Once the Unpivot table is created,the 'Re-Sync Columns' can be used to sync the structure of Unpivot table into Coalesce mapping grid.
* After Re-sync,recreate the table with 'Infer Structure of Unpivot Data' set to false
  ![image](https://github.com/user-attachments/assets/79282085-e9b7-41e1-8bd2-68fdf98eeb00)
* If the above works, it should be deployable as is. Deploy will simply take the columns and execute a create table.
* Hit run to insert data into table keeping the 'Infer Structure of Pivot Data' set to false

#### Unpivot Initial Deployment

### Points to note for deployment
* Create table with ‘Infer UNPIVOT structure’ toggle enabled
* Re-Sync columns to the mapping grid
* Deploy with ‘Infer UNPIVOT structure’ toggle set to false
* Repeat the above steps if you see changes in column of table during redeployment.It is fine to skip for change in materialization type,change in target location or change in node name
* Ensure the new columns added or dropped are part of the inferred UNPIVOT structure and not added/dropped directly in the mapping grid.The deployment will succeed but insert will fail

> 📘 **Deployment**
>
> Ensure 'Infer Unpivot structure' set to false before deployment

When deployed for the first time into an environment the Unpivot node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Unpivot Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Unpivot View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Unpivot Redeployment

After the Unpivot node with materialization type table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Unpivot Table or recreating the Unpivot table.
Unpivot
#### Altering the  Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

#### Unpivot Recreating the Views

The subsequent deployment of Unpivot node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Unpivot View** | Creates a new view with upUnpivotd definition |

#### Unpivot Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table** |  Drop view  Create or Replace Unpivot table/transient table |
| **Table to View** |  Drop table/transient table Create Unpivot view |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Unpivot Deploy Undeployment

If a Unpivot Node of materialization type table/view/transient table are deleted from a Unpivotspace, that Unpivotspace is committed to Git and that commit deployed to a higher level environment then the UnpivotTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Pivot

Pivoting ia crucial feature of data transformation.The [Pivot node](https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-qry-select-pivot) in Coalesce transforms a table by turning the unique values from one column in the input expression into multiple columns and aggregating results where required on any remaining column values. This operation is specified in the `FROM` clause after the table name or subquery.  

It is especially useful for converting narrow tables, such as one with columns for `empid`, `month`, and `sales`, 
into wider tables, for example, `empid`, `jan_sales`, `feb_sales`, and `mar_sales`.

### Pivot Node Configuration

Pivot has three configuration groups: 

* [Node Properties](#pivot-node-properties)
* [General Options](#pivot-general-options)
* [Pivot Options](#pivot-options)

  ![pivot-node-config](https://github.com/user-attachments/assets/edd67d5d-7216-429a-a292-2fe4980d1a9e)

#### Pivot Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Pivot Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Pivot General Options

![pivot-general](https://github.com/user-attachments/assets/5adfb637-3d5e-4401-8151-17bcbe17c884)

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' or 'transient table' |
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

##### Pivot Options

![pivot-options](https://github.com/user-attachments/assets/9e22a2a5-7e70-4c81-bb0a-ef599fcefa3e)

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Pivot table** | Toggle: True/False True,it is the first run and the pivot table structure is yet to be determined.False,when the pivot table is created and generated columns have been Re-synced in Coalesce|
| **Pivot column**|Pivot column(Dropdown)  Pivot column(textbox) The column from the source table or subquery that will be rotated and turned into new columns.|
| **Value Column**|-Value Column(Dropdown)  -Value Column(textbox) Columns to which you want to apply aggregation.-Aggregate FunctionsAggregation you want to apply, like AVG, COUNT, MAX, MIN, and SUM.|
|**Filter Column Values(comma separated list of column values-Ex 1,'North')**|Specified list of column values for the pivot column.For instance if quarter and region as pivot columns,then we can specify column value as 1,'North'|
|**Alias for Filter Column Values(Ex Quarter_North)**|Specify an alias for the combined value of pivot columns.For instance if quarter and region as pivot columns,then Quarter1_North can be an alias|

### Pivot node Usage

* Add a Pivot node on top of source node
* Add the pivot columns,value columns ,aggregation operation,filter column values and alias from config.Find below example for filter columns and alias
* Single pivot column
 
  ![sin-pivot](https://github.com/user-attachments/assets/9d771a45-2793-4782-afd2-866c0d6c20fd)
  
* Multiple pivot columns
  
  ![mul-pivot](https://github.com/user-attachments/assets/fb86f53b-3347-42d9-987c-8ddfa5ccb461)
  
* When you choose the pivot and value dropdown,ensure that the textbox alongside the dropdown is entered with Column name.This textBox information is required once the pivot table structure is synced into Coalesce.
* The toggle 'Infer Structure of Pivot Data' is required to be true when the node is created for the first time.
* Once the pivot table is created,the 'Re-Sync Columns' can be used to sync the structure of pivot table into Coalesce mapping grid.
* After Re-sync,recreate the table with 'Infer Structure of Pivot Data' set to false
* If the above works, it should be deployable as is. Deploy will simply take the columns and execute a create table.
* Hit run to insert data into table keeping the 'Infer Structure of Pivot Data' set to false
  
### Pivot Deployment

### Points to note for deployment
* Create table with ‘Infer PIVOT structure’ toggle enabled
* Re-Sync columns to the mapping grid
* Deploy with ‘Infer PIVOT structure’ toggle set to false
* Repeat the above steps if you see changes in column of table during redeployment.It is fine to skip for change in materialization type,change in target location or change in node name
* Ensure the new columns added or dropped are part of the inferred PIVOT structure and not added/dropped directly in the mapping grid.The deployment will succeed but insert will fail
> 📘 **Deployment**
>
> Ensure 'Infer Pivot structure' set to false before deployment

#### Pivot Initial Deployment

When deployed for the first time into an environment the Pivot node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Pivot Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Pivot View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Pivot Redeployment

After the Pivot node with materialization type table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Pivot Table or recreating the Pivot table.
Pivot
#### Altering the  Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Pivot Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

#### Pivot Recreating the Views

The subsequent deployment of Pivot node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Pivot View** | Creates a new view with pivot definition |

#### Pivot Drop and Recreate View/Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table** |  Drop view  Create or Replace Pivot table/transient table |
| **Table to View** |  Drop table/transient table Create Pivot view |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Pivot Deploy Undeployment

If a Pivot Node of materialization type table/view/transient table are deleted from a workspace, that workspace is committed to Git and that commit deployed to a higher level environment then the PivotTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Code

### Date Dimension Code

| **Component** | **Link** |
|---------------|----------------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/DateDimension-461/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/DateDimension-461/create.sql.j2)|
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/DateDimension-461/run.sql.j2) |

### Unpivot Code

| **Component** | **Link** |
|---------------|----------------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Unpivot-478/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Unpivot-478/create.sql.j2)|
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Unpivot-478/run.sql.j2) |

### Pivot Code

| **Component** | **Link** |
|---------------|----------------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Pivot-468/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Pivot-468/create.sql.j2)|
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/databricks-functional-node-types/blob/main/nodeTypes/Pivot-468/run.sql.j2) |

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.2.0 | August 18, 2025 | Transform changed-Date dimension |
| 1.1.0 | June 12, 2025 | New node type Pivot added |
| 1.0.0 | June 06, 2025 | Initial version of Databricks functional node types |

---

## Incremental Nodes

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/incremental-nodes</code> <CopyButton textToCopy='@coalesce/databricks/incremental-nodes' />
Supported Platform: 
Latest Version: 1.0.0 (November 14, 2025)

batch processingchange detectiondelta processingincrementalincremental loadingperformance optimizationwatermark

## Overview
Incremental node types

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/incremental-nodes</code> <CopyButton textToCopy='@coalesce/databricks/incremental-nodes' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Databricks-Incremental-nodes

The Coalesce Incremental Package includes:

* [Incremental load](#incremental-load)
* [Code](#code)

## Incremental Load

The Coalesce Incremental load node is a versatile node that allows you to develop and deploy a Stage table/view in Databricks where we can perform incremental load in comparison with a persistent table added on top of it.

### Incremental Load Node Configuration

* [Node Properties](#incremental-load-node-properties)
* [Options](#incremental-load-options)

#### Incremental Load Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Incremental node will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Incremental Load Options

| **Options** | **Description** |
|-----------|-----------------|
| **Create As** | Provides option to choose materialization type as `table` or `view`. |
| **Filter data based on Persistent table** | - True - provides option to perform incremental load.- False - a normal initial load of data from source is done. |
| **Persistent table location(required)** | The Coalesce storage location. |
| **Persistent table name(required)** | The table name of the persistent table. |
| **Incremental load column(date)** | A date column based on which incremental data is loaded. |

## Incremental Load Example Workflow

1. Add a source node.
2. Add the Incremental UDN.
3. Leave the 'Filter data based on Persistent Table' option set to False.
4. Create the node.
5. Add the Persistent table to the view.
6. Create and Run the node.
7. Go to the Incremental UDN and change the 'Filter data based on Persistent Table' option to true.
8. Use the pattern based option to match the persistent table (alter the definition of the UDN, if necessary), or add the table name manually in the last config item.
9. Remove the existing (basic) join and use the 'Copy To Editor' to add the new join, including sub-select.
10. Re-run the Incremental UDN.

### Incremental Load Deployment

#### Incremental Load Initial Deployment

When deployed for the first time into an environment the Incremental load node of materialization type table will execute the Create State Table.

| **Stage** | **Description** |
|----------|-----------------|
| **Create Stage Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment. When deployed for the first time into an environment the Work node of materialization type view will execute the Create Stage View. |
| **Create Stage View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |

#### Incremental Load Redeployment

After the Incremental Load  has been deployed for the first time into a target environment, subsequent deployments with column level changes or table level changes may result in altering the target Table.

#### Incremental Load Altering the Stage Tables

There are few column or table changes if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

* Changing the table name
* Dropping an existing column
* Altering Column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Clone Table** | Creates an internal table. |
| **Rename Table \| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation. |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost. |
| **Delete Table** | Drops the internal table. |

#### Incremental Load Recreating the Stage Views

The subsequent deployment of Incremental load node of materialization type view with changes in view definition,adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete View** | Delete the view |
| **Create View**| Create a new view |

### Incremental Load Undeployment

If a Incremental load node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the stage table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete Table** | Coalesce Internal table is dropped. |
| **Delete Table** | Target table in Snowflake is dropped. |

If a Incremental load node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the StageView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete View** | Drops the existing stage view from target environment. |

### Code

#### Incremental Load Code

* [Node definition](https://github.com/coalesceio/Databricks-Incremental-nodes/blob/main/nodeTypes/IncrementalLoad-595/definition.yml)
* [Create Template](https://github.com/coalesceio/Databricks-Incremental-nodes/blob/main/nodeTypes/IncrementalLoad-595/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Databricks-Incremental-nodes/blob/main/nodeTypes/IncrementalLoad-595/run.sql.j2)

* [Macro](https://github.com/coalesceio/Databricks-Incremental-nodes/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.0 | November 14, 2025 | First release of Databricks Incremental node type |

---

## Lakeflow Declarative Pipelines

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/lakeflow-declarative-pipelines</code> <CopyButton textToCopy='@coalesce/databricks/lakeflow-declarative-pipelines' />
Supported Platform: 
Latest Version: 1.1.0 (February 12, 2026)

auto loaderdeclarative pipelinesdelta live tablesldpreal-timestreamingstreaming tables

## Overview
Lakeflow Declarative Pipeline (LDP) in Databricks is a declarative framework for building and managing reliable, scalable, and maintainable data pipelines

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/lakeflow-declarative-pipelines</code> <CopyButton textToCopy='@coalesce/databricks/lakeflow-declarative-pipelines' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# **Lakeflow Declarative Pipelines**

## **Brief Summary**

The Coalesce LDP node type allows you to create a streaming table.
A streaming table is a form of Unity Catalog managed table that is also a streaming target for Lakeflow Declarative Pipelines.

[Lakeflow Declarative Pipelines](https://docs.databricks.com/aws/en/dlt/concepts)
is a declarative framework for developing and running batch and streaming data pipelines in SQL and Python. Lakeflow Declarative Pipelines runs on the performance-optimized Databricks Runtime (DBR), and the Lakeflow Declarative Pipelines flows API uses the same DataFrame API as Apache Spark and Structured Streaming. Common use cases for Lakeflow Declarative Pipelines include incremental data ingestion from sources such as cloud storage (including Amazon S3, Azure ADLS Gen2, and Google Cloud Storage) and message buses (such as Apache Kafka, Amazon Kinesis, Google Pub/Sub, Azure EventHub, and Apache Pulsar), incremental batch and streaming transformations with stateless and stateful operators, and real-time stream processing between transactional stores like message buses and databases.

A [streaming table](https://docs.databricks.com/aws/en/dlt/streaming-tables) is a Delta table with additional support for streaming or incremental data processing. A streaming table can be targeted by one or more flows in an ETL pipeline.

## **Key points**

1.Databricks by default creates tables with lowercase.Hence,it is better to keep table names in lowercase.[https://docs.databricks.com/en/sql/language-manual/sql-ref-names.html](https://docs.databricks.com/en/sql/language-manual/sql-ref-names.html)

2.The node which loads from a file creates a streaming table.For further processing,Re-Sync the columns in the mapping grid using Re-Sync columns button.
The streaming table can be re-created with the Columns inferred using Include Columns Inferred option.

![dlt-resync](https://github.com/user-attachments/assets/8199536e-ef4d-4b24-ab69-ce8c89356938)

3.The streaming tables can be recreated and refreshed if there is a need to drop the inferred columns or add transformations to columns inferred in previous step.The structure of the streaming table is refreshed only on enabling the 'Refresh Stream' Option

### **LDP Node Configuration**

The LDP has two configuration groups:

* [LDP Node Properties](#ldp-node-properties)
* [General Options](#general-options)
* [LDP Options](#ldp-options)
* [File Options](#file-options)
* [General file load Options](#general-file-load-options)
* [File format options](#file-format-options)
* [Advanced file load options](#advanced-file-load-options)
* [Schedule Options](#schedule-options)

 LDP Node Properties 

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### General Options

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Choose materialization type-by default it is STREAMING TABLE  |
|**Schedule refresh**| True / False toggle- **True**: Schedule Option will be visible- **False**:  Schedule Option will be disabled|

 LDP options 

| **Option**                     | **Description** |
|----------------------------------|---------------|
| **Type of Lakeflow Declarative Pipeline**     | - Streaming Table |
| **Read Files**                   | **True / False** toggle - **True**: Allows loading from an external cloud location into a streaming table.- **False**: Allows loading from a table source. |
| **Include Columns Inferred**     | Enables recreating the streaming table with added transformations after syncing columns. |
| **Table Properties**             | Defines table properties like quality. |
| **Refresh Stream**               | Enables full refresh of the table if any changes in structure occur. |
| **Partition By** | Specifies the columns used for partitioning the streaming table. |
| **Table Constraints** | - **Primary Key**- **Foreign Key** |
| **Other Constraints** | - **Column Name**- **Expectation Expression**- **On Violation Action** |

 File options 

| **Option**                     | **Description** |
|----------------------------------|---------------|
| **File Location** | External cloud location of the file. |
| **File Name/File Pattern**     | The pattern to identify files to be processed. |
| **File Type**                  | - Options: **[CSV](#csv-options)**, **[JSON](#json-options)**, **[Parquet](#parquet-options)**, **[Avro](#avro-options)**, **[ORC](#orc-options)**, **[Text](#text-options)**, **[XML](#xml-options)**. |
| **Advanced File Load Options** | Toggle to enable additional loading options for files. |
| **Advanced File Format Options** | Toggle to enable additional formatting options for files. |

 General File Load options 

| **Option**                     | **Description** |
|----------------------------------|---------------|
| **Schema Definition**          | The schema to define the structure of the input data. |
| **Override inferred schema columns**| The schema to define the structure of the input data. |
| **Infer Exact Column Types**   | Toggles whether to infer column data types accurately. |
| **Ignore Corrupt Files**       | Toggle to ignore files that are corrupt during processing. |
| **Ignore Missing Files**       | Toggle to ignore files that are missing during processing. |
| **Partition Columns**          | Comma-separated list of Hive-style partition columns to include|

 Advanced File Load options 

| **Option**                     | **Description** |
|----------------------------------|---------------|
| **Process files modified after (Ex '2024-01-01T00:00:00Z')**   | Only process files modified after this timestamp |
| **Process files modified before (Ex'2024-12-31T23:59:59Z')**       | Only process files modified before this timestamp |
| **Recursive file lookup**       | Search nested directories regardless of naming scheme. Default: false |
| **Use strict glob pattern matching**          | Use strict glob pattern matching. Default: true|

 File Format options 

PARQUET Options

| **Group**        | **Option**            |  **Description**                                            |
|------------------|-----------------------|------------------------------------------------------------|
| Options          | Merge Schema           |  Infer and merge schema across multiple files. Default: false |
| Options          | Column name for rescued data column |  Column name to collect data that doesn't match schema     |
| Advanced         | Datetime rebase mode    | Rebasing DATE/TIMESTAMP between Julian and Proleptic Gregorian calendars. Options: EXCEPTION, LEGACY, CORRECTED. Default: LEGACY |
| Advanced         | Integer rebase mode       | Rebasing INT96 timestamps. Options: EXCEPTION, LEGACY, CORRECTED. Default: LEGACY |

AVRO Options

| **Group**        | **Option**            | **Description**                                            |
|------------------|-----------------------|---------------------------------------------------------------|
| Options          | avroSchema            | User-provided Avro schema (can be evolved schema)        |
| Options          | Merge Schema          | Infer and merge schema across files. Default: false       |
| Options          | Column name for rescued data column     |Column name to collect data that doesn't match schema     |
| Advanced         | Datetime rebase mode    | Rebasing DATE/TIMESTAMP between Julian and Proleptic Gregorian calendars. Options: EXCEPTION, LEGACY, CORRECTED. Default: LEGACY |
| Advanced         | Integer rebase mode       | Rebasing INT96 timestamps. Options: EXCEPTION, LEGACY, CORRECTED. Default: LEGACY |

ORC Options

| **Group**        | **Option**            |  **Description**                                            |
|------------------|-----------------------|------------------------------------------------------------|
| Options          | Merge Schema           | Infer and merge schema across files. Default: false       |

XML Options

| **Group**           | **Option**            |  **Description**                                            |
|---------------------|-----------------------|------------------------------------------------------------|
| Options             | Row Tag                | Required. XML element to treat as a row (e.g., 'book' for `<books><book>...</book></books>`) |
| Options             | Encoding               |Character encoding. Default: UTF-8                        |
| Options             | String representation of null | String representation of null. Default: null             |
| Options             | Column name for rescued data column| Column name to collect data that doesn't match schema     |
| Advanced            | Prefix for attribute  |Prefix for attribute field names. Default: _              |
| Advanced            | valueTag              | Tag for character data in elements with attributes. Default: _VALUE |
| Advanced            | Exclude attributes from elements | Exclude attributes from elements. Default: false          |
| Advanced            | Skip surrounding whitespace| Skip surrounding whitespace. Default: true                |
| Advanced            | Ignore namespace | Ignore namespace prefixes. Default: false                 |
| Advanced            | Sampling Ratio        | Fraction of rows for schema inference. Default: 1.0       |
| Advanced            | Mode                  | Corrupt record handling. Options: PERMISSIVE, DROPMALFORMED, FAILFAST. Default: PERMISSIVE |
| Advanced            | Date Parsing Format   | Date parsing format. Default: yyyy-MM-dd                  |
| Advanced            | Timestamp Parsing Format | Timestamp parsing format. Default: yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX] |
| Advanced            | Path to XSD for row validation| Path to XSD for row validation       

TEXT Options

| **Group**        | **Option**            |  **Description**                                            |
|------------------|-----------------------|------------------------------------------------------------|
| Options          | encoding              | Character encoding. Default: UTF-8                        |
| Options          | Line Separator               | Line separator string. Default: auto-detects \r, \r\n, \n |
| Advanced         | Whole text            | Read entire file as single record. Default: false         |

JSON Options

| **Group**          | **Option**            | **Description**                                            |
|--------------------|-----------------------|------------------------------------------------------------
| Options            | Encoding              |  Character encoding. Default: UTF-8                        |
| Options            | quote                 | Quote character. Default: "                               |
| Options            | escape                | Escape character. Default: \                               |
| Advanced           | Skip surrounding whitespace| Skip surrounding whitespace. Default: true                |
| Options            | Number of rows to skip from beginning      | Number of rows to skip from beginning. Default: 0         |
| Options            | Column name for rescued data column  | Column name for rescued data                              |
| Advanced           | MultiLine             | Records span multiple lines. Default: false               |
| Advanced           | Mode                  | Options: PERMISSIVE, DROPMALFORMED, FAILFAST. Default: PERMISSIVE |
| Advanced           | Character indicating line comment| Character indicating line comment. Default: disabled       |
| Advanced           | dateFormat            | Date parsing format. Default: yyyy-MM-dd                  |
| Advanced           | timestampFormat       | Timestamp parsing format                                   |
| Advanced           | Ignore Leading WhiteSpaces| Default: false                                            |
| Advanced           | ignore Trailing WhiteSpaces| toggleButton     | Default: false                                            |
| Advanced           | Allowcomments | Allow comments in JSON data. Default: false       |
| Advanced           | AllowUnquotedFieldNames|Allow unquoted field names. Default: false        |
| Advanced           | InferPrimitiveTypes    | Infer primitive types correctly. Default: true     |

CSV Options

| **Group**          | **Option**            | **Description**                                            |
|--------------------|-----------------------|------------------------------------------------------------|
| Options            | Delimiter             |  Delimiter used in csv files                              |
| Options            | Header added          |  Header added in csv file                                 |
| Options            | Encoding              |  Character encoding. Default: UTF-8                        |
| Options            | quote                 | Quote character. Default: "                               |
| Options            | escape                | Escape character. Default: \                               |
| Advanced           | Skip surrounding whitespace| Skip surrounding whitespace. Default: true                |
| Options            | Number of rows to skip from beginning      | Number of rows to skip from beginning. Default: 0         |
| Options            | Column name for rescued data column  | Column name for rescued data                              |
| Advanced           | MultiLine             | Records span multiple lines. Default: false               |
| Advanced           | Mode                  | Options: PERMISSIVE, DROPMALFORMED, FAILFAST. Default: PERMISSIVE |
| Advanced           | Character indicating line comment| Character indicating line comment. Default: disabled       |
| Advanced           | dateFormat            | Date parsing format. Default: yyyy-MM-dd                  |
| Advanced           | timestampFormat       | Timestamp parsing format                                   |
| Advanced           | Ignore Leading WhiteSpaces| Default: false                                            |
| Advanced           | ignore Trailing WhiteSpaces| toggleButton     | Default: false                                            |

### Schedule Options

Schedule Options is available only when Schedule Refresh toggle is True

| **Option** | **Description** |
|------------|----------------|
|**Task Schedule**| Options in Task Schedule- Periodic Schedule- CRON    |
|**Schedule refesh-time period**|Available when Task Schedule is Set to Periodic ScheduleOptions in Schedule refesh-time period-Every Hours-Every Days-Every Weeks |
|**Specific interval of periodic refresh(integer value)**|Available when Task Schedule is Set to Periodic Schedule |
|**CRON string**|Available when Task Schedule is Set to CRON|
|**CRON TIME ZONE**|Available when Task Schedule is Set to CRON |

## **LDP Deployment**

### Initial Deployment
When deployed for the first time into an environment DLT node will execute three stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Streaming table** | This stage will execute a CREATE OR REFRESH statement and create a Streaming table in the target environment |

### Redeployment

After the Streaming table has deployed for the first time into a target environment, subsequent deployments may result in either altering the Streaming table or recreating the Streaming table.

If a Streaming table is to be altered this will run the following stage:

#### Altering the Streaming table

The following config changes trigger ALTER statements:

1. Add schedule
2. Alter schedule
3. Drop schedule

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Streaming table** | Executes ALTER to modify parameters |

#### Recreating the Streaming table

If anything changes other than the configuration options specified above then the Streaming table will be recreated by running a CREATE OR REPLACE statement.

### Undeployment

If a Streaming table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the Streaming table in the target environment will be dropped.

This is executed as a single stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Streaming table** | Removes the Streaming table |

## Code

* [Node definition](https://github.com/coalesceio/databricks-DLT/blob/main/nodeTypes/DLT-368/definition.yml)
* [Create Template](https://github.com/coalesceio/databricks-DLT/blob/main/nodeTypes/DLT-368/create.sql.j2)
* [Macros](https://github.com/coalesceio/databricks-DLT/blob/main/macros/macro-1.yml)
* [Run Template](https://github.com/coalesceio/databricks-DLT/blob/main/nodeTypes/DLT-368/run.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.0 | February 12, 2026 | File format and Advanced file format options ,support added for AVRO,ORC,TEXT and XML |
| 1.0.1 | January 09, 2026 | Parquet file type support added |
| 1.0.0 | August 01, 2025 | LDP node type added |

---

## Materialized View

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/databricks/materialized-view</code> <CopyButton textToCopy='@coalesce/databricks/materialized-view' />
Supported Platform: 
Latest Version: 1.0.1 (June 13, 2025)

cachingmaterialized viewperformancequery optimization

## Overview
In Databricks, Materialized views cache the results and update them as the underlying source tables change—either on a schedule or automatically.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/databricks/materialized-view</code> <CopyButton textToCopy='@coalesce/databricks/materialized-view' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Databricks Materialized View 

The Databricks Materialized View UDN is a versatile node that allows you to develop and deploy a Materialized View in Databricks.

A [materialized view](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view) is a view where precomputed results are available for query and can be updated to reflect changes in the input. Each time a materialized view is refreshed, query results are recalculated to reflect changes in upstream datasets. All materialized views are backed by a DLT pipeline. You can refresh materialized views manually or on a schedule.

## Node Configuration

The Databricks Materialized View Stage has Four configuration groups:

* [Node Properties](#node-properties)
* [General Options](#general-options)
* [Materialized View Options](#materialized-view-options)
* [Schedule Option](#schedule-options)

### Node Properties
There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### Key point -limitations

* Materialized View automatically infers the structure of the source table and defines the data type of columns.
* Multi-source is not supported with this version of Databricks Materialized View in Coalesce
* Column-level constraints like NOT NULL,column-level descriptions are also not supported with this version of Databricks Materialized View in Coalesce

### General Options

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Choose materialization type-by default it is MATERIALIZED VIEW  |
|**Schedule refresh**| True / False toggle- **True**: Schedule Option will be visible- **False**:  Schedule Option will be disabled|

### Materialized View Options
Materialized View Options is available only when Infer MV Structure toggle is False.

| **Option** | **Description** |
|------------|----------------|
| **Table properties** |  |
| **Partition by** | True / False Toggle- **True**: Enables the Column Dropdown and Textbox to add columns for partitioning - **False**: Disables the Dropdown and Textbox to add columns  |
|**Table Constraints**| True / False Toggle- **True**: Primary key and Foreign Key toggle will visible- **False** : Primary key and Foreign Key toggle will Disable |
|**Primary key**|Visible when Table Constraints is True options for Primary Key  - Primary key Name : Primary Key name is added in short - Primary key columns : Column Dropdown and Textbox to add columns box appear to select primary key  |
|**Foreign Key**|Visible when Table Constraints is True options for Foreign Key  - Foreign key attributes : Text box appear where we can add Foreign Key Name,Parent table name,Foreign key columns|
|**Other constraints**|True / False Toggle  Constraints : Text box appear where we can add constraints on selected columnOptions-ColumnName :  From column dropdown we can add columns-Expected Expression : can give specific Expression-ON VIOLATION : Two options can be selected from dropdown - Fail Update, Drop Row    |

### Schedule Options

Schedule Options is available only when Schedule Refresh toggle is True

| **Option** | **Description** |
|------------|----------------|
|**Task Schedule**| Options in Task Schedule- Periodic Schedule- CRON    |
|**Schedule refesh-time period**|Available when Task Schedule is Set to Periodic ScheduleOptions in Schedule refesh-time period-Every Hours-Every Days-Every Weeks |
|**Specific interval of periodic refresh(integer value)**|Available when Task Schedule is Set to Periodic Schedule |
|**CRON string**|Available when Task Schedule is Set to CRON|
|**CRON TIME ZONE**|Available when Task Schedule is Set to CRON |

## Deployment

### Initial Deployment
When deployed for the first time into an environment Materialized View will execute three stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Materialized View** | This stage will execute a CREATE OR REPLACE statement and create a Materialized View in the target environment |

### Redeployment

After the Materialized View has deployed for the first time into a target environment, subsequent deployments may result in either altering the Materialized View or recreating the Materialized View.

If a Materialized View is to be altered this will run the following stage:

#### Altering the Materialized View

The following config changes trigger ALTER statements:

1. Add schedule
2. Alter schedule
3. Drop schedule

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Materialized View** | Executes ALTER to modify parameters |

#### Recreating the Materialized View

If anything changes other than the configuration options specified above then the Materialized View will be recreated by running a CREATE OR REPLACE statement.

### Undeployment

If a Materialized View is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the Materialized View in the target environment will be dropped.

This is executed as a single stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Materialized View** | Removes the materialized view |

## Code

* [Node definition](https://github.com/coalesceio/databricks-Materialized-View/blob/main/nodeTypes/MaterializedView-368/definition.yml)
* [Create template](https://github.com/coalesceio/databricks-Materialized-View/blob/main/nodeTypes/MaterializedView-368/create.sql.j2)
* [Run template](https://github.com/coalesceio/databricks-Materialized-View/blob/main/nodeTypes/MaterializedView-368/run.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.1 | June 13, 2025 | Infer toggle removed |
| 1.0.0 | June 06, 2025 | Initial version of Databricks Materialized View  |

---

## Fabric - Base Node Types

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/fabric/fabric-base-node-types</code> <CopyButton textToCopy='@coalesce/fabric/fabric-base-node-types' />
Supported Platform: 
Latest Version: 1.0.0 (May 20, 2026)

data warehousedimension tabledimensional modelingeltfact tablestagingstar schematransformation

## Overview
Coalesce base Node Types to construct facts and dimensions in Fabric Platform

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/fabric/fabric-base-node-types</code> <CopyButton textToCopy='@coalesce/fabric/fabric-base-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Coalesce Base Node Types Package

The Coalesce Base Node Types Package includes:

* [Work](#work)
* [Persistent Stage](#persistent-stage)
* [Dimension](#dimension)
* [Fact](#fact)
* [View](#view)
* [Code](#code)

---

## Work

The Coalesce work node is a versatile node that allows you to develop and deploy a Work table/view in Microsoft Fabric.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Node Configuration

The Work node type has two configuration groups:

* [Node Properties](#work-node-properties)
* [Options](#work-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Work Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Work Options

You can create the node as:

* [Table](#work-options-table)
* [View](#work-options-view)

##### Work Options Table

| **Property** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: DISTINCT data is chosen for processing**False**: Only Select without distinct is executed|
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Work Options View

![image](https://github.com/user-attachments/assets/6c18d488-03df-4073-8932-75c6ae1e7d11)

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |

### Work Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/2dc81bb8-2285-46e1-8b93-ce7082800fc5)

### Work Deployment

#### Work Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Work Table** | This will execute a DROP statement and Drops a table in the target environment |
| **Create Work Table** | This will execute a CREATE statement and create a table in the target environment |

When deployed for the first time into an environment the Work node of materialization type view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Work View** | This will execute a DROP statement and Drops a view in the target environment |
| **Create Work View** | This will execute a CREATE statement and create a view in the target environment |

#### Work Redeployment

After the WORK node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.

#### Altering the Work Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \|**| Alter table statement is executed to perform the alter operation |
| **Rename clone Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Work View

If any of the following change are detected, then the table will be recreated by DROP View first and then CREATE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Work Undeployment

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Microsoft Fabric is dropped |

If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Work view from target environment |

## Persistent Stage 

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Node Configuration

The Persistent node type has two configuration groups:

* [Node Properties](#persistent-stage-node-properties)
* [Options](#persistent-stage-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Persistent Stage Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Options

![image](https://github.com/user-attachments/assets/21c63618-a311-4102-a3c0-ef783282b1ba)

| **Option** | **Description** |
|---------|-------------|
| **Create As** | Table is the only option at this time |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 |
| **Change tracking** | Required column for Type 2 |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Persistent Stage Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![pstage_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/84ce06c9-103c-4700-ad3d-2e8222995b31)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Persistent Stage Deployment

#### Persistent Stage Initial Deployment

When deployed for the first time into an environment the Persistent node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Persistent Stage Redeployment

After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

#### Altering the Persistent Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Rename Column** | Alter table statement is executed to perform the alter operation |
| **Rename Clone Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

### Persistent Stage Undeployment

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Microsoft Fabric is dropped |

---
## Dimension

The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Microsoft Fabric.

A dimension table or dimension entity is a table or entity in a star, snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

### Dimension Node Configuration

The Dimension node type has two configuration groups:

* [Node Properties](#dimension-node-properties)
* [Options](#dimension-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Dimension Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the DIMENSION will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Options

##### Dimension Table Options

| **Options** | **Description** |
|---------|-------------|
| **Create As** | Table or View |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Change tracking** | Required column for Type 2 Dimension |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |

### Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![Dimension_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/5c3df3b0-f56d-4276-a51f-22364206b3c3)

### Dimension Deployment

#### Dimension Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table will execute the Create Dimension Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Dimension Table** | This will execute a DROP statement and Drops a table in the target environment |
| **Create Dimension Table** | This will execute a CREATE statement and create a table in the target environment |
| **Drop Dimension View** | This will execute a DROP statement and Drops a view in the target environmentt |
| **Create Dimension View** | This will execute a CREATE statement and create a view in the target environment |

#### Dimension Redeployment

After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

#### Altering the Dimension Tables

A few types of column or table changes will result in an ALTER statement to modify the Dimension Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \|** | Alter table statement is executed to perform the alter operation |
| **Rename Clone Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Dimension Views

Any of the following changes to views will result in deleting and recreating the Dimension view.

* View defintion
* Renaming view results

### Dimension Undeployment

If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Microsoft Fabric is dropped |

If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension View in the target environment will be dropped.

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Dimension view from target environment. |
---
## Fact

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Microsoft Fabric.

A fact table or a fact entity is a table or entity in a star or snowflake schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

### Fact Node Configuration

The Fact node has two configuration groups:

* [Node Properties](#fact-node-properties)
* [Options](#fact-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Fact Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the FACT will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Fact Options

| **Options** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Business key** | Required column for Fact table creation |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: DISTINCT data is chosen for processing**False**: DISTINCT is not added in select query|
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

#### Fact View

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: DISTINCT data is chosen for processing**False**: DISTINCT is not added in select query|

### Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/e540d2d0-2623-4b99-a435-a26df5fe3306)

### Fact Deployment

#### Fact Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table will execute the Create Fact Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Fact Table** | This will execute a DROP statement and Drops a table in the target environment |
| **Create Fact Table** | This will execute a CREATE statement and create a table in the target environment |
| **Drop Fact View** | This will execute a DROP statement and Drops a view in the target environmentt |
| **Create Fact View** | This will execute a CREATE statement and create a view in the target environment |

#### Fact Redeployment

After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Fact Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \|** | Alter table statement is executed to perform the alter operation |
| **Rename Clone Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Fact Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Fact Undeployment

If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Microsoft Fabric is dropped |

---
## View 

The Coalesce View UDN is a versatile node that allows you to develop and deploy a View in Microsoft Fabric.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.

### View Node Configuration

The View node type has two configuration groups:

* [Node Properties](#view-node-properties)
* [Options](#view-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### View Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True/False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Distinct** | Toggle: True/False**True**: DISTINCT data is chosen for creating view**False**: DISTINCT is not added in select query for view creation
| **Pre-SQL** | SQL to execute before view creation|
| **Post-SQL** | SQL to execute after view creation |

### View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

### View Deployment

#### View Initial Deployment

When deployed for the first time into an environment the View node will execute the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE statement and create a View in the target environment |

#### View Redeployment

The subsequent deployment of View node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops existing view |
| **Create View** | Creates new view with updated definition |

#### View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the view from the environment |
---

#### Code

### Work Code

* [Node definition](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Work-147/definition.yml)
* [Create Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Work-147/create.sql.j2)
* [Run Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Work-147/run.sql.j2)

### Persistent Stage Code

* [Node definition](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/PersistentStage_10-150/definition.yml)
* [Create Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/PersistentStage_10-150/create.sql.j2)
* [Run Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/PersistentStage_10-150/run.sql.j2)

### Dimension Code

* [Node definition](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Dimension_10-148/definition.yml)
* [Create Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Dimension_10-148/create.sql.j2)
* [Run Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Dimension_10-148/run.sql.j2)

### Fact Code

* [Node definition](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Fact_10-149/definition.yml)
* [Create Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Fact_10-149/create.sql.j2)
* [Run Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/Fact_10-149/run.sql.j2)

### View Code

* [Node definition](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/View_10-153/definition.yml)
* [Create Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/View_10-153/create.sql.j2)
* [Run Template](https://github.com/coalesceio/fabric-nodes/blob/main/nodeTypes/View_10-153/run.sql.j2)

[Macros](https://github.com/coalesceio/fabric-nodes/blob/main/macros/macro-1.yml).

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.0 | May 20, 2026 | Initial release with basic functionality and core features support |

---

## Base Node Types Advanced Deploy

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/base-node-types-advanced-deploy</code> <CopyButton textToCopy='@coalesce/snowflake/base-node-types-advanced-deploy' />
Supported Platform: 
Latest Version: 3.0.0 (July 06, 2026)

alter tableclusterdata retentiondimension tabledimensional modelingfact tableforeign keyprimary keystagingtime traveltransformation

## Overview
The Coalesce Advanced Deploy Nodes are Node Types that allows you to develop and deploy objects in Snowflake.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/base-node-types-advanced-deploy</code> <CopyButton textToCopy='@coalesce/snowflake/base-node-types-advanced-deploy' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Snowflake - Base Node Types Advanced Deploy – Brief Summary

- **Fact nodes**  
  Represent measurable business events such as sales, transactions, or usage. These are the core tables used for reporting, KPIs, and trend analysis.

- **Dimension nodes**  
  Provide business context for facts, like customer, product, time, or location. They help slice and analyze facts in meaningful ways.

- **Factless fact nodes**  
  Capture business events that do not have numeric measures, such as attendance, eligibility, or process milestones. Useful for compliance, tracking, and operational analysis.

- **Persistent nodes**  
  Store curated, reusable data that remains stable over time. They act as trusted reference layers, reduce reprocessing, and ensure consistency across reports and teams.

- **Work nodes**  
  Temporary or intermediate processing layers used during transformations. They support complex logic and performance optimization but are not intended for direct business consumption.

- **External Sync**
    Integrate Snowflake objects managed by external pipelines into the Coalesce DAG for passive lineage and mapping. They allow you to reference and map data created outside of Coalesce without requiring the tool to own the DDL or DML.     

**Summary:**  
Together, these node types ensure data is accurate, reusable, scalable, and aligned with business reporting and decision-making needs.

----

## Nodetypes Config Matrix

| Category | Feature                         | Dim | Fact | Factless | Work | PStage | External Sync |
|----------|----------------------------------|-----|------|----------|------|--------|--------|
| Create   | Create As Table                  | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Create   | Create As Transient Table        | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Create   | Create As View                   | ✅  | ✅   | ⬜       | ✅   | ⬜     | ✅     |
| Create   | Create with Override SQL         | ✅  | ✅   | ⬜       | ✅   | ⬜     | ✅     |
| Create   | Primary Key                      | ✅  | ✅   | ⬜       | ⬜   | ⬜     | ⬜     |
| Create   | Cluster Key                      | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Create   | Deploy Execute                   | ⬜  | ⬜   | ⬜       | ⬜   | ⬜     | ✅     |
| Load     | MultiSource                      | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Load     | Insert Strategy                  | ✅  | ✅   | ✅       | ✅   | ✅     | ⬜     |
| Load     | Update Strategy                  | ✅  | ⬜   | ⬜       | ⬜   | ⬜     | ⬜     |
| Load     | Unmatched Record Strategy        | ✅  | ⬜   | ⬜       | ⬜   | ⬜     | ⬜     |
| Load     | Enable Source Soft Delete Handling | ⬜  | ✅   | ✅       | ⬜   | ⬜     | ⬜     |
| Load     | Source Delete Strategy           | ⬜  | ✅   | ✅       | ⬜   | ⬜     | ⬜     |
| Load     | Source Deleted Record Indicator  | ⬜  | ✅   | ✅       | ⬜   | ⬜     | ⬜     |
| Load     | Business Key                     | ✅  | ✅   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Last Modified Comparison         | ✅  | ✅   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Lookback Days                    | ⬜  | ✅   | ✅       | ⬜   | ⬜     | ⬜     |
| Load     | Change Tracking                  | ✅  | ⬜   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Delete Strategy(source DML-flag based)                  | ⬜  | ⬜   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Column that Identifies DML Operations(DELETE) | ⬜  | ⬜   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Delete Value | ⬜  | ⬜   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Exclude Columns from Merge       | ✅  | ✅   | ⬜       | ⬜   | ✅     | ⬜     |
| Load     | Truncate Before                  | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Load     | Distinct                         | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Load     | Group By All                     | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Load     | Order By                         | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Load     | Insert Zero Key Record           | ✅  | ⬜   | ⬜       | ⬜   | ⬜     | ⬜     |
| Load     | ASOF Join Options                | ⬜  | ⬜   | ⬜       | ✅   | ⬜     | ✅     |
| Load     | Methods                          | MERGEINSERT/UPDATE | MERGEINSERT | MERGE |INSERT | MERGEINSERT |INSERT |
| Load     | Run Execute                      | ⬜  | ⬜   | ⬜       | ⬜   | ⬜     | ✅     |
| Others   | Enable Tests                     | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Others   | Pre-SQL                          | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |
| Others   | Post-SQL                         | ✅  | ✅   | ✅       | ✅   | ✅     | ✅     |

---

## Base Node Types Advanced Deploy

The Coalesce Base Node Types Package includes:

* [Work Advanced Deploy](#work-advanced-deploy)
* [Persistent Stage Advanced Deploy](#persistent-stage-advanced-deploy)
* [Dimension Advanced Deploy](#dimension-advanced-deploy)
* [Fact Advanced Deploy](#fact-advanced-deploy)
* [Factless Fact Advanced Deploy](#factless-fact-advanced-deploy)
* [External Sync](#external-sync)
* [Code](#code)

---

## Work Advanced Deploy

The Coalesce Work Node is a versatile node that allows you to develop and deploy a Work table/view in Snowflake.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Advanced Deploy Node Configuration

The Work node type has two configuration groups:

* [Node Properties](#work-advanced-deploy-node-properties)
* [Options](#work-advanced-deploy-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/45d22ea5-32ca-49f5-a464-b266cb29b516)

#### Work Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Work Advanced Deploy Options

You can create the node as:

* [Table](#work-advanced-deploy-create-as-table)
* [View](#work-advanced-deploy-create-as-view)
* [Transient Table](#work-advanced-deploy-create-as-transient-table)

##### Work Advanced Deploy Create as Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **ASOF Join** | Toggle: True/False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Work Advanced Deploy Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **ASOF Join** | Toggle: True/False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |

##### Work Advanced Deploy Create as Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Transient Table|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **ASOF Join** | Toggle: True/False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### ASOF Join Options
| **Setting** | **Description** |
|---------|-------------|
| **Match Condition** | Toggle: True/False  Match Condition Clause from Snowflake ASOF join  **True**: Allows you to specify the Match Condtion.- **Right Table Storage Location**: Add right table storage location- **Right Table Name**: Add name of the right table- **Match Condition**: Add a match condition in the format "Left Table Name"."Column Name" Condition Operator "Right Table Name"."Column Name" **False** : No Match Condition Added|
| **On** | Toggle: True/False ON Clause with Match Condition from Snowflake ASOF join.Using will be invisible  **True**: Allows you to add the ON Clause. **ON Condition**: Add a match condition in the format "Left Table Name"."Column Name" = "Right Table Name"."Column Name"  **False**: No ON Clause Added.Using will be visible|
| **Using** | Toggle: True/False Using Clause with Match Condition from Snowflake ASOF join.On will be invisible  **True**: Allows you to add the Using Clause. **Using Column Name** : Add a Column Name for Using clause **False**: No Using Clause Added.On will be visible|

### Work Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/7acff10b-8845-4c44-851a-b7a0bc7eaf41)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Work Advanced Deploy ASOF Join

After selecting options for ASOF Join,Click on Generate join, use the 'Copy To Editor' to add the new ASOF join.

### Work Advanced Deploy Deployment

#### Work Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Work View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Work Advanced Deploy Redeployment

After the WORK node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.

#### Altering the Work Tables and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation|

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Work Advanced Deploy Recreating the Work Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with updated definition |

#### Work Advanced Deploy Drop and Recreate Work View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Work table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Work view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Work table/transient table |

> 📘 **Materialization Work Node**
>
> When the materialization type of Work node is changed from table/transient table to View and use Override Create SQL for view creation. This ensures that the following change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/0296abf8-0747-4ae8-8478-0782e5e2e545)

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Work Advanced Deploy Undeployment

If a Work Node of materialization type table/view/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

---

## Persistent Stage Advanced Deploy

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Advanced Deploy Node Configuration

The Persistent node type has two configuration groups:

* [Node Properties](#persistent-stage-advanced-deploy-node-properties)
* [Options](#persistent-stage-advanced-deploy-options)

#### Persistent Stage Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** |  If TRUE the node will be deployed or redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Advanced Deploy Options

You can create the node as:

* [Table](#persistent-stage-table)
* [Transient Table](#persistent-stage-transient-table)

##### Persistent Stage Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Business key** | Required column for both Type 1 and Type 2 .**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution.|
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 |
| **Delete Strategy** | Visible only when a Business Key is configured. **NO DELETE**: Deleted records are not processed.**SOFT DELETE**:  Deleted records are kept in the target but marked as inactive.**HARD DELETE**: Deleted records and all their version history are permanently removed from the target. |
| **Column that Identifies DML Operations** | The column in the source table that tells whether a record is an Insert, Update or Delete. |
| **Delete Value** | The value in the DML column that signals a record should be deleted. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Persistent Stage Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Business key** | Required column for both Type 1 and Type 2 |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 |
| **Delete Strategy** | Visible only when a Business Key is configured. **NO DELETE**: Deleted records are not processed.**SOFT DELETE**:  Deleted records are kept in the target but marked as inactive.**HARD DELETE**: Deleted records and all their version history are permanently removed from the target. |
| **Column that Identifies DML Operations** | The column in the source table that tells whether a record is an Insert, Update or Delete. |
| **Delete Value** | The value in the DML column that signals a record should be deleted. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Persistent Stage Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![pstage_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/4c3a4c11-b837-48d2-9ba9-1b0797a8aed9)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Persistent Stage Advanced Deploy Deployment

#### Persistent Stage Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Persistent node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Stage Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Persistent Stage Advanced Deploy Redeployment

After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

#### Altering the Persistent Tables/Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | ALTER table statement is executed to perform the alter operation accordingly |

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Drop and Recreate Persistent Stage Table/Transient Table

When the materialization type of Persistent stage node is changed from table to transient table or transient table to table, the below stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes existing table |
| **Create or Replace Persistent stage table/transient table** | Creates new table with updated configuration |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Persistent Stage Advanced Deploy Undeployment

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

This is executed in the stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table or View** | Removes the table from the environment |

---

## Dimension Advanced Deploy

The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Snowflake.

A dimension table or dimension entity is a table or entity in a star, snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

### Dimension Advanced Deploy Node Configuration

* [Node Properties](#dimension-advanced-deploy-node-properties)
* [Options](#dimension-advanced-deploy-options)

#### Dimension Advanced Deploy Node Properties

| **Property** | Description |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** |  If TRUE the node will be deployed or redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Advanced Deploy Options

You can create the node as:

* [Table](#dimension-advanced-deploy-table)
* [Transient Table](#dimension-advanced-deploy-transient-table)
* [View](#dimension-advanced-deploy-view)

##### Dimension Advanced Deploy Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention**True**:  Zero Key Record Options enabled.**False**: Zero Key Record not added|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Primary key** | Allows you to specify one or more columns based on which primary constraint is set on the table. **Primary Key Name**: Primary key constraint name. If not specified defaults to **pk_TABLENAME** |
| **Update Strategy**| Options : MERGE,INSERT/UPDATE - **MERGE**: Uses a single MERGE statement to handle both insert and update operations based on matching keys.- **INSERT/UPDATE**: Separately executes UPDATE for existing records and INSERT for new ones using custom logic.For preferred choice,refer [Preferences](#preferences)|
| **Business key** | Required column for Type 1 and Type 2 Dimensions .**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 Dimension |
| **Unmatched Record Strategy** | Available for single source nodes with Merge as update strategy- **NO DELETE**: An option introduced to ensure existing data flows remain intact and unchanged, preventing any delete operation on the target table.- **SOFT DELETE**: Marks records as logically deleted (isSystemCurrentFlag = 0) while retaining the history.- **HARD DELETE**: Permanent removal of records from the target table. |
| **Exclude Columns from Merge** | Available only for SCD type 1 Merges. Allows you to specify one or more columns that are excluded during both the **comparison** (matching) and **updating** phases of the MERGE statement. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Cluster key** | **True**: Allows you to specify the column based on which clustering is to be done Allow Expressions Cluster Key: Allows to add an expression to the specified cluster key**False**: No clustering done |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. *False**: Sort options invisible |
| **Zero Key Record Options** |Add custom zero key record values for :  -Default Surrogate Key Value  -Default String Value  -Default Date Value (Date Format DD-MM-YYYY) -Default Timestamp Value (Timestamp Format YYYY-MM-DD HH24:MI:SS.FF)  -Default Boolean Value|
| **Advanced Zero Key Record Options** | Toggle: True/False**True**: Select Columns and the default value of the column for zero key record **False**: Advanced Zero Key Record Options not enabled|
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension Advanced Deploy Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Transient Table |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Primary key** | Allows you to specify one or more columns based on which primary constraint is set on the table. **Primary Key Name**: Primary key constraint name. If not specified defaults to **pk_TABLENAME** |
| **Update Strategy**| Options : MERGE,INSERT/UPDATE - **MERGE**: Uses a single MERGE statement to handle both insert and update operations based on matching keys.- **INSERT/UPDATE**: Separately executes UPDATE for existing records and INSERT for new ones using custom logic.For preferred choice,refer [Preferences](#preferences)|
| **Unmatched Record Strategy** | Available for single source nodes with Merge as update strategy- **NO DELETE**: An option introduced to ensure existing data flows remain intact and unchanged, preventing any delete operation on the target table.- **SOFT DELETE**: Marks records as logically deleted (isSystemCurrentFlag = 0) while retaining the history.- **HARD DELETE**: Permanent removal of records from the target table. |
| **Exclude Columns from Merge** | Available only for SCD type 1 Merges. Allows you to specify one or more columns that are excluded during both the **comparison** (matching) and **updating** phases of the MERGE statement. |
| **Business key** | Required column for Type 1 and Type 2 Dimensions |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 Dimension |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Cluster key** | **True**: Allows you to specify the column based on which clustering is to be done Allow Expressions Cluster Key: Allows to add an expression to the specified cluster key**False**: No clustering done |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension Advanced Deploy View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View |
| **Override Create SQL** | Toggle: True/False**True**: Custom Create SQL**False**: Generated view SQL |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Business key** | Required column for Type 1 and Type 2 Dimensions |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |

## Preferences

* With Update Strategy config option in Dimension,we can choose either to perform a MERGE or INSERT/UPDATE operation to create various SCDs(Slowly Changing Dimensions).
* Insert/Update operations scale well up to 10 million rows but MERGE is recommended for its superior execution performance.
* For large datasets,MERGE operation is recommended.
  
### Dimension Advanced Deploy Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![Dimension_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/d7f7dcc5-fe17-447c-9e6e-78fb1b8924ab)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Dimension Advanced Deploy Deployment

#### Dimension Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table or view will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Dimension View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Dimension Advanced Deploy Redeployment

After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

#### Altering the Dimension Tables and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Recreating the Dimension Views

The subsequent deployment of Dimension node of materialization type view with changes in view definition, adding table description or renaming view results in recreating the dimension view.

#### Drop and Recreate Dimension View/Table/Transient

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view Create Dimension table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Dimension view |
| **Table to transient table or vice versa** |  Drop table/transient table Create Dimension table/transient table |

> 📘 **Materialization type of Dimension node**
>
> When the materialization type of Dimension node is changed from table/transient table to View and use Override Create SQL for view creation to ensure that the below change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/07bba9e6-802f-45d0-9a39-5f6bd619d53d)

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dimension Advanced Deploy Undeployment

If a Dimension Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Fact Advanced Deploy

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Snowflake.

A fact table or a fact entity is a table or entity in a star or snowflake schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

### Fact Advanced Deploy Node Configuration

* [Node Properties](#fact-advanced-deploy-node-properties)
* [Options](#fact-advanced-deploy-options)

#### Fact Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location**| Storage Location where the view will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled**| If **TRUE**: node will be deployed or redeployed when changes are detectedIf **FALSE**: node will not be deployed or will be dropped during redeployment |

#### Fact Advanced Deploy Options

You can create the node as:

* [Table](#fact-advanced-deploy-table)
* [Transient Table](#fact-advanced-deploy-transient-table)
* [View](#fact-advanced-deploy-view)

##### Fact Advanced Deploy Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Primary key** | Allows you to specify one or more columns based on which primary constraint is set on the table. **Primary Key Name**: Primary key constraint name. If not specified defaults to **pk_TABLENAME** |
| **Business key** | Required column for Type 1 and Type 2 Dimensions .**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Lookback Days** | Number of days to look back from the latest target record for incremental load. Default is 3. Available only when Last Modified Based Incremental Load is enabled with a TIMESTAMP column. |
| **Enable Source Soft Delete Handling** |Toggle: True/False **True:** Records flagged as deleted in source are hard deleted from target. **False:** No delete handling.  Available when no business key is present or multisource is enabled. |
| **Source Delete Strategy** | Available for single source nodes with a business key. **NO DELETES**: Deleted records in source are ignored. **HARD DELETED**: Records that no longer exist in source are deleted from target  **SOFT DELETED**: Records flagged as deleted via a flag column are deleted from target. |
| **Source Deleted Record Indicator** | The flag column in source that marks a record as deleted. Visible when Soft Delete is selected or multisource with soft delete handling is enabled. **Note:** Only BOOLEAN data type is supported for this column. |
| **Exclude Columns from Merge** | Available only when business key is chosen. Allows you to specify one or more columns that are excluded during both the **comparison** (matching) and **updating** phases of the MERGE statement. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Cluster key** | **True**: Allows you to specify the column based on which clustering is to be done Allow Expressions Cluster Key: Allows to add an expression to the specified cluster key**False**: No clustering done |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Fact Advanced Deploy Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Transient Table |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Primary key** | Allows you to specify one or more columns based on which primary constraint is set on the table. **Primary Key Name**: Primary key constraint name. If not specified defaults to **pk_TABLENAME** |
| **Business key** | Required column for Type 1 and Type 2 Dimensions |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Lookback Days** | Number of days to look back from the latest target record for incremental load. Default is 3. Available only when Last Modified Based Incremental Load is enabled with a TIMESTAMP column. |
| **Enable Source Soft Delete Handling** |Toggle: True/False **True:** Records flagged as deleted in source are hard deleted from target. **False:** No delete handling.  Available when no business key is present or multisource is enabled. |
| **Source Delete Strategy** | Available for single source nodes with a business key. **NO DELETES**: Deleted records in source are ignored. **HARD DELETED**: Records that no longer exist in source are deleted from target  **SOFT DELETED**: Records flagged as deleted via a flag column are deleted from target. |
| **Source Deleted Record Indicator** | The flag column in source that marks a record as deleted. Visible when Soft Delete is selected or multisource with soft delete handling is enabled. **Note:** Only BOOLEAN data type is supported for this column. |
| **Exclude Columns from Merge** | Available only when business key is chosen. Allows you to specify one or more columns that are excluded during both the **comparison** (matching) and **updating** phases of the MERGE statement. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Cluster key** | **True**: Allows you to specify the column based on which clustering is to be done Allow Expressions Cluster Key: Allows to add an expression to the specified cluster key**False**: No clustering done |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Fact Advanced Deploy View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View |
| **Override Create SQL** | Toggle: True/False**True**: Executes custom Create SQL**False**: Creates view based on chosen options |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |

### Fact Advanced Deploy Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/2f765410-67b4-437e-a281-6169901c382c)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Fact Advanced Deploy Deployment

#### Fact Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Fact View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Fact Advanced Deploy Redeployment

After the Fact node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Fact Tables/Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Recreating the Fact Advanced Deploy Views

The subsequent deployment of Fact node of materialization type view with changes in view definition, adding table description or renaming view results recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with updated definition |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Fact Advanced Deploy Undeployment

If a Fact Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

#### Drop and recreate Fact View/Table/Transient

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view Create Fact table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Fact view |
| **Table to transient table or vice versa** |  Drop table/transient table Create Fact table/transient table |

> 📘 **Materialization Type of Dimension node**
>
> When the materialization type of Dimension node is changed from table/transient table to View and use Override Create SQL for view creation, ensure that the below change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/7cf9e0a4-6832-452c-ba5e-cedad42d1c40)

---

## Factless Fact Advanced Deploy

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Snowflake.

A factless fact table is used to record events or situations that have no measures, and it has the same level of detail as the dimensions.

### Factless Fact Advanced Deploy Node Configuration

* [Node Properties](#factless-fact-advanced-deploy-node-properties)
* [Options](#factless-fact-advanced-deploy-options)

#### Factless Fact Advanced Deploy Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location**| Storage Location where the view will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled**| If **TRUE**: node will be deployed or redeployed when changes are detectedIf **FALSE**: node will not be deployed or will be dropped during redeployment |

#### Factless Fact Advanced Deploy Options

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table or Transient Table |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done.  **Note:** When Last Modified is enabled, a validation stage is triggered to ensure the timestamp column contains no NULL values. Any NULL values detected will result in an error, immediately halting execution. |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Lookback Days** | Number of days to look back from the latest target record for incremental load. Default is 3. Available only when Last Modified Based Incremental Load is enabled with a TIMESTAMP column. |
| **Enable Delete Handling** |Toggle: True/False **True:** Enables delete handling for source deleted records. **False:** No delete handling. |
| **Source Delete Strategy** | Available for single source nodes with a business key. **NO DELETES**: Deleted records in source are ignored. **HARD DELETED**: Records that no longer exist in source are deleted from target  **SOFT DELETED**: Records flagged as deleted via a flag column are deleted from target. |
| **Source Deleted Record Indicator** | The flag column in source that marks a record as deleted. Visible when Soft Delete is selected or multisource with soft delete handling is enabled. **Note:** Only BOOLEAN data type is supported for this column. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Factless Fact Advanced Deploy Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/e2d362ef-e34e-4153-910a-cc84a6a0ab4a)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Factless Fact Advanced Deploy Deployment

#### Factless Fact Advanced Deploy Initial Deployment

When deployed for the first time into an environment the Factless Fact node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |

#### Factless Fact Advanced Deploy Redeployment

After the Fact node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Factless Fact Tables/Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation accordingly |

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Drop and Recreate Factless Fact Table/Transient Table
  
When the materialization type of Factless Fact node is changed from table to transient table or transient table to table, the below stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes existing table |
| **Create Factless Fact table/transient table** | Creates new table with updated configuration |

-----------------

## Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Factless Fact Advanced Deploy Undeployment

If a Fact Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

### Redeployment With No Changes
 
If the nodes are redeployed with no changes compared to previous deployment, then no stages are executed

---
## External Sync

The External Sync node integrates externally managed Snowflake tables and views into your Coalesce DAG for lineage and mapping purposes.
It allows you to synchronize metadata from objects created by external pipelines without requiring Coalesce to own the DDL. With independent toggles for deployment and execution, this node acts as a flexible reference point that can safely transition from a passive lineage object to a fully managed pipeline component.

### External Sync Node Configuration

The External Sync node type has two configuration groups:

* [Node Properties](#external-sync-node-properties)
* [Options](#external-sync-options)
* [Create Options](#external-sync-create-options)
* [Load Options](#external-sync-load-options)

#### External Sync Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the node will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### External Sync Options

You can create the node as:

* Table
* View
* Transient Table

#### External Sync Options.

| **Setting** | **Description** |
|---------|-------------|
| **Deploy Execute** | Toggle: True/False**True**: Coalesce manages the DDL. Full creation, alteration, or dropping of the Snowflake object occurs during deployment.**False**: No DDL is executed. The node acts as a passive lineage reference in the DAG. |
| **Run Execute** | Toggle: True/False**True**: Coalesce manages the DML. Data loading (Insert/Truncate) and tests are performed during pipeline runs.**False**: No DML is executed. No data processing or testing occurs during pipeline runs. | 

#### External Sync Create Options.

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Dropdown: Select the desired materialization.  - Table  - Transient Table  - View.|
| **Cluster key** | Toggle: True/False  If the dimension is clustered or not.  **True**: Allows you to specify the column based on which clustering is to be done.- **Allow Expressions Cluster Key**: Allows to add an expression to the specified cluster key **False**:No clustering done|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### External Sync Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **ASOF Join** | Toggle: True/False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### ASOF Join Options
| **Setting** | **Description** |
|---------|-------------|
| **Match Condition** | Toggle: True/False  Match Condition Clause from Snowflake ASOF join  **True**: Allows you to specify the Match Condtion.- **Right Table Storage Location**: Add right table storage location- **Right Table Name**: Add name of the right table- **Match Condition**: Add a match condition in the format "Left Table Name"."Column Name" Condition Operator "Right Table Name"."Column Name" **False** : No Match Condition Added|
| **On** | Toggle: True/False ON Clause with Match Condition from Snowflake ASOF join.Using will be invisible  **True**: Allows you to add the ON Clause. **ON Condition**: Add a match condition in the format "Left Table Name"."Column Name" = "Right Table Name"."Column Name"  **False**: No ON Clause Added.Using will be visible|
| **Using** | Toggle: True/False Using Clause with Match Condition from Snowflake ASOF join.On will be invisible  **True**: Allows you to add the Using Clause. **Using Column Name** : Add a Column Name for Using clause **False**: No Using Clause Added.On will be visible|

#### Note
If changes occur in the source column list, use the **Resync** option to update the node's structure from Snowflake.

### External Sync Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/7acff10b-8845-4c44-851a-b7a0bc7eaf41)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### External Sync ASOF Join

After selecting options for ASOF Join,Click on Generate join, use the 'Copy To Editor' to add the new ASOF join.

### External Sync Deployment

#### External Sync Initial Deployment

When deployed for the first time into an environment the External Sync node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### External Sync Redeployment

After the External Sync node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the External Sync Table or recreating the External Sync table.

#### Altering the External Sync Tables and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the External Sync Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation|

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### External Sync Recreating Views

The subsequent deployment of External Sync node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with updated definition |

#### External Sync Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create  view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace table/transient table |

> 📘 **Materialization External Sync Node**
>
> When the materialization type of External Sync node is changed from table/transient table to View and use Override Create SQL for view creation. This ensures that the following change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/0296abf8-0747-4ae8-8478-0782e5e2e545)

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### External Sync Undeployment

If a External Sync Node of materialization type table/view/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the External Sync Table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Table | Table | 1. Warning (if applicable)2. Metadata Update(if applicable)3. Alter |
| Transient Table | TransientTable | 1. Warning (if applicable)2. Metadata Update(if applicable)3. Alter |
| View | View | 1. Warning (if applicable)2. Create |
| Any Other | Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | View | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Transient Table | 1. Warning (if applicable)2. Drop  3. Create |

Review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)(Pending Release) |

--------------

## Code

### Work Advanced Deploy Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/WorkAdvancedDeploy-179/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/WorkAdvancedDeploy-179/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/WorkAdvancedDeploy-179/run.sql.j2) |

### Persistent Stage Advanced Deploy Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/PersistentStageAdvancedDeploy-391/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/PersistentStageAdvancedDeploy-391/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/PersistentStageAdvancedDeploy-391/run.sql.j2) |

### Dimension Advanced Deploy Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/DimensionAdvancedDeploy-388/run.sql.j2) |

### Fact Advanced Deploy Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactAdvancedDeploy-389/run.sql.j2) |

### Factless Fact Advanced Deploy Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-390/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-390/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/FactlessFactAdvancedDeploy-390/run.sql.j2) |

### External Sync

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/ExternalSync-678/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/ExternalSync-678/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/nodeTypes/ExternalSync-678/run.sql.j2) |

[Macros](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/blob/main/macros/macro-1.yml)

  

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 3.0.0 | July 06, 2026 | NM-245 New Node Type - External Sync |
| 2.5.7 | June 19, 2026 | The Dimension Adv Node was failing for the Merge + Soft Delete + SCD2 combination; the related bugs  |
| 2.5.5 | June 05, 2026 | Add Incremental Loading and Soft Delete Handling to Fact adv and factless fact adv deploy. Credit to Ryan Schlenz - https://github.com/RyanSchlenz/Multisource-Incremental-Nodes |
| 2.5.4 | May 27, 2026 | NULL handling in Last Modified update and remove treatNULLasCurrentTimestamp - fixed |
| 2.5.3 | May 07, 2026 | Zerokey bug fix |
| 2.5.2 | May 06, 2026 | Add delete option to Persistent Stage node (Snowflake) |
| 2.5.1 | April 22, 2026 | Metadata Stage fix for proper handling of Jinja syntax (&#x7B;&#x7B; &#x7D;&#x7D;, &#x7B;# #&#x7D;, &#x7B;% %&#x7D;) in related text inputs |
| 2.5.0 | February 27, 2026 | get_clause() macro adapted to support column names with keywords in it |
| 2.4.0 | February 18, 2026 | Node Type Switching supported in Advanced Deploy Templates |
| 2.3.1 | December 22, 2025 | Added Last Modified Comparison to Dimension,Pstage,Fact and bugs fixed |
| 2.3.0 | December 18, 2025 | Added Last Modified Comparison to Dimension,Pstage,Fact |
| 2.2.1 | November 18, 2025 | Added an else block to the RUN template to handle cases where the materializationType is View |
| 2.2.0 | November 13, 2025 | Added Primary Key, Delete Strategy, Merge Exclude Features |
| 2.1.1 | August 26, 2025 | Metadata Updates - Block Comment Changes |
| 2.1.0 | August 20, 2025 | Metadata Refresh during deployment - Added |
| 2.0.1 | July 29, 2025 | Join clause update for job refresh |
| 2.0.0 | July 02, 2025 | Removal of deploy phases from stage function across all node types |

---

## Base Node Types(Package)

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/base-node-types</code> <CopyButton textToCopy='@coalesce/snowflake/base-node-types' />
Supported Platform: 
Latest Version: 1.6.2 (June 02, 2026)

data warehousedimension tabledimensional modelingeltfact tablestagingstar schematransformation

## Overview
Coalesce base Node Types to construct facts and dimensions.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/base-node-types</code> <CopyButton textToCopy='@coalesce/snowflake/base-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Coalesce Base Node Types Package

The Coalesce Base Node Types Package includes:

* [Work](#work)
* [Persistent Stage](#persistent-stage)
* [Dimension](#dimension)
* [Fact](#fact)
* [Factless Fact](#factless-fact)
* [View](#view)
* [Code](#code)

---

## Work

The Coalesce Work Node lets you develop and deploy a Work table or view in Snowflake.

A Work Node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

### Work Node Configuration

The Work Node type has two configuration groups:

* [Node Properties](#work-node-properties)
* [Options](#work-options)

![Work Node configuration](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Work Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Work table or view will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Work Options

You can create the Node as:

* [Table](#work-options-table)
* [View](#work-options-view)

##### Work Options Table

![Work_options_table1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/f5643f40-fb37-4182-a11b-577bdc3e8f8d)

| **Property** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: single-source Node or multiple sources combined using a join |
| **Truncate Before** | Toggle: True or FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is not truncated before data load |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True or False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **ASOF Join** | Toggle: True or False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Work Options View

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True or False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **ASOF Join** | Toggle: True or False**True**: ASOF Join Options will be visible. **False**: ASOF Join Options will be invisible |

##### ASOF Join Options

| **Setting** | **Description** |
|---------|-------------|
| **Match Condition** | Toggle: True or False  Match Condition Clause from Snowflake ASOF join  **True**: Allows you to specify the match condition.- **Right Table Storage Location**: Add right table storage location- **Right Table Name**: Add name of the right table- **Match Condition**: Add a match condition in the format "Left Table Name"."Column Name" Condition Operator "Right Table Name"."Column Name" **False** : No Match Condition Added|
| **On** | Toggle: True or False  ON Clause with Match Condition from Snowflake ASOF join. **Using** will be invisible  **True**: Allows you to add the ON Clause. **ON Condition**: Add a match condition in the format "Left Table Name"."Column Name" = "Right Table Name"."Column Name"  **False**: No ON Clause Added. **Using** will be visible |
| **Using** | Toggle: True or False  Using Clause with Match Condition from Snowflake ASOF join. **On** will be invisible  **True**: Allows you to add the Using Clause. **Using Column Name** : Add a Column Name for Using clause **False**: No Using Clause Added. **On** will be visible |

### Work Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the Coalesce App.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/2dc81bb8-2285-46e1-8b93-ce7082800fc5)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Work ASOF Joins
After selecting options for ASOF Join, click **Generate join**, then use **Copy To Editor** to add the new ASOF join.

### Work Deployment

#### Work Initial Deployment

When deployed for the first time into an Environment the Work Node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table** | This will execute a CREATE OR REPLACE statement and create a table in the target Environment |
| **Create Work View** | This will execute a CREATE OR REPLACE statement and create a view in the target Environment |

#### Work Redeployment

After the Work Node with materialization type table has been deployed for the first time into a target Environment, subsequent deployments may result in either altering the Work Table or recreating the Work table.

#### Altering the Work Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target Environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Work Tables

If any of the following changes are detected, then the table will be recreated using a CREATE OR REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Work Views

The subsequent deployment of the Work Node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Removing a Work Node

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the WorkTable in the target Environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the WorkView in the target Environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Work view from the target Environment |

---

## Persistent Stage

The Persistent Stage Node serves as an intermediary object and is commonly used to persist data across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

### Persistent Stage Node Configuration

The Persistent Stage Node type has two configuration groups:

* [Node Properties](#persistent-stage-node-properties)
* [Options](#persistent-stage-options)

![Persistent Stage Node configuration](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Persistent Stage Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Persistent Stage table will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Persistent Stage Options

| **Option** | **Description** |
|---------|-------------|
| **Create As** | Table is the only option at this time |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled, timestamp-based/ ID based CDC is performed.If the chosen column is timestamp-based, the test condition validates the source data for NULL values in the selected Last Modified column before the merge operation. The merge is blocked if NULL values are found and proceeds only after they are resolved**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 |
| **Delete Strategy** | Visible only when a Business Key is configured. **NO DELETE**: Deleted records are not processed.**SOFT DELETE**:  Deleted records are kept in the target but marked as inactive.**HARD DELETE**: Deleted records and all their version history are permanently removed from the target. |
| **Column that Identifies DML Operations** | The column in the source table that tells whether a record is an Insert, Update or Delete. |
| **Delete Value** | The value in the DML column that signals a record should be deleted. |
| **Truncate Before** | Toggle: True or FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is not truncated before data load |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True or False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Persistent Stage Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the Coalesce App.

![pstage_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/84ce06c9-103c-4700-ad3d-2e8222995b31)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Persistent Stage Deployment

#### Persistent Stage Initial Deployment

When deployed for the first time into an Environment the Persistent Stage Node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Persistent Table** | This will execute a CREATE OR REPLACE statement and create a table in the target Environment |

#### Persistent Stage Redeployment

After the Persistent Stage Node has been deployed for the first time into a target Environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

#### Altering the Persistent Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target Environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Persistent Tables

If any of the following changes are detected, then the table will be recreated using a CREATE OR REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

### Removing a Persistent Stage Node

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the Persistent Table in the target Environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Drops table |
| **Drop Table or View** | Removes the table |

---

## Dimension

The Coalesce Dimension UDN lets you develop and deploy a Dimension table in Snowflake.

A dimension table or dimension entity is a table or entity in a star, snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

### Dimension Node Configuration

The Dimension Node type has two configuration groups:

* [Node Properties](#dimension-node-properties)
* [Options](#dimension-options)

![Dimension Node configuration](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Dimension Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the dimension table or view will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Dimension Options

##### Dimension Table Options

| **Options** | **Description** |
|---------|-------------|
| **Create As** | Table or View |
| **Insert Zero Key Record** | Toggle: True or FalseInsert Zero Key Record to Dimension**True**:  Zero Key Record Options enabled.**False**: Zero Key Record not added|
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled, timestamp-based/ ID based CDC is performed.If the chosen column is timestamp-based, the test condition validates the source data for NULL values in the selected Last Modified column before the merge operation. The merge is blocked if NULL values are found and proceeds only after they are resolved**False**:Regular CDC based on Change tracking columns is performed |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2 Dimension |
| **Truncate Before** | Toggle: True or FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is not truncated before data load |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True or False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Zero Key Record Options** |Add custom zero key record values for :  -Default Surrogate Key Value  -Default String Value  -Default Date Value (Date Format DD-MM-YYYY) -Default Timestamp Value (Timestamp Format YYYY-MM-DD HH24:MI:SS.FF)  -Default Boolean Value|
| **Advanced Zero Key Record Options** | Toggle: True or False**True**: Select Columns and the default value of the column for zero key record **False**: Advanced Zero Key Record Options not enabled|
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Dimension View Options

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True or False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the Coalesce App.

![Dimension_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/5c3df3b0-f56d-4276-a51f-22364206b3c3)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Dimension Deployment

#### Dimension Initial Deployment

When deployed for the first time into an Environment the Dimension Node of materialization type table will execute the Create Dimension Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Table** | This will execute a CREATE OR REPLACE statement and create a table in the target Environment |
| **Create Dimension View** | This will execute a CREATE OR REPLACE statement and create a view in the target Environment |

#### Dimension Redeployment

After the Dimension Node of materialization type table has been deployed for the first time into a target Environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

#### Altering the Dimension Tables

A few types of column or table changes will result in an ALTER statement to modify the Dimension Table in the target Environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Dimension Tables

If any of the following changes are detected, then the table will be recreated using a CREATE OR REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Dimension Views

Any of the following changes to views will result in deleting and recreating the Dimension view.

* View definition
* Adding table description
* Renaming the view

### Removing a Dimension Node

If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the Dimension Table in the target Environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the Dimension View in the target Environment will be dropped.

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Dimension view from the target Environment. |

---

## Fact

The Coalesce Fact UDN lets you develop and deploy a Fact table in Snowflake.

A fact table or a fact entity is a table or entity in a star or snowflake schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

### Fact Node Configuration

The Fact Node has two configuration groups:

* [Node Properties](#fact-node-properties)
* [Options](#fact-options)

![Fact Node configuration](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### Fact Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the fact table will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Fact Options

| **Options** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Business key** | Required column for Fact table creation.**Note:** Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled, timestamp-based/ ID based CDC is performed.If the chosen column is timestamp-based, the test condition validates the source data for NULL values in the selected Last Modified column before the merge operation. The merge is blocked if NULL values are found and proceeds only after they are resolved**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Truncate Before** | Toggle: True or FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is not truncated before data load |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True or False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

#### Fact View

![Work_options_view1](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/7881c46e-0424-46ca-9a89-d111b5dbc379)

| **Setting** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True or False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible except 'Enable Tests'**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |

### Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce App.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/e540d2d0-2623-4b99-a435-a26df5fe3306)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Fact Deployment

#### Fact Initial Deployment

When deployed for the first time into an Environment the Fact Node of materialization type table will execute the Create Fact Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target Environment |
| **Create Fact View** | This will execute a CREATE OR REPLACE statement and create a view in the target Environment |

#### Fact Redeployment

After the Fact Node has been deployed for the first time into a target Environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

#### Altering the Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Fact Table in the target Environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Fact Tables

If any of the following changes are detected, the Fact table is recreated using `CREATE OR REPLACE`:

* Join clause
* Adding transformation
* Change in business key
* Changes in configuration like adding distinct, group by, or order by

One of the following stages is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

#### Recreating the Fact Views

The subsequent deployment of the Fact Node of materialization type view with changes in view definition, adding table description, or renaming the view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Removing a Fact Node

If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the Fact Table in the target Environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

---

## Factless Fact

The Coalesce Factless Fact UDN lets you develop and deploy a factless fact table in Snowflake.

A factless fact table is used to record events or situations that have no measures, and it has the same level of detail as the dimensions.

### Factless Fact Node Configuration

The Factless Fact Node has two configuration groups:

* [Node Properties](#factless-fact-node-properties)
* [Options](#factless-fact-options)

#### Factless Fact Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the factless fact table will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Factless Fact Options

![Factless_options](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/bf2430c4-2a62-4195-a817-e702461b2d14)

| **Options** | **Description** |
|---------|-------------|
| **Create As** | Table is the only option at this time |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Truncate Before** | Toggle: True or FalseThis determines whether a table will be truncated before data load. **True**:Truncate table stage gets executed**False**: Table is not truncated before data load |
| **Enable tests** | Toggle: True or FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Order By** | Toggle: True or False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Factless Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce App.

![fact_join](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/fc432c42-0315-4fb5-8f93-7836190086ff)

> 📘 **Specify Group by and Order by Clauses**
>
> You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Factless Fact Deployment

#### Factless Fact Initial Deployment

When deployed for the first time into an Environment the Factless Fact Node of materialization type table will execute the Create Fact Table stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create Fact Table** | This will execute a CREATE OR REPLACE statement and create a table in the target Environment |

#### Factless Fact Redeployment

After the Factless Fact Node has been deployed for the first time into a target Environment, subsequent deployments may result in either altering the Factless Fact table or recreating the Factless Fact table.

#### Altering the Factless Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Factless Fact Table in the target Environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Factless Fact Tables

If any of the following changes are detected, then the table will be recreated using a CREATE OR REPLACE.

* Join clause
* Adding transformations
* Changes in configuration like adding distinct, group by, or order by

One of the following stages is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |
| **Replace Table** | Replaces an existing table|

### Removing a Factless Fact Node

If a Factless Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the Factless Fact table in the target Environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

---

## View

The Coalesce View UDN lets you develop and deploy a View in Snowflake.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.

### View Node Configuration

The View Node type has two configuration groups:

* [Node Properties](#view-node-properties)
* [Options](#view-options)

![View Node configuration](https://github.com/coalesceio/Coalesce-Base-Node-Types/assets/7216836/6f863c3b-3abd-4318-bd7e-ed50a829f911)

#### View Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the view will be created |
| **Node Type** | Name of template used to create Node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### View Options

| **Options** | **Description** |
|---------|-------------|
| **Override Create SQL** | Toggle: True or False**True**: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible**False**: Create view SQL based on options chosen are framed and executed |
| **Multi Source** | Toggle: True or FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single NodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: single-source Node or multiple sources combined using a join |
| **Distinct** | Toggle: True or False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True or False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Secure** | Toggle: True or False**True**: A secured view is created**False**: A normal view is created |

### View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce App.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### View Deployment

#### View Initial Deployment

When deployed for the first time into an Environment the View Node will execute the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a View in the target Environment |

#### View Redeployment

The subsequent deployment of the View Node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

#### Removing a View Node

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment then the View in the target Environment will be dropped.

This is executed in the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes the view from the Environment |

---

## Code

### Work Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Work-204/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Work-204/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Work-204/run.sql.j2)

### Persistent Stage Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/PersistentStage-173/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/PersistentStage-173/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/PersistentStage-173/run.sql.j2)

### Dimension Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Dimension-194/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Dimension-194/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Dimension-194/run.sql.j2)

### Fact Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Fact-205/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Fact-205/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/Fact-205/run.sql.j2)

### Factless Fact Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/FactlessFact-248/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/FactlessFact-248/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/FactlessFact-248/run.sql.j2)

### View Code

* [Node definition](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/View-188/definition.yml)
* [Create Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/View-188/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/nodeTypes/View-188/run.sql.j2)

### Macros

* [macro-1.yml](https://github.com/coalesceio/Coalesce-Base-Node-Types/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.6.2 | June 02, 2026 | treatNullAsCurrentTimestamp toggle removed in P_stg, Dim, Fact and minor bug fixes(P_stg). |
| 1.6.1 | May 08, 2026 | NM-236 NM-236-Add delete option to Persistent Stage node (Snowflake) and minor bug fixes |
| 1.5.0 | February 27, 2026 | get_clause() macro adapted to support column names with keywords  |
| 1.4.0 | January 14, 2026 | Adding Last Modified configuration |
| 1.3.7 | November 18, 2025 | Added an else block to the RUN template to handle cases where the materializationType is View |
| 1.3.6 | November 04, 2025 | Config UI changes |

---

## Cortex

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/cortex</code> <CopyButton textToCopy='@coalesce/snowflake/cortex' />
Supported Platform: 
Latest Version: 4.1.0 (May 20, 2026)

aianomaly detectionclassificationdocument aiforecastingllmmachine learningmlragsemantic searchsentiment analysissummarizationtranslation

## Overview
Leverage the power of Snowflake Cortex functions from within Coalesce.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/cortex</code> <CopyButton textToCopy='@coalesce/snowflake/cortex' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Cortex Package

* [ML Forecast](#ml-forecast)
* [ML Anomaly Detection](#ml-anomaly-detection)
* [LLM Cortex Functions](#llm-cortex-functions)
* [Top Insights](#top-insights)
* [Classification](#classification)
* [AI Extract](#ai-extract)
* [Cortex Search Service](#cortex-search-service)
* [Code](#code)
---

## ML Forecast

The Coalesce ML Forecast UDN is a versatile node that allows you to create a forecast table and insert forecasts of time series data using the Snowflake built-in class [FORECAST](https://docs.snowflake.com/sql-reference/classes/forecast).

[Snowflake Cortex](https://docs.snowflake.com/en/user-guide/ml-powered-forecasting) is Snowflake's intelligent, fully-managed service that enables organizations to quickly analyze data and build AI applications, all within Snowflake. This service makes Machine Learning (ML) functionality accessible to data engineers to enrich data pipelines while still using SQL. Forecasting employs a machine learning algorithm to predict future data by using historical time series data.

### Node Configuration

The ML Forecast has two configuration groups:

* [Node Properties](#node-properties)
* [Forecast Model Input](#forecast-model-input)

#### Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Forecast table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Forecast Model Input

| **Option** | **Description** |
|------------|----------------|
| **Model Instance Name** | (Required) Name of the model that needs to be created |
| **Create Model** | True/False toggle to determine model creation:- **True**: Forcefully create Forecast model-- **Series Column** (required for multi-series): For multiple time series models, the name of the column defining the multiple time series in input data.- **False**: Refer to existing Forecast model |
| **Multi-Series Forecast** | True/False toggle for forecast type:- **True**: Create multi-series forecast model with series column, timestamp column and target column- **False**: Specify the timestamp column and target column to create single-series forecast model |
| **Series Column** | (Required for multi-series) Column defining multiple time series in input data |
| **Timestamp Column** | (Required) Column containing timestamps in input data |
| **Target Column** | (Required) Column containing target values in input data | 
| **Config object** | OBJECT containing key-value pairs to configure forecast job |
| **Series value** | Required for multi-series forecasts. Single value or VARIANT |
| **Exogenous Variables** | True/False toggle:- **True**: Add future-valued exogenous data using multi-source toggle- **False**: Create forecast model based on days to forecast only |
| **Multi Source** | Toggle to add future-valued Exogenous data |
| **Days to Forecast** | (Required for forecasts without exogenous variables) Number of steps ahead to forecast |

### ML Forecast Preprocessing of Data

When the forecast model returns an error, the error message returned by Snowflake is captured and surfaced directly in the Coalesce application for troubleshooting.

Common scenarios you may encounter:

* NULLS in the source data. The model will cope with some, but not too many NULLS.
* Missing time periods. If the model is unable to determine a consistent frequency in the time series it will cause an error.
* Missing exogenous variables. If the model was trained with exogenous variables.
* Exogenous variables need to be provided into the future to predict future values.

A data preparation step in Coalesce can be used prior to the ML Forecast node to address these issues.

### ML Forecast Deployment

#### ML Forecast Initial Deployment

When deployed for the first time into an environment the ML Forecast node will execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Forecast Table** | This will execute a CREATE OR REPLACE statement and create a Forecast Table in the target environment |

#### ML Forecast Redeployment

After the ML Forecast node has been deployed for the first time into a target environment, subsequent deployments may result in altering the forecast table.

#### ML Forecast Altering the Forecast Tables

The following column or table changes that is made in isolation or all-together will result in an ALTER statement to modify the Forecast table in the target environment:

* Change in table name
* Dropping existing column  
* Alter column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table/Alter Column/Delete Column/Add Column/Edit table description** | Alter table statement is executed to perform the alter operation accordingly |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

### ML Forecast Undeployment

If a ML Forecast table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Forecast Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

---

## ML Anomaly Detection

The Coalesce ML Anomaly Detection UDN is a versatile node that allows you to create an Anomaly table and insert anomalies of time series data using the Snowflake built-in class [ANOMALY DETECTION](https://docs.snowflake.com/en/sql-reference/classes/anomaly_detection).

[Snowflake Cortex](https://docs.snowflake.com/en/user-guide/ml-powered-anomaly-detection) is Snowflake's intelligent, fully-managed service that enables organizations to quickly analyze data and build AI applications, all within Snowflake. This service makes Machine Learning (ML) functionality accessible to data engineers to enrich data pipelines while still using SQL. Anomalies in data are detected by analyzing the dataset using a machine learning algorithm.

### ML Anomaly Detection Node Configuration

The ML Anomaly has two configuration groups:

* [Node Properties](#ml-anomaly-node-properties)
* [Anomaly Model Input](#ml-anomaly-model-input)

#### ML Anomaly Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### ML Anomaly Model Input

| **Option** | **Description** |
|------------|----------------|
| **Model Instance Name** | (Required) Name of the model that needs to be created |
| **Create Model** | True/False toggle to determine if model can be created or refer to an existing model:- **True**: Forcefully create Anomaly model- **False**: Refer to existing Anomaly model |
| **Multi-Series** | True/False toggle for series type:- **True**: Create multi-series model with series column, timestamp column and target column- **False**: Create single-series model |
| **Series Column** | (Required for multi-series) Column defining multiple time series in input data |
| **Timestamp Column** | (Required) Column containing timestamps in input data |
| **Target Column** | (Required) Column containing target(dependent) values in input data |
| **Config object** | Object containing configuration settings for anomaly detection job |
| **Supervised Data** | Toggle to train model using labeled data through multi-source toggle |
| **Labeled Column** |(Availbe with supervised datas) Essential for supervised data, distinguishes between normal and anomalous instances |
| **Unsupervised Data** | Toggle to train model using historical data through multi-source toggle |
| **Multi Source** | Toggle to add data for analysis |

### ML Anomaly Detection Preprocessing of Data

When the Anomaly model returns an error, the error message returned by Snowflake is captured and surfaced directly in Coalesce for troubleshooting.

Common scenarios you may encounter:

* Missing time periods. If the model is unable to determine a consistent frequency in the time series it will cause an error.
* Missing labeled column. If the model was trained with supervised data it's necessary to pass a labeled column.

Often, a data preparation step in Coalesce can be used prior to the ML Anomaly Detection node to address these issues.

### ML Anomaly Detection Deployment

#### ML Anomaly Detection Initial Deployment

When deployed for the first time into an environment the ML Anomaly Detection node will execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Anomaly Detection Table** | This will execute a CREATE OR REPLACE statement and create an Anomaly Table in the target environment |

#### ML Anomaly Detection Redeployment

After the ML Anomaly node has been deployed for the first time into a target environment, subsequent deployments may result in altering the Anomaly table.

#### ML Anomaly Detection Altering the Anomaly Tables

The following column or table changes that is made in isolation or all-together will result in an ALTER statement:

* Change in table name
* Dropping existing column
* Alter column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table/Alter Column/Delete Column/Add Column/Edit table description** | Alter table statement executed to perform operation |
| **Swap cloned Table** | Clone replaces main table after successful updates |
| **Delete Table** | Drops the internal table |

### ML Anomaly Detection Undeployment

If a ML Anomaly table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Anomaly Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

---

## LLM Cortex Functions

The Coalesce Cortex Function UDN provides instant access to industry-leading large language models (LLMs) developed by researchers. Additionally, it offers models that Snowflake has finely tuned for specific use cases.

[Snowflake Cortex - LLM Functions](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions) LLMs are fully hosted and managed by Snowflake, using them requires no setup. Your data stays within Snowflake, giving you the performance, scalability, and governance you expect.

### LLM Cortex Functions Node Configuration

The LLMs Cortex function has three configuration groups:

* [Node Properties](#llm-cortex-functions-node-properties)
* [Options](#llm-cortex-functions-options)
* [Cortex Package](#llm-cortex-functions-cortex-package)

#### LLM Cortex Functions Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### LLM Cortex Functions Options

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Provides option to choose materialization type as `table` |
| **Multi Source** | True/False toggle for SQL UNIONs:- **True**: Multiple sources combined using Multi Source Strategy- **False**: Single source or join |
| **Truncate Before** | True/False toggle:- **True**: INSERT OVERWRITE used- False: INSERT used to append data |
| **Enable tests** | Toggle to enable testing features |
| **Pre-SQL** | SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

#### LLM Cortex Functions Cortex Package

| **Option** | **Description** |
|------------|----------------|
| **SUMMARIZE** | True/False toggle if the data from the column should be returned as a summary:- **True**: System prompts to add a column- **False**: Function remains inactive |
| **SENTIMENT** | True/False toggle to return sentiment score (-1 to 1) for English text, with -1 being the most negative, 0 is neutral, and 1 is positive.:- **True**: System prompts to add a column- **False**: Function remains inactive |
| **TRANSLATE** | True/False toggle to translate text:- True: System prompts to add a column- False: Function remains inactive |
| **EXTRACT ANSWER** | True/False toggle to extract answers:- **True**: System prompts to add a column- **False**: Function remains inactive |

### LLM Cortex Functions Key Points for Consideration

1. Review the [required privileges](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions)
2. Datatype of target column for using function "Extract Answer" should be an Array
3. If source data is required in the target column, duplication of column is necessary before applying a cortex function

### LLM Cortex Functions Deployment

#### LLM Cortex Functions Initial Deployment

When deployed for the first time into an environment the LLM node will execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | This will execute a CREATE OR REPLACE statement and create a Table in the target environment |

#### LLM Cortex Functions Redeployment

After the LLM node has been deployed for the first time into a target environment, subsequent deployments may result in altering the table.

#### Altering the LLM Cortex Functions Tables

The following column or table changes if made in isolation or all-together will result in an ALTER statement:

* Change in table name
* Dropping existing column
* Alter Column data type  
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table/Alter Column/Delete Column/Add Column/Edit table description** | Alter table statement executed to perform operation |
| **Swap cloned Table** | Clone replaces main table after successful updates |
| **Delete Table** | Drops the internal table |

### LLM Cortex Functions Undeployment

If a LLM Node is deleted from a Workspace, and that Workspace is committed to Git, subsequently deployed to a higher-level environment, then the table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

---

## Top Insights

The Coalesce Top Insights UDN is a versatile node that allows you to streamline and improve the process of root cause analysis around changes in observed metrics. Learn more about [Top Insights](https://docs.snowflake.com/en/user-guide/snowflake-cortex/ml-functions/contribution-explorer).

### Top Insights Node Configuration

The Top Insights node has two configuration groups:

* [Node Properties](#ml-contribution-explore-node-properties)
* [Top Insights](#ml-contribution-explorer-config)

#### Top Insights Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Top Insights Configuration

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Provides option to create a 'view' |
| **CATEGORICAL DIMENSIONS** | (Required) Categorical attributes essential for analysis |
| **CONTINUOUS DIMENSIONS** | (Required) Columns representing continuous aspects that vary within a range |
| **Metric** | (Required) Column representing target metric being investigated |
| **Label** | (Required) Column distinguishing between control (FALSE) and test (TRUE) data |
| **Filter Insights** | Textbox to customize filtering of top insights |
| **Order By** | True/False toggle:- **True**: Sort column and sort order visible and required for order by clause- **False**: Sort column and sort order invisible |

### Top Insights Deployment

#### Top Insights Initial Deployment

When deployed for the first time into an environment the View node will execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create or replace View** | This will execute a CREATE OR REPLACE statement and create a View in the target environment |

#### Top Insights Redeployment

The subsequent deployment of a View node with changes in the view definition, altering options, or renaming the view results in the deletion of the existing view and the recreation of the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

### Top Insights Undeployment

If a View Node is removed from a Workspace, and the changes are committed to Git and deployed to a higher-level environment, the corresponding View in the target environment will be dropped.

This is executed in a single stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing View from target environment |

---

## Classification

The Coalesce Classification is a versatile node that allows you to create a classification table and classification model to classify data into different classes using patterns detected in training data using in-built snowflake ML function [CLASSIFICATION](https://docs.snowflake.com/en/sql-reference/classes/classification).

Classification uses machine learning algorithms to sort data into different classes using patterns detected in training data. Binary classification (two classes) and multi-class classification (more than two classes) are supported. Common use cases of classification include customer churn prediction, credit card fraud detection, and spam detection.

### Classification Node Configuration

The Classification node has two configuration groups:

* [Node Properties](#classification-node-properties)
* [Classification Model Input](#classification-model-input)

#### Classification Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Classification Model Input

| **Option** | **Description** |
|------------|----------------|
| **Model Instance Name** | (Required) Name of the model that needs to be created |
| **Create Model** | True/False toggle to determine model creation:- **True**: Forcefully create Classification model- **False**: Refer to existing Classification model |
| **Target Column** | (Required) Column containing target values in input data |
| **Multi Source** | Toggle to add data for analysis |

### Classification Deployment

#### Classification Initial Deployment

When deployed for the first time into an environment the Classification node will execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Classification Table** | This will execute a CREATE OR REPLACE statement and create a Classification Table in the target environment |

#### Classification Redeployment

After the Classification node has been deployed for the first time into a target environment, subsequent deployments may result in altering the Classification table.

#### Altering the Classification Tables

The following column or table changes that is made in isolation or all-together will result in an ALTER statement:

* Change in table name
* Dropping existing column
* Alter column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table/Alter Column/Delete Column/Add Column/Edit table description** | Alter table statement executed to perform operation |
| **Swap cloned Table** | Clone replaces main table after successful updates |
| **Delete Table** | Drops the internal table |

### Classification Undeployment

If a Classification table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Classification Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

## AI Extract

### AI_EXTRACT (Document AI legacy models)

> **DECOMMISSIONED FEATURE**
>
> Document AI and the `<model_build_name>!PREDICT` method are decommissioned. For more information, see [Document AI decommission](https://docs.snowflake.com/en/user-guide/document-ai/decommissioning).

The Coalesce AI Extract UDN is a node that allows you to develop and deploy a document processing pipeline using the **SNOWFLAKE.CORTEX.AI_EXTRACT** function. This pipeline enables the extraction of structured information from documents stored in a Snowflake stage by using natural language questions, eliminating the need for manual model training or complex UI-based setups.

AI_EXTRACT is a modern Cortex AI function that serves as the successor to the legacy AI Extract workflow. It uses a high-performance vision-language model (Arctic-Extract) to identify and retrieve entities, lists, and tables from unstructured files like invoices, receipts, or financial statements. More information about AI_EXTRACT can be found in the official [Snowflake’s Introduction to AI_EXTRACT](https://docs.snowflake.com/en/sql-reference/functions/ai_extract).

### Usage of AI Extract node type
* Set up the required objects and privileges
* Provide the stage path and file details for the documents to be processed within the configuration section of the node.
* The node creates a pipeline to process documents
* The data is available in the target table only after uploading new documents to the internal stage specified in config.
* If the node is created with 'Development mode-ON',no task is created and data can instantly loaded into target from documents using run option once the files are uploaded.
* If the node is created with 'Development mode-OFF',task is created to process the uploaded files 
* Stages created with Client-side encryption are not supported.

### AI Extract Node Configuration

The AI Extract node has the following configuration groups:

* [Node Properties](#stream-and-insert-or-merge-node-properties)
* [General Options](#AI-Extract-general-options)
* [Stream Options](#AI-Extract-stream-options)
* [Source Data](#AI-Extract-Source-Data)
* [Scheduling Options](#AI-Extract-scheduling-options)
* [Advance Scheduling Options](#AI-Extract-advance-scheduling-options)
* [Notification Options](#AI-Extract-notification-options)

#### AI Extract Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the stream,table,task will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### AI Extract General Options

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if SQL executes as DML**True** - Table created and SQL executes as Run action**False** - SQL wrapped in task with specified Scheduling Options. When Run is executed, a message appears prompting the user to **wait** or suggesting a **manual run**. |
| **CREATE AS** | Choose target object type:- Table : Permanent table with data retention and fail-safe- Transient Table : Temporary table without data retention |
| **Truncate Before** | True / False toggle determines whether a table will be overwritten each time a task executes**True** - Uses INSERT OVERWRITE**False** - Uses INSERT to append data |

#### AI Extract Stream Options

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | **Directory Table**:- A directory table is an object that sits on top of a stage, similar to an external table, and stores metadata about the files in the stage. It doesn’t have its own privileges and is used to reference file-level data. Both external (cloud storage) and internal (Snowflake) stages support directory tables. You can add a directory table to a stage when creating it with CREATE STAGE or modify it later using ALTER STAGE. |
| **Redeployment Behavior** | options for Redeployment : - Create or Replace- Create if Not Exists- Create at Existing Stream |

#### AI Extract Source Data

| **Option** | **Description** |
|------------|----------------|
| **Colaesce Storage Location of stage** | The Storage location Name in Coalesce where the stage is located |
| **Stage Name** | The Stage name created in Snowflake |
| **Path or Subfolder** | The path or the subfolder name where the file is present inside a stage. |
| **Response Format Dictionary Toggle** | A toggle to choose the input method. Set to **TRUE** to use a dictionary or **FALSE** to use a structured table grid. |
| **Extraction Dictionary** | (Visible when Toggle is **TRUE**) A textbox where you can directly provide a dictionary containing the field names and their corresponding questions. |
| **AI Extract Extraction Schema (Table)** | (Visible when Toggle is **FALSE**) A list where you define what information the AI should look for. **Field Name**: The name of the column where the answer will be saved. **Question:** The question you want to ask the AI (e.g., "What is the invoice date?"). |

#### AI Extract Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | **Warehouse Task**: User managed warehouse executes tasks |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse on which to run task** | Enter the name of the warehouse you want the task to run on without quotes.|
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax. |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format*|

#### AI EXTRACT Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### AI EXTRACT Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### AI Extract - System Columns

The set of columns which has source data and file metadata information.

| **System Column**        | **Description** |
|--------------------------|------------------|
|**FILENAME**               | Name of the staged documents extracted|
| **FILE_URL**              | The location url of the document|
| **FILE_LAST_MODIFIED**    | The last modified timestamp of the staged documents|
| **SIZE**                  | The size of the document extracted|
| **EXTRACTED_DATA**        | The data extracted from documents|
| **DATA_EXTRACT_TIMESTAMP**| The load timestamp of document extraction|

#### Usage Example

* Extraction Dictionary 

* AI Extract Extraction Schema

### AI Extract Deployment

#### AI Extract Deployment Parameters

The AI Extract node includes an environment parameter that allows you to specify a different warehouse used to run a task in different environments.

The parameter name is `targetTaskWarehouse` with default value `DEV ENVIRONMENT`.

```json
{
    "targetTaskWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than DEV ENVIRONMENT` the node will attempt to create the task using a Snowflake warehouse with the specified value.

For example, with the below setting for the parameter in a QA environment, the task will execute using a warehouse named `SNOWFLAKE_AI_EXTRACT_WH`.

MDXSAFE_PLACEHOLDER_1_ENDPLACEHOLDER

#### AI Extract Initial Deployment

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Creates Stream in target environment |
| **Create Work Table/Transient Table** | Creates table loaded by task |
| **Create Task** | Creates scheduled task |
| **Resume Task** | Enables task execution |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.
The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change

#### AI Extract Redeployment

Stream redeployment behavior:

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| **Create Stream if not exists** | Re-Create Stream at existing offset |
| **Create or Replace** | Create Stream |
| **Create at existing stream** | Re-Create Stream at existing offset |

Table changes execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/Alter Column/Delete Column/Add Column/Edit description** | Alters table as needed |

If the materialization type is changed from one type to another(transient table/table) the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table/Transient Table** | Drop the target table|
| **Create Work/Transient table**| Create the target table|

Task changes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Creates scheduled task |
| **Resume Task**| Resumes the task|

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### AI Extract Undeployment

When node is deleted, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Stream** | Removes the stream |
| **Drop Table** | Drop the table |
| **Drop Current Task** | Drop the task |

## Cortex Search Service

The Cortex Search UDN enables low-latency, high-quality “fuzzy” search over your Snowflake data. Cortex Search powers a broad array of search experiences for Snowflake users including Retrieval Augmented Generation (RAG) applications leveraging Large Language Models (LLMs).

[Cortex Search](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) gets you up and running with a hybrid (vector and keyword) search engine on your text data in minutes, without having to worry about embedding, infrastructure maintenance, search quality parameter tuning, or ongoing index refreshes.

### Usage of Cortex Search Service node type
* Set up the required objects and privileges
* To create Cortex Search Service,keep 'Create Cortex Search Service' toggle ON.Provide schedule and advanced options.Hit create button and CSS is created
* To preview/query the Cortex Search Service,keep 'Preview Cortex Search Service' toggle ON.Provide the necessary configs.Hit run button and a target view with the search results
* You can create a Cortex Search Service once on a source data and use the same to create multiple target views
* 
![cssmultiple](https://github.com/user-attachments/assets/51b99f16-059a-440d-8a58-c6aed82ad2a1)

### Cortex Search Service Node Configuration

The Cortex Search service node has the following configuration groups:

* [Node Properties](#stream-and-insert-or-merge-node-properties)
* [General Options](#cortex-search-service-general-options)
* [Cortex search schedule Options](#cortex-search-schedule-options)
* [Cortex search Advanced Options](#cortex-search-advanced-options)
* [Preview search Options](#preview-search-options)

![cssnodeconfig](https://github.com/user-attachments/assets/123b80e6-4697-4824-b1ec-34d6925f5f33)

#### Cortex Search service Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the stream,table,task will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Cortex Search service General Options

| **Option** | **Description** |
|------------|----------------|
| **Create Cortex Service** | True / False toggle that determines whether to create a cortex search service **True** -  SQL executes as Create action to create cortex search service **False** - No cortex search service created |
| **Preview/Query the Cortex Service** | True / False toggle that determines whether to query the cortex search service created **True** -  SQL executes as run action to create view with the results of querying the cortex search service **False** - No view is created with the query results of cortex search service. When Run is executed, a message appears prompting the user to toggle on Preview/Query. |

![cssgeneral](https://github.com/user-attachments/assets/1c051a9d-4c7f-4ef9-8415-d31ae035c9c8)

#### Cortex search schedule Options

| **Option** | **Description** |
|------------|----------------|
| **Warehouse** | (Required) Name of warehouse used to refresh the Dynamic Table |
| **Lag Specification** |(Required) Specifies the maximum amount of time that the Cortex Search service content should lag behind updates to the base tables specified in the source query.Set refresh schedule with:- **Time Value**: Frequency of the refresh- **Time Period**: Seconds/Minutes/Hours/Days |
| **Initialize** | Initial refresh behavior:- **BLANK('')**: If the blank option is selected, the default behavior will be ON_CREATE.- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |

![cssschedule](https://github.com/user-attachments/assets/ba0e3bc5-633a-4db4-9b08-b3ffc4cc2e4a)

#### Cortex search Advanced Options

| **Option** | **Description** |
|------------|----------------|
| **Embedding model** | Parameter that specifies the embedding model to use in the Cortex Search Service. If the EMBEDDING_MODEL is not specified, the **default** model is used. The default model is snowflake-arctic-embed-m-v1.5. |
| **Search column** |(Required) Specifies the text column in the base table that you wish to search on. This column must be a text value. |
| **Attribute Columns** | (Required)Specifies comma-separated list of columns in the base table that you wish to filter on when issuing queries to the service.|

![cssadvanced](https://github.com/user-attachments/assets/571dce34-2aee-49ac-95b6-382f51acf705)

#### Preview search options

| **Option** | **Description** |
|------------|----------------|
| **Use existing Cortex Search Service** |It allows to use an existing Cortes search service with same source information  |
| **All Columns in the preview** |It allows all columns part of search service to be part of target view|
| **Columns in Target View**|It allows to choose only specific columns to be part of target view.If the above toggle is true,this option is ddisabled|
| **Filter**|Cortex Search supports filtering on the ATTRIBUTES columns specified|
| **Multiple filter conditions**|Allows you to add multiple filter criteria|
| **Apply numeric boost to Search Service**|Numeric boosts are applied as weighted averages to the returned fields, while decays leverage a log-smoothed function to demote less recent values.|
| **Apply time decay to Search Service**|Date or time metadata that boosts more recent results. The influence of recency signals decays over time.|
| **Query**|It searches query against the created cortex seacrh service|

![previewsearch](https://github.com/user-attachments/assets/454bcd06-cffe-4341-9774-93390227b9a3)

#### Cortex Search Service Deployment Parameters

The Cortex Search Service node includes an environment parameter that allows you to specify a different warehouse used to run a task in different environments.

The parameter name is `searchServiceWarehouse` with default value `DEV ENVIRONMENT`.

MDXSAFE_PLACEHOLDER_2_ENDPLACEHOLDER

When set to any value other than DEV ENVIRONMENT` the node will attempt to create the task using a Snowflake warehouse with the specified value.

For example, with the below setting for the parameter in a QA environment, the task will execute using a warehouse named `COMPUTE_WH`.

```json
{
    "searchServiceWarehouse": "COMPUTE_WH"
}
```
#### Cortex Search Service Redeployment

After initial deployment, subsequent deployments may alter or recreate the Cortex Search Service.

#### Altering the Cortex Search Service

The following config changes trigger ALTER statements:

1. Warehouse name
2. Description of Cortex Search Service
3. Lag specification

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Cortex Search Service** | Alters Cortex Search service |
| **Refresh Cortex Search Service** | Refreshes Cortex Search Service |

Other column or table level changes like data type change, column name change, column addition/deletion or config level changes result in a `CREATE` statement.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Cortex Search Service Undeployment

A Cortex Search Service will be dropped if all of these are true:

* The  Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher level environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Create Search Service** | Removes table from target environment |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Any Other | Table(AI EXTRACT) | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Transient Table(AI EXTRACT) | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Cortex Search Service | 1. Warning (if applicable)2. Drop  3. Create |

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added when there's a node type switch |
| 11 | Any Other | AI EXTRACT | Existing table columns are not removed automatically. Such columns need to be manually dropped before redeployment. |
| 12 | Any Other |  AI EXTRACT | When switching to the AI EXTRACT node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |

--------------

## Code

### ML Forecast Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Forecast-262/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Forecast-262/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Forecast-262/run.sql.j2)

### ML Anomaly Detection Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AnomalyDetection-261/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AnomalyDetection-261/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AnomalyDetection-261/run.sql.j2)

### LLM Cortex Functions Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/CortexFunctions-271/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/CortexFunctions-271/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/CortexFunctions-271/run.sql.j2)

### TopInsights Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/TopInsights-279/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/TopInsights-279/create.sql.j2)
  
### Classification Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Classification-337/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Classification-337/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Classification-337/run.sql.j2)

### AI Extract Code

* [Node definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AIExtract-429/definition.yml)
* [Create Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AIExtract-429/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/AIExtract-429/run.sql.j2)

### Cortex Search Service

* [Node_definition](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Cortexsearchservice-499/definition.yml)
* [Create_Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Cortexsearchservice-499/create.sql.j2)
* [Run_Template](https://github.com/coalesceio/Cortex-Node-types/blob/main/nodeTypes/Cortexsearchservice-499/run.sql.j2)

[Macros](https://github.com/coalesceio/Cortex-Node-types/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 4.1.0 | May 20, 2026 | NM-220 - Added Node type switch support and NM-209 - Task Enhancements |
| 4.0.0 | April 08, 2026 | NM-225 Replaced deprecated !PREDICT method with AI_EXTRACT function |
| 3.0.2 | November 14, 2025 | Empty SQL and Stream Name Fix |
| 3.0.1 | November 07, 2025 | NM-173 Added blank option in the Initialize option for Cortex search service node type |
| 3.0.0 | July 23, 2025 | Deploy phase removed and added at appropriate stages |
| 2.1.0 | June 03, 2025 | New node type Cortex Search Service added to the package |
| 2.0.3 | March 17, 2025 | Force Users to Enter Parameters When Required |
| 2.0.0 | February 06, 2025 | New node type Document AI added to Cortex package |
| 1.1.1 | November 14, 2024 | Removed underscore from name of Cortex Functions (NM-80) |
| 1.1.0 | October 07, 2024 | New node type Classification added and Contribution Explorer node type renamed |
| 1.0.1 | May 20, 2024 | Release readiness |

---

## Create or Alter package

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/create-or-alter-package</code> <CopyButton textToCopy='@coalesce/snowflake/create-or-alter-package' />
Supported Platform: 
Latest Version: 1.6.1 (April 08, 2026)

alter tableddl automationnon-destructive deploymentproduction safeschema evolution

## Overview
The package includes Create or Alter node types

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/create-or-alter-package</code> <CopyButton textToCopy='@coalesce/snowflake/create-or-alter-package' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Create or Alter package

Create or Alter package includes:
* [Create or Alter table](#create-or-alter-table)
* [Create or Alter view](#create-or-alter-view)
* [Create or Alter Dimension](#create-or-alter-dimension)
* [Create or Alter Fact](#create-or-alter-fact)
* [Create or Alter Persistent Stage](#create-or-alter-persistent-stage)
* [Create or Alter Task](#create-or-alter-task)
* [Create or Alter DAG Root Task](#create-or-alter-dag-root-task)
* [Create DAG Root Resume Task](#Create-DAG-Root-Resume-Task)
* [Code](#code)

## Brief summary

The Create or Alter package is a suite of Coalesce node types designed to streamline the management of Snowflake objects by leveraging Snowflake’s CREATE OR ALTER syntax. This approach simplifies the deployment lifecycle by automatically determining whether to create a new object or modify an existing one based on the defined configuration.

### Key Components
The package supports several essential data modeling and orchestration objects:
* Data Structures: Tables, Views, Dimensions, Facts, and Persistent Stages.
* Orchestration: Tasks, DAG Root Tasks, and Resume Task nodes for managing complex Directed Acyclic Graphs (DAGs) within Snowflake.
Core Functionality
* Automated Deployment: It handles initial creation, redeployment (applying changes like adding columns or updating metadata), and undeployment (dropping objects) across different environments.
* Configuration Management: Users can toggle advanced Snowflake features through a UI, such as clustering keys, change tracking, schema evolution, data retention time, and inline/out-of-line constraints.
* Task Orchestration: It includes specific logic to manage Snowflake Tasks, allowing users to define schedules (Cron or interval), warehouse sizes, and predecessor dependencies. It specifically handles the suspension and resumption of root tasks during deployments to ensure DAG integrity.
* Change Handling: The package is designed to detect changes in metadata or configuration and execute the appropriate DDL (e.g., ALTER TABLE for column additions or DROP/CREATE for materialization type changes).

### Primary Benefit

The main advantage of this package is declarative state management. Instead of writing manual migration scripts, users define the desired state of their Snowflake environment, and the package automates the SQL required to reach that state.

# Create or Alter table

The [Create or alter table](https://docs.snowflake.com/en/sql-reference/sql/create-table#create-or-alter-table) creates table if it doesn’t exist, or alters it according to the table definition. 

### Create or Alter Node properties

Create Or Alter has two configuration groups: 

* [Node Properties](#create-or-alter-node-properties)
* [Create table Options](#create-table-options)
* [Insert data Options](#insert-data-options)

#### Create or Alter Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes 
are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table' or 'transient table' |
| **Cluster key** | True/False toggle for clustering: **True**: Specify clustering 
column and optional expressions **False**: No clustering |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Enable Schema Evolution** | Toggle to enable or disable schema evolution. |
| **Inline Constraints** | Toggle to enable inline constraints:**True**: Specify column name and constraint specification **False**: No constraints |
| **Out-Of-Line Constraints** | Toggle to enable or disable out-of-line constraints. 
When enabled, you can define both primary and foreign keys. |
| **Data Retention Time** | Set the number of days for data retention for Time Travel actions. |
| **Default DDL Collation** | Set the default collation specification for the DDL operations. |

#### Insert data Options

| **Options** | **Description** |
|-------------|-----------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |

### Limitations of Create or Alter Table

* Collation specifications cannot be altered.
* Setting or unsetting an inline primary key changes the nullability of the column accordingly
* New columns can only be added to the end of the column list
* Columns cannot be renamed. If you attempt to rename a column, the column is dropped and a new column is added.WHen you rename a column in Coalesce mapping grid.Ensure to move the same to the end of column list in mapping grid.

### Create Or Alter Table Deployment

### Create Or Alter Table Initial Deployment
When deployed for the first time into an environment the Create or Alter table node of materialization type table or transient table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table in the target environment |

#### Create Or Alter Table Redeployment

#### Altering the Tables and Transient Tables

When redeployed with change in target location or change in node name,results in alter of the table

| **Stage** | **Description** |
|-----------|----------------|
| **Rename/Move Table** | Alter table statement is executed to perform the alter operation |

### Changing Materialization Type 

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Table to transient table or vice versa** |  Drop table/transient table Create or Alter table/transient table |

### Recreating the Create Or Alter Table

When the Create or Alter table node is redeployed with any changes in table or config changes result in re-creating table

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table in the target environment |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

### Create Or Alter Tables Undeployment

If a Create or Alter table node of materialization type table/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes the table from the environment |

# Create or alter view

A [create or alter view](https://docs.snowflake.com/en/sql-reference/sql/create-view#label-create-or-alter-view-syntax) node creates a new view if it doesn’t already exist, or updates the properties of an existing view to match those defined in the statement.

### Create or Alter View Node properties

Create Or Alter has two configuration groups: 

* [Node Properties](#create-or-alter-view-node-properties)
* [Create view Options](#create-view-options)

#### Create or Alter View Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes 
are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create view Options

| **Options** | **Description** |
|-------------|-----------------|
| **Secure** | True/False toggle for secure view: **True**:creates a secure view **False**: Normal view |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination- **INSERT**: Individual insert for each source**False**: Single source node or multiple sources combined using a join. |

### Limitations of Create or Alter View

* The CREATE OR ALTER VIEW command doesn’t support changing a view definition once a view is created. In Coalesce,it is supported by dropping and recreating the view during redeployment.

### Create Or Alter View Deployment

### Create Or Alter View Initial Deployment
When deployed for the first time into an environment the Create or Alter view node will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter view** | This will execute a CREATE OR ALTER statement and create a view in the target environment |

#### Create Or Alter View Redeployment

#### Altering the View

When redeployed with change in target location or change in node name,results in alter of the view

| **Stage** | **Description** |
|-----------|----------------|
| **Rename/Move View** | Alter view statement is executed to perform the alter operation |

### Recreating the Create Or Alter View

When the Create or Alter view node is redeployed with any changes in secure,change_tracking or multi-source config options,create or alter view is executed again.

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter View** | This will execute a CREATE OR ALTER statement and create a view in the target environment |

### Change in view definition

Change in view definition like change in columns,add or drop columns,change in data type ,adding distinct or group by all results in below stages

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter View** | This will execute a CREATE OR ALTER statement and create a view in the target environment |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Create Or Alter View Undeployment

If a Create or Alter view node are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop View** | Removes the table or view from the environment |

----

## Create or Alter Dimension

The [Create or alter](https://docs.snowflake.com/en/sql-reference/sql/create-table#create-or-alter-table) creates dimension table if it doesn’t exist, or alters it according to the table definition. 

### Create or Alter Dimension Node

Create Or Alter has two configuration groups: 

* [Node Properties](#Create-or-Alter-Dimension-Node-Properties)
* [Create Table Options](#create-table-options-1)
* [Insert Data Options](#insert-data-options-1)

#### Create or Alter Dimension Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter Dimension Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create Table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table' or 'transient table' |
| **Cluster key** | True/False toggle for clustering**True**: Specify clustering column and optional expressions**False**: No clustering |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Enable Schema Evolution** | Toggle to enable or disable schema evolution. |
| **Inline Constraints** | Toggle to enable inline constraints**True**: Specify column name and constraint specification**False**: No constraints |
| **Out-Of-Line Constraints** | Toggle to enable or disable out-of-line constraintsWhen enabled, you can define both primary and foreign keys. |
| **Data Retention Time** | Set the number of days for data retention for Time Travel actions.Available only for materialization type - table |
| **Maximum Data Extension time** | Set the number of days for max data extension for Historical data access.Available only for materialization type - table |
| **Default DDL Collation** | Set the default collation specification for the DDL operations. |

#### Insert data Options

| **Options** | **Description** |
|-------------|-----------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Business key** | Required column for both Type 1 and Type 2 DimensionsNote: Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)** | Records with NULL timestamp are updated in target|
| **Type 2 Dimension(Enabled for Last Modified Comparison)** |CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2. If not chosen, treated as type 1|
| **Distinct** | Toggle: True/False**True**: Group by All is invisible.DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisibleData grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause.**False**: Sort options invisible |
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimension**True**: Zero Key Record Options enabled.**False**: Zero Key Record not added |
| **Zero Key Record Options** | Add custom zero key record values for :-Default Surrogate Key -Default String Value-Default Date Value (Date Format DD-MM-YYYY)-Default Timestamp Value (Timestamp Format YYYY-MM-DD HH24:MI:SS.FF)-Default Boolean Value |
| **Advanced Zero Key Record Options** | Toggle: True/False**True**: Select Columns and the default value of the column for zero key record**False**: Advanced Zero Key Record Options not enabled |

### Limitations of Create or Alter Dimension Table

* Collation specifications cannot be altered.
* Setting or unsetting an inline primary key changes the nullability of the column accordingly
* New columns can only be added to the end of the column list
* Columns cannot be renamed. If you attempt to rename a column, the column is dropped and a new column is added. When you rename a column in Coalesce mapping grid, ensure to move the same to the end of column list in mapping grid.
* A column's default value can only be set during its initial creation using a `CREATE OR ALTER` statement; re-running the statement with a different default value will result in an error and will not update/alter/drop the existing default or add a new default.

### Create Or Alter Dimension Table Deployment

### Create Or Alter Dimension Table Initial Deployment
When deployed for the first time into an environment the Create or Alter table node of materialization type table or transient table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table/transient table in the target environment |

#### Create Or Alter Dimension Table Redeployment

#### Altering the Tables and Transient Tables

When redeployed with change in target location or change in node name, results in alter of the table

| **Stage** | **Description** |
|-----------|----------------|
| **Rename/Move Table** | Alter table statement is executed to perform the alter operation |

### Changing Materialization Type 

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Table to transient table or vice versa** |  Drop table/transient table Create or Alter table/transient table |

### Recreating the Create Or Alter Table

When the Create or Alter table node is redeployed with any changes in table or config changes result in re-creating table

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table in the target environment |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment, then no stages are executed

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

### Create Or Alter Tables Undeployment

If a Create or Alter table node of materialization type table/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes the table from the environment |

----

## Create or Alter Fact

The [Create or alter](https://docs.snowflake.com/en/sql-reference/sql/create-table#create-or-alter-table) creates fact table if it doesn’t exist, or alters it according to the table definition. 

### Create or Alter Fact Node

Create Or Alter has two configuration groups: 

* [Node Properties](#Create-or-Alter-Fact-Node-Properties)
* [Create Table Options](#create-table-options-2)
* [Insert Data Options](#insert-data-options-2)

#### Create or Alter Fact Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter fact Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create Table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table' or 'transient table' |
| **Cluster key** | True/False toggle for clustering**True**: Specify clustering column and optional expressions**False**: No clustering |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Enable Schema Evolution** | Toggle to enable or disable schema evolution. |
| **Inline Constraints** | Toggle to enable inline constraints**True**: Specify column name and constraint specification**False**: No constraints |
| **Out-Of-Line Constraints** | Toggle to enable or disable out-of-line constraintsWhen enabled, you can define both primary and foreign keys. |
| **Data Retention Time** | Set the number of days for data retention for Time Travel actions.Available only for materialization type - table |
| **Maximum Data Extension time** | Set the number of days for max data extension for Historical data access.Available only for materialization type - table |
| **Default DDL Collation** | Set the default collation specification for the DDL operations. |

#### Insert Data Options

| **Options** | **Description** |
|-------------|-----------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Business key** | Required column for Fact table creation.Note: Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)**| Records with NULL timestamp are updated in target|
| **Distinct** | Toggle: True/False**True**: Group by All is invisible.DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisibleData grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause.**False**: Sort options invisible |

### Limitations of Create or Alter Fact Table

* Collation specifications cannot be altered.
* Setting or unsetting an inline primary key changes the nullability of the column accordingly
* New columns can only be added to the end of the column list
* Columns cannot be renamed. If you attempt to rename a column, the column is dropped and a new column is added. When you rename a column in Coalesce mapping grid, ensure to move the same to the end of column list in mapping grid.
* A column's default value can only be set during its initial creation using a `CREATE OR ALTER` statement; re-running the statement with a different default value will result in an error and will not update/alter/drop the existing default or add a new default.

### Create Or Alter Fact Table Deployment

### Create Or Alter Fact Table Initial Deployment
When deployed for the first time into an environment the Create or Alter table node of materialization type table or transient table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table/transient table in the target environment |

#### Create Or Alter Fact Table Redeployment

#### Altering the Tables and Transient Tables

When redeployed with change in target location or change in node name, results in alter of the table

| **Stage** | **Description** |
|-----------|----------------|
| **Rename/Move Table** | Alter table statement is executed to perform the alter operation |

### Changing Materialization Type 

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Table to transient table or vice versa** |  Drop table/transient table Create or Alter table/transient table |

### Recreating the Create Or Alter Table

When the Create or Alter table node is redeployed with any changes in table or config changes result in re-creating table

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table in the target environment |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment, then no stages are executed

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

### Create Or Alter Tables Undeployment

If a Create or Alter table node of materialization type table/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes the table from the environment |

----

## Create or Alter Persistent Stage

The [Create or alter](https://docs.snowflake.com/en/sql-reference/sql/create-table#create-or-alter-table) creates Persistent Stage table if it doesn’t exist, or alters it according to the table definition. 

### Create or Alter Persistent Stage Node

Create Or Alter has two configuration groups: 

* [Node Properties](#Create-or-Alter-Persistent-Stage-Node-Properties)
* [Create Table Options](#create-table-options-3)
* [Insert Data Options](#insert-data-options-3)

#### Create or Alter Persistent Stage Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter Persistent Stage Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create Table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table' or 'transient table' |
| **Cluster key** | True/False toggle for clustering**True**: Specify clustering column and optional expressions**False**: No clustering |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Enable Schema Evolution** | Toggle to enable or disable schema evolution. |
| **Inline Constraints** | Toggle to enable inline constraints**True**: Specify column name and constraint specification**False**: No constraints |
| **Out-Of-Line Constraints** | Toggle to enable or disable out-of-line constraintsWhen enabled, you can define both primary and foreign keys. |
| **Data Retention Time** | Set the number of days for data retention for Time Travel actions.Available only for materialization type - table |
| **Maximum Data Extension time** | Set the number of days for max data extension for Historical data access.Available only for materialization type - table |
| **Default DDL Collation** | Set the default collation specification for the DDL operations. |

#### Insert Data Options

| **Options** | **Description** |
|-------------|-----------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Business key** | Required column for both Type 1 and Type 2.Note: Geometry and Geography data type columns are not supported as business key columns. |
| **Last Modified Comparison** | **True**:When enabled we can do timestamp based CDC**False**:Regular CDC based on Change tracking columns is done |
| **Last Modified Column(Enabled for Last Modified Comparison)** | Timestamp/Incremental ID column can be chosen.Based on which CDC is done |
| **Treat Null as Current timestamp(Enabled for Last Modified Comparison)**| Records with NULL timestamp are updated in target|
| **Type 2 Dimension(Enabled for Last Modified Comparison)**|CDC is based on timestamp/ID column chosen above.Change tracking columns are not enabled for this scenario|
| **Change tracking** | Required column for Type 2. If not chosen, treated as type 1|
| **Distinct** | Toggle: True/False**True**: Group by All is invisible.DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisibleData grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause.**False**: Sort options invisible |

### Limitations of Create or Alter Persistent Stage Table

* Collation specifications cannot be altered.
* Setting or unsetting an inline primary key changes the nullability of the column accordingly
* New columns can only be added to the end of the column list
* Columns cannot be renamed. If you attempt to rename a column, the column is dropped and a new column is added. When you rename a column in Coalesce mapping grid, ensure to move the same to the end of column list in mapping grid.
* A column's default value can only be set during its initial creation using a `CREATE OR ALTER` statement; re-running the statement with a different default value will result in an error and will not update/alter/drop the existing default or add a new default.

### Create Or Alter Persistent Stage Table Deployment

### Create Or Alter Persistent Stage Table Initial Deployment
When deployed for the first time into an environment the Create or Alter table node of materialization type table or transient table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table/transient table in the target environment |

#### Create Or Alter Persistent Stage Table Redeployment

#### Altering the Tables and Transient Tables

When redeployed with change in target location or change in node name, results in alter of the table

| **Stage** | **Description** |
|-----------|----------------|
| **Rename/Move Table** | Alter table statement is executed to perform the alter operation |

### Changing Materialization Type 

| **Change** | **Stages Executed** |
|------------|-------------------|
| **Table to transient table or vice versa** |  Drop table/transient table Create or Alter table/transient table |

### Recreating the Create Or Alter Table

When the Create or Alter table node is redeployed with any changes in table or config changes result in re-creating table

| **Stage** | **Description** |
|-----------|----------------|
| **Create/Alter table/transient table** | This will execute a CREATE OR ALTER statement and create a table in the target environment |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment, then no stages are executed

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

### Create Or Alter Tables Undeployment

If a Create or Alter table node of materialization type table/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the table in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Removes the table from the environment |

----

## Create Or Alter Task

The [Create or Alter Task](https://docs.snowflake.com/en/sql-reference/sql/create-task#create-or-alter-task) creates task if it doesn’t exist, or alters it according to the task definition. 

### Create or Alter Task Node

Create Or Alter Task node has two or three configuration groups depending on config options selected: 

* [Node Properties](#Create-or-Alter-Task-Node-Properties)
* [Create Table Options](#create-table-options-4)
* [Insert Data Options](#insert-data-options-4)
* [Scheduling Options](#Task-Scheduling-Options)

#### Create or Alter Task Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Create Or Alter Task will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create Table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Cluster key** | True/False toggle for clustering**True**: Specify clustering column and optional expressions**False**: No clustering |
| **Change Tracking** | Toggle to enable or disable change tracking on the table. |
| **Enable Schema Evolution** | Toggle to enable or disable schema evolution. |
| **Inline Constraints** | Toggle to enable inline constraints**True**: Specify column name and constraint specification**False**: No constraints |
| **Out-Of-Line Constraints** | Toggle to enable or disable out-of-line constraintsWhen enabled, you can define both primary and foreign keys. |
| **Data Retention Time** | Set the number of days for data retention for Time Travel actions.Available only for materialization type - table |
| **Maximum Data Extension time** | Set the number of days for max data extension for Historical data access.Available only for materialization type - table |
| **Default DDL Collation** | Set the default collation specification for the DDL operations. |

#### Insert Data Options

| **Options** | **Description** |
|-------------|-----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action.**True** - A table will be created and SQL will execute as a Run action**False** - After testing the SQL as a Run action, setting to false will wrap SQL in a task with specified Scheduling Options |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join. |
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes.**True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible.DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisibleData grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause.**False**: Sort options invisible |

#### Task Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced|

#### Create or Alter Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Create or Alter Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Create or Alter Task Deployment

#### Create or Alter Task Deployment Parameters

The Create or Alter Task includes an environment parameter that allows you to specify a different warehouse used to run a task in different environments.

The parameter name is `targetTaskWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT` the value entered in the Scheduling Options config Select Warehouse on which to run the task will be used when creating the task.

```json
{
    "targetTaskWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the task using a Snowflake warehouse with the specified value.

For example, with the below setting for the parameter in a QA environment, the task will execute using a warehouse named `compute_wh`.

```json
{
    "targetTaskWarehouse": "compute_wh"
}
```

#### Create or Alter Task Initial Deployment

When deployed for the first time into an environment the Create or Alter Task node will execute the following stages:

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Create Task** | Creates task that will load the target table on schedule |
| **Resume Task** | Resumes the task so it runs on schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called "Task DAG Resume Root." This node will resume the root node once all the dependent tasks have been created as part of a deployment.

#### Create or Alter Task Redeployment

After initial deployment, changes in task schedule, warehouse, or scheduling options will result in a SUSPEND, CREATE or ALTER TASK, RESUME

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Task** | Suspend task |
| **Create Task** | Create Or Alter task with new schedule |
| **Resume Task** | Resumes task with new schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Create Or Alter task with new schedule |

#### Create or Alter Task Altering Tables

Subsequent deployments with changes in table like add or drop column and change in data type will result in an ALTER table statement followed by CREATE OR ALTER TASK AND RESUME TASK statements being issued.

| **Stage** | **Description** |
|-----------|----------------|
| **Change Column Attributes/Delete Column/Add Column/Change table description** | Create or Alter table statement is executed to perform the alter operation. |
| **Suspend Task** | Suspend task |
| **Create Task** | Create Or Alter task with new schedule |
| **Resume Task** | Resumes task with new schedule |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates when the **DEV_MODE** is **ON**. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

### Create or Alter Task Undeployment

If a Create or Alter Task node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then all objects created by the node in the target environment will be dropped.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** |  Drop the table originally created to be loaded by the task. |
| **Drop Current Task** | Drop the task |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table |
| **Suspend Root Task** | Root task needs to be put into a suspended state. |
| **Drop Task** | Drops the task |

If a task is part of a DAG of tasks the DAG needs to include a node type called **Task Dag Resume Root**. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

### Limitations of Create or Alter Task

* Collation specifications cannot be altered.
* Setting or unsetting an inline primary key changes the nullability of the column accordingly
* New columns can only be added to the end of the column list
* Columns cannot be renamed. If you attempt to rename a column, the column is dropped and a new column is added. When you rename a column in Coalesce mapping grid, ensure to move the same to the end of column list in mapping grid.
* A column's default value can only be set during its initial creation using a `CREATE OR ALTER` statement; re-running the statement with a different default value will result in an error and will not update/alter/drop the existing default or add a new default.
* All Task nodes, predecessor and root should be present in same schema.
* Renaming a task isn’t supported. Instead it drops old task and recreates the task with new name and schedule specified.

-----

## Create Or Alter DAG Root Task

The Coalesce Create Or Alter DAG Root Task UDN is a node that helps to create/alter a standalone root task.

The root task should have a defined schedule that initiates a run of the DAG. Each of the other tasks has at least one defined predecessor to link the tasks in the DAG. A child task runs only after all of its predecessor tasks run successfully to completion.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or perform other periodic work.

More information about Tasks can be found in [Snowflake's Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Create Or Alter DAG Root Task Node Configuration

The Create Or Alter DAG Root Task node has two configuration groups:

* [Node Properties](#Create-Or-Alter-DAG-Root-Task-Node-Properties)
* [Scheduling Options](#Create-Or-Alter-DAG-Root-Task-Scheduling-Options)

#### Create Or Alter DAG Root Task Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create Or Alter DAG Root Task Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task** - User managed warehouse executes tasks- **Serverless Task** - Uses serverless compute |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task.  Name of warehouse to run task on without quotes |
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task  Initial compute size for serverless tasks. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax. |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 |
| **Enter root task SQL** | The SQL statement to be run when a standalone root task executes |
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced|

#### Create or Alter Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Create or Alter Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Create Or Alter DAG Root Task Deployment

#### Create Or Alter DAG Root Task Deployment Parameters

The parameter name is `targetTaskWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT` the value entered in the Scheduling Options config Select Warehouse on which to run task will be used when creating the task.

```json
{
    "targetTaskWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the task using a Snowflake warehouse with the specified value.

For example, with the below setting for the parameter in a QA environment, the task will execute using a warehouse named `compute_wh`.

```json
{
    "targetTaskWarehouse": "compute_wh"
}
```

#### Create Or Alter DAG Root Task Initial Deployment

When deployed for the first time into an environment, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Root Task** | Creates task that will execute Root SQL on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Create DAG Root Resume Task`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

#### Create Or Alter DAG Root Task Redeployment

After the Task has deployed for the first time into a target environment, subsequent deployments will execute two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task |
| **Create Root Task** | Recreates root task |

### Create Or Alter DAG Root Task Undeployment

When a Create Or Alter DAG Root Task node is deleted, two stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root task** | Suspends root task |
| **Drop current task** | Removes the task |

----

## Create DAG Root Resume Task

The Coalesce Create DAG Root Resume Task UDN is a node type that helps to resume the root task and its dependents or child tasks. Recursively resumes all dependent tasks tied to a specified root task in a DAG using the root task name specified.

The root task should have a defined schedule that initiates a run of the DAG. Each of the other tasks has at least one defined predecessor to link the tasks in the DAG. A child task runs only after all of its predecessor tasks run successfully to completion.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or perform other periodic work.

More information about Tasks can be found in [Snowflake's Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Create DAG Root Resume Task Node Configuration

The Task DAG Resume Root node has two configuration groups:

* [Node Properties](#Create-DAG-Root-Resume-Task-Node-Properties)
* [Scheduling Options](#Create-DAG-Root-Resume-Task-scheduling-options)

#### Create DAG Root Resume Task Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Create DAG Root Resume Task Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Enter root task name** | Name of the root task to be resumed - recursively resumes all dependent tasks tied to this specified root task |

### Create DAG Root Resume Task Deployment

#### Create DAG Root Resume Task Initial Deployment

When deployed for the first time into an environment the following stage executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Try Enable Root Task** | Resumes root task and all its dependents |

#### Create DAG Root Resume Task Undeployment

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Updates** | Metadata Updates |

------

## Code

### Create or Alter table Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterTable-345/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterTable-345/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterTable-345/run.sql.j2)

### Create or Alter view Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterView-419/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterView-419/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateorAlterView-419/run.sql.j2)

### Create or Alter Dimension Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterDimension-535/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterDimension-535/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterDimension-535/run.sql.j2)

### Create or Alter Fact Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterFact-534/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterFact-534/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterFact-534/run.sql.j2)

### Create or Alter Persistent Stage Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterPersistentStage-533/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterPersistentStage-533/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterPersistentStage-533/run.sql.j2)

### Create or Alter Task

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterTask-548/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterTask-548/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterTask-548/run.sql.j2)

### Create Or Alter DAG Root Task

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterDAGRootTask-549/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateOrAlterDAGRootTask-549/create.sql.j2) |

### Create DAG Root Resume Task

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateDAGRootResumeTask-550/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/create-alter-node-types/blob/main/nodeTypes/CreateDAGRootResumeTask-550/create.sql.j2) |

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.6.1 | April 08, 2026 | NM-209 - Task Node Enhancements - Notification Integrations, Execute As User, and Advanced Task Para |
| 1.6.0 | March 06, 2026 | Node switching support,get_clause() macro change |
| 1.5.0 | February 19, 2026 | Last Modified Comparison toggle added to Create/Alter package |
| 1.4.0 | November 03, 2025 | Metadata Refresh during deployment - Added |
| 1.2.0 | August 11, 2025 | Added 3 new node types - Create or Alter Task, DAG Root and Resume Task |
| 1.1.0 | July 30, 2025 | 3 New Node types Added |
| 1.0.1 | March 04, 2025 | Redeployment fixes for Create package |
| 1.0.0 | January 28, 2025 | Create or Alter table, Create or Alter view node types added |

---

## Data quality

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/data-quality</code> <CopyButton textToCopy='@coalesce/snowflake/data-quality' />
Supported Platform: 
Latest Version: 2.2.0 (February 20, 2026)

data governancedata metric functionsdata profilingdata qualityduplicate detectionfreshnessmonitoringnull countquality rulesrow countvalidation

## Overview
The data quality package is a set of node types which provide the flexibility to test the data quality of source.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/data-quality</code> <CopyButton textToCopy='@coalesce/snowflake/data-quality' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Data Quality Nodetypes

The Coalesce Data Quality Nodetypes Package includes:

* [DMF](#DMF)
* [Data Profiling](#data-profiling)
* [Code](#code)

---

### DMF

The Coalesce DMF node allows you to to monitor the state and integrity of your data. You can use DMFs to measure key metrics, such as, but not limited to, freshness and counts that measure duplicates, NULLs, rows, and unique values.This is based on Data Metric functions concept in Snowflake.For more information on roles and privileges refer [docs](https://docs.snowflake.com/en/user-guide/data-quality-intro#about-data-quality-and-dmfs).

You can set a DMF on the following kinds of table objects:
* Dynamic table
* Event table
* External table
* Apache Iceberg™ table
* Materialized view
* Table (Including temporary and transient tables)
* View

### Limitations:
* You cannot set a DMF on a hybrid table or a stream object.
* You can only have 10,000 total associations of DMFs on objects per account. Each instance of setting a DMF on a table or view counts as one association.
* You cannot grant privileges on a DMF to share or set a DMF on a shared table or view.
* Setting a DMF on an object tag is not supported.
* You cannot set a DMF on objects in a reader account.
* Trial accounts do not support this feature.
* In coalesce,this node cannot be used as part of a pipeline.This node can be added at the end of the pipeline to infer Data Metrics
* Package DMF settings are static once defined and deployed. Redeployment behavior is driven only by universal enable or disable toggles, not by changes to package configuration values.
  
### DMF Node Configuration

The Work node type has seven configuration groups:

* [Node Properties](#node-properties)
* [Scheduling Options](#scheduling-options)
* [Universal DMFs](#universal-dmfs)
* [Object Level DMFs](#object-level-dmfs)
* [Column Level DMFs](#column-level-dmfs)
* [Custom DMFs](#custom-DMFs)
* [Alerting Options](#alerting-options)

#### Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Scheduling Options

You can create DMF run schedule to automate the data quality measurements on a table, as:

| **Option** | **Description** |
|------------|----------------|
| **Schedule Options** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between DMF runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the DMF. Supports a subset of standard cron utility syntax.- **TRIGGER_ON_CHANGES** - Set the data metric function to run when a general DML operation, such as inserting a new row, modifies the table |

#### Universal DMFs

Package Config-driven defaults that apply common DMF patterns across all DMFS nodes, with per-node toggle controls covering all Snowflake system DMFs suitable for universal application

When the universal toggle and the object-level universal toggle are enabled, package DMF settings are applied at the **table level.**

When the universal toggle and the column-level universal toggle are enabled, package DMF settings are applied to the corresponding **columns**.

**Package Config Settings:**

```sql
{% set packageConfiguration = namespace
    (
        universalObjectDmfs=['ROW_COUNT', 'FRESHNESS'], 
        nullCountUniversalColumns=[], 
        nullPercentUniversalColumns=[],
        blankCountUniversalColumns=[],
        blankPercentUniversalColumns=[],
        duplicateCountUniversalColumns=[],
        uniqueCountUniversalColumns=[],
        freshnessUniversalColumns=[]
    )
%}
```

| **Property** | **Description** |
|----------|-------------|
| **Enable Universal DMFs** | Master toggle for Universal DMFs section. `Default: true` |
| **Apply Universal Object Level DMFs** | Enable object-level DMFs. `Default: true` |
| **Object Level DMFs** | TextBox for DMFs to apply at table level. Defaults from Package Config universalObjectDmfs. Visible when toggle is true |
| **Apply Null Count DMF** | Enable NULL_COUNT on specified columns. `Default: true` |
| **Null Count Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Null Percent DMF** | Enable NULL_PERCENT on specified columns. `Default: false` |
| **Null Percent Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Blank Count DMF** | Enable BLANK_COUNT on specified columns. `Default: false` |
| **Blank Count Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Blank Percent DMF** | Enable BLANK_PERCENT on specified columns. `Default: false` |
| **Blank Percent Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Duplicate Count DMF** | Enable DUPLICATE_COUNT on specified columns. `Default: true` |
| **Duplicate Count Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Unique Count DMF** | Enable UNIQUE_COUNT on specified columns. `Default: false` |
| **Unique Count Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |
| **Apply Freshness DMF** | Enable FRESHNESS on specified columns. `Default: true` |
| **Freshness Columns** | Comma-separated column names. Defaults from Package Config. Visible when toggle is true |

#### Object Level DMFs

| **Property** | **Description** |
|----------|-------------|
| **Add Object Level DMF** | Enable this to add object level DMF |
| **DMF Function** | Select the DMF function to use for object |

#### Column Level DMFs

| **Property** | **Description** |
|----------|-------------|
| **Add Column Level DMF** | Enable this to add object level DMF |
| **DMF Function** | Select the Column name and DMF function to use |

#### Custom DMFs

Custom DMFs in Snowflake allow you to define your own data quality checks using SQL logic and apply them to tables or columns based on your requirements. For more info on **Custom DMFs** [please refer this documentation](https://docs.snowflake.com/en/user-guide/data-quality-custom-dmfs)

**Edit the placeholders shown here to reflect your own table and column references.**

**Example input for only base table columns**

**Example input for reference table columns**

#### Alerting Options

Optional Snowflake ALERT creation that sends notifications when data quality issues are detected. More about Alerting [please refer this documentation.](https://docs.snowflake.com/en/guides-overview-alerts)

**Prerequisites**

1. Notification Integration (Required for Alerting):

    Users must create a notification integration before enabling alerts:

    ```sql
    CREATE NOTIFICATION INTEGRATION my_email_integration
    TYPE = EMAIL
    ENABLED = TRUE
    ALLOWED_RECIPIENTS = ('user@company.com');
    ```
2. Required Privileges for Alerting:

    CREATE ALERT on the schema
    EXECUTE ALERT privilege
    USAGE on the notification integration
    USAGE on the alert warehouse

3. Compute Cost Consideration:

    Alerts require a warehouse to evaluate conditions. Frequent schedules across many DMFS nodes will incur compute costs. Default schedule is set to 60 minutes to balance responsiveness with cost.

| **Property** | **Description** |
|----------|-------------|
| **Enable Email Alerts** | Master toggle for alerting. `Default: false` |
| **Override Global Parameters** | If enabled, use node config values instead of deployment parameters. `Default: false`. Visible when Enable Email Alerts is true |
| **Notification Integration Name** | Snowflake notification integration name. Defaults to parameter DMF_ALERT_INTEGRATION. Visible when Enable Email Alerts is true |
| **Alert Email Address** | Email recipient for alerts. Defaults to parameter DMF_ALERT_EMAIL. Visible when Enable Email Alerts is true |
| **Warehouse for Alert** | Warehouse to execute alert checks. Defaults to parameter DMF_ALERT_WAREHOUSE. Visible when Enable Email Alerts is true |
| **Alert Check Frequency (Minutes)** | How often to check for issues. `Default: 60`. Defaults to parameter DMF_ALERT_SCHEDULE. Visible when Enable Email Alerts is true |
| **Trigger on Null Count** | Alert when NULL_COUNT > 0. `Default: true`. Visible when Enable Email Alerts is true |
| **Trigger on Null Percent** | Alert when NULL_PERCENT exceeds threshold. `Default: false`. Visible when Enable Email Alerts is true 
| **Null Percent Threshold (%)** | Alert if null percentage exceeds this value. `Default: 5`. Visible when Trigger on Null Percent is true |
| **Trigger on Blank Count** | Alert when BLANK_COUNT > 0. `Default: false`. Visible when Enable Email Alerts is true |
| **Trigger on Blank Percent** | Alert when BLANK_PERCENT exceeds threshold. `Default: false`. Visible when Enable Email Alerts is true |
| **Blank Percent Threshold (%)** | Alert if blank percentage exceeds this value. `Default: 5`. Visible when Trigger on Blank Percent is true |
| **Trigger on Duplicates** | Alert when DUPLICATE_COUNT > 0. `Default: true`. Visible when Enable Email Alerts is true |
| **Trigger on Freshness** | Alert when FRESHNESS exceeds threshold. `Default: true`. Visible when Enable Email Alerts is true |
| **Freshness Threshold (Hours)** | Alert if data older than this many hours. `Default: 24`. Visible when Trigger on Freshness is true |

**Email Alerting Options**

**Override Options**

**Alert Parameters**
```sql
{
    "DMF_ALERT_INTEGRATION": "my_email_integration",
    "DMF_ALERT_EMAIL": "xxxxxxxxxx@xxxxx.xxx",
    "DMF_ALERT_WAREHOUSE": "COMPUTE_WH",
    "DMF_ALERT_SCHEDULE": "90 MINUTE",
    "DMF_EMAIL_BODY": "",
    "DMF_EMAIL_SUBJECT": ""
}
```

### DMF Deployment

#### DMF Initial Deployment

When deployed for the first time into an environment the DMF node will update the metadata with DMFs.

| **Stage** | **Description** |
|-----------|----------------|
| **Create ** | This will execute a Alter statement and create a DMF entries in metadata |

#### DMF Redeployment

The subsequent deployment of DMF node with changes in DMFs definition, adding/dropping DMFs results in updating metadata with latest changes.

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.29+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

#### DMF Undeployment

If a DMF Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the all the DMFs will be dropped from metadata.

---

### Data Profiling

The Data Profiling node type creates a data profiling table containing details such as the object name, column name, metric function applied, and the corresponding profiling result. It supports operations like count, min, max, avg, and other statistical measures on source table columns. Additionally, this node type allows scheduling, enabling automated and periodic data profiling to maintain data quality and integrity.
The Data Profiling node type will create the Data Profiling table with the following fixed column structure: OBJECT_NAME, COLUMN_NAME, FUNCTION_APPLIED, and RESULT.
The Coalesce Mapping grid will display the source column list; however, the Data Profiling table created in Snowflake will maintain a fixed column structure.

The node name in Coalesce will be used as the name of the Snowflake Data Profiling table.

### Data Profiling Node Configuration

The Data Profiling node type has four configuration groups:

* [Node Properties](#node-properties)
* [Options](#options)
* [Counts](#counts)
* [Min/Max/Avg](#min-max-avg)
* [Column Details](#column-details)
* [Scheduling Options](#data-profiling-scheduling-options)

#### Node Properties

| **Option** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the Data Profiling table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### Options

| **Option** | **Description** |
|---------|-------------|
| **Schedule Task** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action.**True** - The insert data SQL will wrap in a task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. **False** - A table will be created and SQL will execute as a Run action|
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.True: Table is truncated before the DML operation runs.False: The DML operation runs without truncation. |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Sample Mode** | Options: Sample/Full Table**Sample**: Options: Percent/Fixed Number of Rows **Full Table**: Performs Data Profiling on all records of source table|

#### Counts

| **Option** | **Description** |
|---------|-------------|
| **Row Count** | True / False toggle that determines weather to add a record for the row count of the source table.**True** - Add a record for the row count of the source table. **False** - A record for the row count of the source table is not added.|
| **Distinct Count** | Add column/columns to get the count of distinct values of the selected column|
| **Null Count** | Add column/columns to get the count of null values of the selected column |
| **Not Null Count** | Add column/columns to get the count of not null values of the selected column |

#### Min / Max / Avg

| **Option** | **Description** |
|---------|-------------|
| **MAX Value** | Add column/columns to get the Maximum value of the selected column|
| **MIN Value** | Add column/columns to get the Minimum value of the selected column|
| **Average Value** | Add column/columns to get the Average value of the selected column|

#### Column Details

| **Option** | **Description** |
|---------|-------------|
| **Top 100 Value Distribution** | Add column/ columns to get the JSON array that contains the top 100 distinct values of the column along with their occurrence counts|
| **Max Data Length** | Add column/columns to get the maximum length of non-null values in the selected column|
| **Min Data Length** | Add column/columns to get the minimum length of non-null values in the selected column|
| **Standard Deviation** | Add column/columns to get  the standard deviation of the lengths of non-null selected column values|

#### Data Profiling Scheduling Options

![Task Scheduling Options](https://github.com/user-attachments/assets/2f8bc6c3-117c-4130-9619-45ee50feebef)

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Example of Serverless Task with Multiple Predecessors and Root Task

![Example_of_Serverless_Task](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/0af0c04a-5e88-42d1-85de-fd71a792436d)

#### Example of Warehouse Task With 60 Minute Task Schedule

![Example_of_Warehouse_Task](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/04afc6b2-aebc-4346-b1fe-33340e7df0bf)

#### Example of Warehouse Task With Cron Schedule Not Using a Stream

![Example_of_Warehouse_Task_with_Cron](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/89c85551-59af-44e6-b383-a9bdaacf7cce)

### Data Profiling Deployment

#### Deployment Parameters

The Data Profiling with Task includes an environment parameter that allows you to specify a different warehouse used to run a task in different environments.

The parameter name is `targetTaskWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT` the value entered in the Scheduling Options config Select Warehouse on which to run the task will be used when creating the task.

```json
{
    "targetTaskWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the task using a Snowflake warehouse with the specified value.

For example, with the below setting for the parameter in a QA environment, the task will execute using a warehouse named `compute_wh`.

```json
{
    "targetTaskWarehouse": "compute_wh"
}
```
The parameter is to be added in the Workspace settings and Environment settings for deployment

![image](https://github.com/user-attachments/assets/1a360cec-2517-4748-9a81-650d06f96850)

#### Data Profiling  With Task Initial Deployment

When deployed for the first time into an environment the Data Profiling  With Task node will execute the following stages:

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Data Profiling Table** | Creates a Data Profiling table that will be loaded by the task.Note that the structure of the data profiling table is defined in the create template and will not be same as the columns in the mapping grid of Coalesce node |
| **Create Task** | Creates task that will load the Data Profiling table on schedule |
| **Resume Task** | Resumes the task so it runs on schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called "Task DAG Resume Root." This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change.

#### Data Profiling With Task Redeployment

After initial deployment, changes in task schedule, warehouse, or scheduling options will result in a CREATE or RESUME

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Recreates task with new schedule |
| **Resume Task** | Resumes task with new schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Recreates task with new schedule |

#### Data Profiling With Task Altering Tables

The structure of the data profiling table is defined int the create template and cannot be changed so altering the table is not supported in redeployment.

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.29+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Data Profiling With Task Undeployment

If a Data Profiling with Task node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then all objects created by the node in the target environment will be dropped.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** |  Drop the data profiling table originally created to be loaded by the task. |
| **Drop Current Task** | Drop the task |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the data profiling table |
| **Suspend Root Task** | Drop a task from a DAG of task the root task needs to be put into a suspended state. |
| **Drop Task** | Drops the task |

-----------------

#### Node Type Switching Logic

| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Any Other | View(Static Data Metric) | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Table(Static Data Profile)  | 1. Warning (if applicable)2. Drop  3. Create |

Please review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |

--------------

## Code

### DMF Code

* [Node definition](https://github.com/coalesceio/data-quality-nodetypes/blob/main/nodeTypes/DMFS-448/definition.yml)
* [Create Template](https://github.com/coalesceio/data-quality-nodetypes/blob/main/nodeTypes/DMFS-448/create.sql.j2)
* [Macros](https://github.com/coalesceio/data-quality-nodetypes/blob/main/macros/macro-1.yml)

### Data Profiling Code

* [Node definition](https://github.com/coalesceio/data-quality-nodetypes/blob/main/nodeTypes/DataProfiling-416/definition.yml)
* [Create Template](https://github.com/coalesceio/data-quality-nodetypes/blob/main/nodeTypes/DataProfiling-416/create.sql.j2)
* [Run Template](https://github.com/coalesceio/data-quality-nodetypes/blob/main/nodeTypes/DataProfiling-416/run.sql.j2)
* [Macros](https://github.com/coalesceio/data-quality-nodetypes/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.2.0 | February 20, 2026 | Node Type Switching supported in Advanced Deploy Templates |
| 2.1.0 | December 31, 2025 | Custom, Alert and Universal DMF addition |
| 2.0.1 | November 13, 2025 | NM-190 Empty SQL Statement issue fixed |
| 2.0.0 | July 23, 2025 | Deploy phasing removed and added at appropriate places |
| 1.1.0 | April 02, 2025 | Added Data Profiling Node Type |
| 1.0.0 | March 31, 2025 | New node type Data Metric Functions added to Data Quality package |

---

## Data security package

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/data-security-package</code> <CopyButton textToCopy='@coalesce/snowflake/data-security-package' />
Supported Platform: 
Latest Version: 2.0.0 (June 19, 2026)

column-securitydata-governancedata-protectiondata-securitydynamic-maskingmaskingmasking-policypii-protectionsecure-viewsecuritysensitive-data

## Overview
Data Security package has node types focused on providing secured data

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/data-security-package</code> <CopyButton textToCopy='@coalesce/snowflake/data-security-package' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Data Security Package

The Coalesce Data Security Package helps you apply column-level data masking and row-level access policies to Snowflake views.

The package includes:

* [Dynamic Masking View](#dynamic-masking-view)
* [Masking Enabled Materialized View](#masking-enabled-materialized-view)
* [Code](#code)

---

## Dynamic Masking View

The Coalesce Dynamic Data Masking View Node type allows you to create a view with masking policies applied to a column within a table or view.

[Dynamic Data Masking](https://docs.snowflake.com/en/user-guide/security-column-ddm-use) is a Column-level Security feature that uses masking policies to selectively mask data at query time.

Depending on the masking policy conditions, the SQL execution context, and role hierarchy, Snowflake query operators may see the plain-text value, a partially masked value, or a fully masked value.
This Node type applies [column-level data masking](https://docs.snowflake.com/user-guide/security-column-intro) and [row-level access policy](https://docs.snowflake.com/en/user-guide/security-row-intro) to the target view.

### Prerequisites for Dynamic Masking View

* Create a Snowflake masking policy for column-level security and a row-level access policy for row-level security. This is used in the Node type to create a masked view.

Snowflake supports masking policies as a schema-level object to protect sensitive data from unauthorized access while allowing authorized users to access sensitive data at query runtime.

### Limitations of Dynamic Masking View

* A column can only be associated with one masking policy at a time.
* The input and output data types in a masking policy must match; you can't define a policy to target a timestamp and return a string.
* Once a materialized view is created from a table, you cannot set masking policies on any of its columns
* Cannot apply a masking policy to a table column if a materialized view is already created from the underlying table.
* A given table or view column can be specified in either a row access policy signature or a masking policy signature.

#### Examples

![image](https://github.com/user-attachments/assets/21b89bd5-60fb-4dfb-b1ed-c6e1eb7a6cb1)

![image](https://github.com/user-attachments/assets/b77cfcca-e465-429d-88c5-d6bcc0912890)

### Dynamic Masking View Node Configuration

The Dynamic Masking View Node type has 4 configuration groups:

* [Node Properties](#node-properties)
* [Options](#options)

#### Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the target view will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Options

| **Options** | **Description** |
|---------|-------------|
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination**False**: Single source node or multiple sources combined using a join |
| **Enable Column Masking** | Toggle: True/False Provides option to enable column masking |
| **Coalesce Storage Location of Data Masking Policy**| Enabled when Column Masking is true. Storage location in Coalesce where the Masking policy resides |
| **Snowflake Masking Policy**| Name of Snowflake masking policy to mask columns of available column patterns |
| **Override Masking columns**| Toggle: True/False Provides option to enable masking for specific column specified config |
| **Snowflake masking Column Name**| Enabled when Override Masking columns option is true. The column on which data masking is applied |
| **Snowflake masking policy Name**| Name of the Snowflake masking policy. Different masking policies for different columns are supported |
| **Enable row level security** | Toggle: True/False Provides option to enable row level access restriction |
| **Coalesce Storage Location of row access policy**| Enabled when row level security is true. Storage location in Coalesce where the Row access policy resides |
| **Row access policy name**| Name of Snowflake row access policy |
| **Row access column name**| The column name or names that control row-level access in the table |

### Enable Column Masking

![image](https://github.com/user-attachments/assets/5b4d6d43-acee-40b3-87bb-e90af053f7dd)

### Enable Row Level Security

![image](https://github.com/user-attachments/assets/d4886da5-9f92-4c76-b11f-831d257104d7)

### Dynamic Masking View Deployment

#### Dynamic Masking View Initial Deployment

When deployed for the first time into an environment, the View Node executes the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a View in the target environment |

#### Dynamic Masking View Redeployment

Subsequent deployment of the View Node with changes to the view definition, table description, secure option, or view name deletes the existing view and recreates it.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

#### Dynamic Masking View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level Environment, then the View in the target Environment is dropped.

This executes the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes the view from the environment |

---

## Masking Enabled Materialized View

The Coalesce Masking Enabled Materialized View Node type allows you to create a Snowflake Materialized View with masking policies and row access policies applied to sensitive data.

These features help protect sensitive information while allowing authorized users to access data based on their assigned roles and policy conditions.

This Node Type applies
* Column-level data masking using Snowflake Masking Policies.
* Row-level security using Snowflake Row Access Policies.
* Materialized View clustering and secure view options.

## Prerequisites for Masking Enabled Materialized View

Before using this Node type:

* Create the required Snowflake Masking Policies.
* Create the required Snowflake Row Access Policies.
* Ensure the policies are deployed in a Snowflake schema accessible by the target environment.
* Configure the corresponding Storage Locations in Coalesce.

Snowflake masking policies are schema-level objects that protect sensitive data from unauthorized access while still allowing authorized users to access data at query runtime.

## Limitations of Masking Enabled Materialized View

* A column can only have one masking policy attached at a time.
* The input and output data types in a masking policy must match.
* A given column can participate in either a masking policy signature or a row access policy signature.
* Materialized Views only support deterministic expressions.
* Changes to source columns, transformations, joins, or materialized view definitions may require the Materialized View to be recreated.
* Snowflake limitations applicable to Materialized Views also apply to this Node type.

## Masking Enabled Materialized View Node Configuration

The Masking Enabled Materialized View Node type contains the following configuration groups:

* [Node Properties](#node-properties)
* [Options](#options)

#### Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the target view will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detected If FALSE the Node will not be deployed or will be dropped during redeployment |

#### Options

| **Options** | **Description** |
|---------|-------------|
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Cluster key** | True/False to determine whether Materialized view is to be clustered or not- **True**: Allows you to specify the column based on which clustering is to be done  -**Allow Expressions Cluster Key**: True allows to add an expression to the specified cluster key- **False**: No clustering done |
| **Secure** | True / False Toggle to determine whether Materialized view to be created in a secured mode- **True**: Materialized view created in a secured mode- **False**: No additional secure option added during Materialized view creation |
| **Enable Column Masking** | Toggle: True/False Provides option to enable column masking |
| **Coalesce Storage Location of Data Masking Policy**| Enabled when Column Masking is true. Storage location in Coalesce where the Masking policy resides |
| **Snowflake Masking Policy**| Name of Snowflake masking policy to mask columns of available column patterns |
| **Override Masking columns**| Toggle: True/False Provides option to enable masking for specific column specified config |
| **Snowflake masking Column Name**| Enabled when Override Masking columns option is true. The column on which data masking is applied |
| **Snowflake masking policy Name**| Name of the Snowflake masking policy. Different masking policies for different columns are supported |
| **Enable row level security** | Toggle: True/False Provides option to enable row level access restriction |
| **Coalesce Storage Location of row access policy**| Enabled when row level security is true. Storage location in Coalesce where the Row access policy resides |
| **Row access policy name**| Name of Snowflake row access policy |
| **Row access column name**| The column name or names that control row-level access in the table |

## Column Masking Behavior

### Default Masking Policy

When **Enable Column Masking** is enabled and **Override Masking Columns** is disabled, the selected Snowflake Masking Policy is automatically applied to supported column patterns.

Supported column patterns include:

```text
PHONE
ADDRESS
NAME
ACCOUNT
EMAIL
INCOME
```

Any column containing one of these patterns in its name will automatically receive the selected masking policy.

### Override Masking Columns

When **Override Masking Columns** is enabled, specific columns can be assigned their own masking policies.

This allows different columns within the same Materialized View to use different masking policies.

Example:

| Column | Policy      |
| ------ | ----------- |
| SSN    | SSN_MASK    |
| SALARY | SALARY_MASK |
| EMAIL  | EMAIL_MASK  |

## Row Level Security Behavior

When **Enable Row Level Security** is enabled, a Snowflake Row Access Policy is attached to the Materialized View.

The selected columns are passed to the Row Access Policy signature.

> **Note:** The number and order of columns selected in Coalesce must match the columns defined in the Snowflake Row Access Policy signature. If the policy is created using one column, select one column in Coalesce. If the policy is created using multiple columns, select the corresponding columns in the same order.

## Masking Enabled Materialized View Deployment

### Initial Deployment

When deployed for the first time, the Node executes a CREATE OR REPLACE statement to create the Materialized View.

The following stage is executed:

| Stage                                    | Description                                                                   |
| ---------------------------------------- | ----------------------------------------------------------------------------- |
| Create Masking Enabled Materialized View | Creates the Materialized View with configured masking and row access policies |

## Redeployment

During redeployment, the Node intelligently determines whether a CREATE or ALTER operation is required.

### CREATE Operations

The Materialized View is recreated when:

* Source columns change.
* Column transformations change.
* Source joins change.
* Materialization type changes.
* DISTINCT configuration changes.

The following stages may be executed:

| Stage                                    | Description                            |
| ---------------------------------------- | -------------------------------------- |
| Drop Materialized View                   | Removes the existing Materialized View |
| Create Masking Enabled Materialized View | Creates the updated Materialized View  |

### ALTER Operations

The following changes are handled through ALTER statements without recreating the Materialized View:

#### Materialized View Properties

| Stage                                   | Description                           |
| --------------------------------------- | ------------------------------------- |
| Alter Materialized View - Rename        | Renames the Materialized View         |
| Alter Materialized View - Comment       | Updates the Materialized View comment |
| Alter Materialized View - Secure Option | Sets or removes Secure Mode           |
| Recluster Materialized View             | Updates clustering configuration      |
| Resume Recluster Materialized View      | Resumes automatic reclustering        |

#### Column Masking

| Stage                  | Description                                                        |
| ---------------------- | ------------------------------------------------------------------ |
| Set Masking Policy     | Applies a masking policy to a column                               |
| Replace Masking Policy | Replaces an existing masking policy                                |
| Unset Masking Policy   | Removes a masking policy from a column                             |
| Apply Default Policy   | Applies the selected node-level masking policy                     |
| Remove Override Policy | Removes an override masking policy and restores node-level masking |

#### Row Level Security

| Stage                     | Description                                           |
| ------------------------- | ----------------------------------------------------- |
| Add Row Access Policy     | Attaches a Row Access Policy                          |
| Drop Row Access Policy    | Removes a Row Access Policy                           |
| Replace Row Access Policy | Replaces an existing Row Access Policy with a new one |

## Undeployment

If the Node is removed from a Workspace and the changes are deployed, the Materialized View is dropped from the target environment.

The following stage is executed:

| Stage                  | Description                                        |
| ---------------------- | -------------------------------------------------- |
| Drop Materialized View | Removes the Materialized View from the environment |

## Code

#### Dynamic Masking View
* [Node definition](https://github.com/coalesceio/data-security/blob/main/nodeTypes/DynamicMaskingView-455/definition.yml)
* [Create Template](https://github.com/coalesceio/data-security/blob/main/nodeTypes/DynamicMaskingView-455/create.sql.j2)

#### Masking Enabled Materialized View
* [Node definition](https://github.com/coalesceio/data-security/blob/main/nodeTypes/MaskingEnabledMaterializedView-178/definition.yml)
* [Create Template](https://github.com/coalesceio/data-security/blob/main/nodeTypes/MaskingEnabledMaterializedView-178/create.sql.j2)

[Macros](https://github.com/coalesceio/data-security/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.0.0 | June 19, 2026 | NM-240 Add Masking Enabled Materialized View node to the Data Security package |
| 1.0.1 | October 24, 2025 | Fix for processing and identifier quotes issues |
| 1.0.0 | April 10, 2025 | Dynamic Masking View node type released |

---

## Data Vault by Scalefree

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/data-vault-by-scalefree</code> <CopyButton textToCopy='@coalesce/snowflake/data-vault-by-scalefree' />
Supported Platform: 
Latest Version: 3.0.0 (October 07, 2025)

audit trailbusiness vaultdata vaultdata vault 2.0enterprise data warehousehistorizationhublinkraw vaultsatellite

## Overview
Data Vault for Coalesce powered by Scalefree.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/data-vault-by-scalefree</code> <CopyButton textToCopy='@coalesce/snowflake/data-vault-by-scalefree' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
Welcome to datavault4coalesce!

***

The following documentation sheds some light on the node types that have been developed by Scalefree to make your Coalesce-Experience more comfortable!
The documentation can be found by clicking the links in the sidebar on the right side. In the documentation, the macros and their parameters are explained and further exemplified.

> **_NOTE:_**  Coalesce has released a Hands-On Guide for using Datavault4Coalesce. Check it our [here](https://guides.coalesce.io/data-vault/index.html#0)!

### Included node types
- Staging Area 
- Hubs, Links & Satellites 
- PIT Tables
- Snapshot Control

With datavault4coalesce you will get a lot of awesome features, including:
### Features
- Ghost Records
- Multi Batch Processing
- Automatic PIT cleanup based on logarithmic snapshot logic
- Pattern based on years of experience in Data Vault Loading
- Virtual Load Enddate

### Requirements

To use the node types efficiently, there are a few prerequisites you need to provide:
- Coalesce Environment connected to a Snowflake Instance
- Basic Data Vault Knowledge (If this is new to you, check the following [Article](https://www.scalefree.com/what-is-data-vault/))

### Installation

To have access to the Datavault4Coalesce node definitions, you have to write an email to support@coalesce.io and request that the package is added to your environment!

### Resources:
- Learn more about coalesce [in the docs](https://docs.coalesce.io/docs)
- Contact us via datavault4coalesce@scalefree.com
- Check out the [Scalefree-Blog](https://www.scalefree.com/blog/)
- Blog Articles:
   - [Datavault4Coalesce Announcement](https://www.scalefree.com/scalefree-newsletter/bring-your-data-vault-automation-to-the-next-level-with-datavault4coalesce/)
   - [PIT Tables](https://www.scalefree.com/scalefree-newsletter/point-in-time-tables-insurance/)
   - [Hash Keys in Data-Vault](https://www.scalefree.com/architecture/hash-keys-in-the-data-vault/)
- [Scalefree Github Repo](https://github.com/ScalefreeCOM/datavault4coalesce.git)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 3.0.0 | October 07, 2025 | https://github.com/ScalefreeCOM/datavault4coalesce/releases/tag/v1.3.1 |
| 2.0.1 | March 12, 2025 | See https://github.com/ScalefreeCOM/datavault4coalesce/releases/tag/v1.3.0 |
| 2.0.0 | March 10, 2025 | See https://github.com/ScalefreeCOM/datavault4coalesce/releases/tag/v1.3.0 |
| 1.1.5 | November 11, 2024 | Hash Algorithm and explicit column selection |
| 1.1.4 | November 05, 2024 | Updated macros in package |
| 1.0.0 | June 11, 2024 | Init |

---

## Deferred Merge

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/deferred-merge</code> <CopyButton textToCopy='@coalesce/snowflake/deferred-merge' />
Supported Platform: 
Latest Version: 2.0.3 (April 08, 2026)

cdcchange data capturehybrid viewlambda architecturemergereal-timesoft deletestreamingupsert

## Overview
Deferred Merge package nodes are a simplified implementation of a Lambda architecture, focusing on the integration of an ongoing real-time or near real-time updates with previously loaded data.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/deferred-merge</code> <CopyButton textToCopy='@coalesce/snowflake/deferred-merge' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Snowflake - Deferred Merge - Brief Summary

- **Deferred Merge - Append Stream** Streamlines high-frequency data ingestion by capturing incremental changes (CDC) from source tables or views. These nodes efficiently stage new records, updates, and deletes into a buffer, providing the necessary agility for real-time pipelines while shielding large base tables from the performance overhead of constant micro-partition rewrites.
- **Deferred Merge - Delta Stream** Manages the intermediate delta layer by processing change streams into a queryable, state-aware buffer. By handling record versioning and complex DML logic natively, these nodes ensure that the most recent record states are immediately available via hybrid views, maintaining a "single source of truth" while deferring heavy-duty merge operations to optimized, lower-frequency schedules.

----

## Nodetypes Config Matrix

| Category | Feature | Append Stream | Delta Stream |
| :--- | :--- | :---: | :---: |
| **General** | Development Mode  | ✅ | ✅ |
| **General** | Create Target As (Table/Transient) | ✅ | ✅ |
| **Stream** | Source Object  | ✅ | ✅ |
| **Stream** | Show Initial Rows | ✅ | ✅ |
| **Stream** | Redeployment Behavior | ✅ | ✅ |
| **Loading** | Table Keys (Business Keys) | ✅ | ✅ |
| **Loading** | Record Versioning (Latest Tracking) | ✅ | ✅ |
| **DML Ops** | Column Identifier | ✅ | ✅ |
| **DML Ops** | Include Value for Update | ✅ | ✅ |
| **DML Ops** | Insert Value | ✅ | ✅ |
| **DML Ops** | Delete Value | ✅ | ✅ |
| **Delete** | Soft Delete Toggle | ✅ | ✅ |
| **Delete** | Retain Last Non-Deleted Values | ✅ | ✅ |
| **Clustering** | Cluster Key Support | ✅ | ✅ |
| **Clustering** | Allow Expressions in Cluster Key | ✅ | ✅ |
| **Scheduling** | Scheduling Mode | ✅ | ✅ |
| **Scheduling** | When Source Stream has Data Flag | ✅ | ✅ |
| **Scheduling** | Warehouse Selection | ✅ | ✅ |
| **Scheduling** | Initial Serverless Size | ✅ | ✅ |
| **Scheduling** | Task Schedule (Minutes/Cron/Predecessor) | ✅ | ✅ |
| **Scheduling** | Predecessor Task Management | ✅ | ✅ |
| **Scheduling** | Root Task Name Selection | ✅ | ✅ |

---

The Deferred Merge Package includes mechanisms for handling merge operations with different streaming strategies:

* [Deferred Merge Append Stream](#deferred-merge-append-stream)
* [Deferred Merge Delta Stream](#deferred-merge-delta-stream)

---

## Deferred Merge Append Stream

The Deferred Merge - Append Stream Node includes several key steps to ensure efficient and up-to-date data processing. First, a stream is created to capture row inserts. Then a target table is created and initially loaded with data. A hybrid view is established to provide access to the most current data by combining initial updates.

Finally, a scheduled task manages ongoing updates by merging changes into the target table. This process ensures that the target table remains synchronized with the source data maintaining accuracy and timeliness.

### Deferred Merge Append Stream Node Configuration

The Deferred Merge Append node has the following configuration groups:

* [General Options](#append-stream-general-options)  
* [Stream Options](#append-stream-options)
* [Target Loading Options](#append-stream-target-loading-options)
* [Target Row DML Operations](#append-stream-target-row-dml-operations)
* [Target Delete Options](#append-stream-target-delete-options)
* [Target Clustering Options](#append-stream-target-clustering-options)
* [Scheduling Options](#append-stream-scheduling-options)

#### Append Stream General Options

![Append General Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/e6a7d6a5-da3b-4001-b84f-24e2a85708e7)

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True/False toggle that determines task creation:- **True**: Table created and SQL executes as Run action- **False**: SQL wraps into task with Scheduling Options. Displays a message prompting the user to wait or run manually when executed. |
| **Create Target As** | Choose target object type:- **Table**: Creates table- **Transient Table**: Creates transient table |

Prior to creating a task, it is helpful to test the SQL the task will execute to make sure it runs without errors and returns the expected data.

#### Append Stream Options

![Append Stream Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/c4daf590-5a13-49bd-b389-3b3cdc8f40af)

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation (Required):- **Table**- **View** |
| **Show Initial Rows** | **True**: Returns only rows that existed when stream created**False**: Returns DML changes since most recent offset |
| **Redeployment Behavior** | Determines stream recreation behavior |

| **Redeployment Behavior** | **Stage Executed** |
|---|---|
| **Create Stream if not exists**| Re-Create Stream at existing offset|
| **Create or Replace** | Create Stream|
| **Create at existing stream**|  Re-Create Stream at existing offset |

#### Append Stream Target Loading Options

![Append Target Loading Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/77856b16-3bd6-47f5-a8ad-ffc2f73888dd)

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | Business keys columns for merging into target table |
| **Record Versioning** | Date Time or Date and Timestamp column for latest record tracking |

#### Append Stream Target Row DML Operations

![Append Target Row DML Operations](https://github.com/coalesceio/Deferred-Merge/assets/169126315/507731aa-d5a1-49b9-9b14-bc7dbf335cab)

| **Option** | **Description** |
|------------|----------------|
| **Column Identifier** | Column identifying DML operations |
| **Include Value for Update** |  For records flagged under Update, the existing records in the target table are updated with the corresponding values from the source table. |
| **Insert Value** | It indicates that the corresponding record is meant to be inserted into the target table. This condition ensures that only records flagged for insertion are actually inserted into the target table during the merge operation.|
| **Delete Value** | This value indicates that the corresponding record should either be soft-deleted (if the condition is met by enabling the soft delete toggle) or hard-deleted from the target table. |

#### Append Stream Target Delete Options

![Append Target Delete Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/6fe8d21b-80ea-41b0-a56a-5f9ccb7b3874)

| **Option** | **Description** |
|------------|----------------|
| **Soft Delete** | Toggle to maintain deleted data record |
| **Retain Last Non-Deleted Values** | Preserves most recent non-deleted record, even as other records are marked as deleted or become inactive. |

#### Append Stream Target Clustering Options

![Append Target Clustering Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/6b168402-3bec-47d9-9870-be72e871f404)

| **Option** | **Description** |
|------------|----------------|
| **Cluster key** | **True**: Specify clustering column and allow expressions**False**: No clustering implemented |
| **Allow Expressions Cluster Key**| Aadd an expression to the specified cluster key|

#### Append Stream Scheduling Options

If development mode is set to false then Scheduling Options can be used to configure how and when the task will run.

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Append Stream Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Append Stream Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Append Stream Limitations

> 🚧 **Appyling Transformation**
> This node can't apply transformations to the columns for this node type.

### Append Stream Deployment

#### Append Stream Parameters

It includes an environment parameter that allows you to specify a different warehouse to run a task in different environments.
The parameter name is `targetTaskWarehouse`, and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Scheduling Options config "Select Warehouse on which to run the task" will be used when creating the task.

```json
{"targetTaskWarehouse": "DEV ENVIRONMENT"}
```

When set to any value other than `DEV ENVIRONMENT`, the node will attempt to create the task using a Snowflake warehouse with the specified value.
For example, with the setting below for the parameter in a QA environment, the task will execute using a warehouse named `compute_wh`.

```json
{"targetTaskWarehouse": "compute_wh"}
```

#### Append Stream Initial Deployment

When deployed for the first time into an environment, executes the following stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Executes CREATE OR REPLACE statement to create Stream in target environment |
| **Create Target Table** | Creates destination table for processed data storage |
| **Target Table Initial Load** | Populates the target table with the existing data from the source object. This step ensures that the target table starts with a complete set of data before any further changes are applied |
| **Create Hybrid View** | Provides access to the most up-to-date data by combining the initial data load with any subsequent changes captured by the stream. The hybrid view ensures that users always have access to the latest version of the data without waiting for batch updates.|
| **Create Task** | Creates merge operation task for stream changes |
| **Resume Task** | Activates the created task for scheduled execution |
| **Apply Table Clustering** | Alters table with cluster key if enabled |
| **Resume Recluster Clustering** | Enables periodic reclustering to maintain optimal clustering |

##### Append Stream Predecessor Task Deployment

When deploying with predecessor tasks, executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task to add task into DAG |
| **Create Task** | Creates task for loading target table |

> 📘 **Task DAG Note**
>
> For tasks in a DAG, include Task Dag Resume Root node type to resume root node after dependent task creation. Tasks use CREATE OR REPLACE only, with no ALTER capabilities.

##### Append Stream Redeployment

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| **Create Stream if not exists** | Re-Create Stream at existing offset |
| **Create or Replace** | Create Stream |
| **Create at existing stream** | Re-Create Stream at existing offset |

##### Append Stream Table Redeployment

The following column/table changes trigger ALTER statements:

* Table name changes
* Column drops
* Column data type alterations
* New column additions

Executes these stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Table Operations** | Executes appropriate ALTER statements for schema changes |
| **Target Initial Load** | Executes load based on configuration:- If initial load enabled + "Create or Replace": Uses INSERT OVERWRITE- All other scenarios: Uses INSERT INTO |

##### Append Stream View Redeployment

Stream or table changes trigger Hybrid View recreation.

##### Append Stream Task Redeployment

Stream or table changes trigger:

1. Task recreation
2. Task resumption

> 🚧 Redeployment Behavior
>
> Redeployment with changes in Stream/Table/Task properties will result in execution of all steps mentioned in inital deployment.

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Append Stream Undeployment

When node is deleted, executes:

* Drop Stream
* Drop Table/Transient Table  
* Drop View
* Drop Current Task

---

## Deferred Merge Delta Stream

The Deferred Merge - Delta Stream Node includes several key steps to ensure efficient and up-to-date data processing. First, a stream is created to capture changes from the source object to tracks all DML changes to the source object, including inserts, updates, and deletes. Then a target table is created and initially loaded with data. A hybrid view is established to provide access to the most current data by combining initial updates.

Finally, a scheduled task manages ongoing updates by merging changes into the target table. This process ensures that the target table remains synchronized with the source data maintaining accuracy and timeliness

### Deferred Merge Delta Stream Node Configuration

The Deferred Merge Append node has the following configuration groups:

* [General Options](#append-stream-general-options)  
* [Stream Options](#append-stream-options)
* [Target Loading Options](#append-stream-target-loading-options)
* [Target Row DML Operations](#append-stream-target-row-dml-operations)
* [Target Delete Options](#append-stream-target-delete-options)
* [Target Clustering Options](#append-stream-target-clustering-options)
* [Scheduling Options](#append-stream-scheduling-options)

#### Delta Stream Stream General Options

![Append General Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/e6a7d6a5-da3b-4001-b84f-24e2a85708e7)

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True/False toggle that determines task creation:- **True**: Table created and SQL executes as Run action- **False**: SQL wraps into task with Scheduling Options. Displays a message prompting the user to wait or run manually when executed. |
| **Create Target As** | Choose target object type:- **Table**: Creates table- **Transient Table**: Creates transient table |

Prior to creating a task, it is helpful to test the SQL the task will execute to make sure it runs without errors and returns the expected data.

#### Delta Stream Options

![Append Stream Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/c4daf590-5a13-49bd-b389-3b3cdc8f40af)

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation (Required):- **Table**- **View** |
| **Show Initial Rows** | **True**: Returns only rows that existed when stream created**False**: Returns DML changes since most recent offset |
| **Redeployment Behavior** | Determines stream recreation behavior |

| **Redeployment Behavior** | **Stage Executed** |
|---|---|
| **Create Stream if not exists**| Re-Create Stream at existing offset|
| **Create or Replace** | Create Stream|
| **Create at existing stream**|  Re-Create Stream at existing offset |

#### Delta Stream Target Loading Options

![Append Target Loading Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/77856b16-3bd6-47f5-a8ad-ffc2f73888dd)

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | Business keys columns for merging into target table |
| **Record Versioning** | Date Time or Date and Timestamp column for latest record tracking |

#### Delta Stream Target Row DML Operations

![Append Target Row DML Operations](https://github.com/coalesceio/Deferred-Merge/assets/169126315/507731aa-d5a1-49b9-9b14-bc7dbf335cab)

| **Option** | **Description** |
|------------|----------------|
| **Column Identifier** | Column identifying DML operations |
| **Include Value for Update** |  For records flagged under Update, the existing records in the target table are updated with the corresponding values from the source table. |
| **Insert Value** | It indicates that the corresponding record is meant to be inserted into the target table. This condition ensures that only records flagged for insertion are actually inserted into the target table during the merge operation.|
| **Delete Value** | This value indicates that the corresponding record should either be soft-deleted (if the condition is met by enabling the soft delete toggle) or hard-deleted from the target table. |

#### Delta Stream Target Delete Options

![Append Target Delete Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/6fe8d21b-80ea-41b0-a56a-5f9ccb7b3874)

| **Option** | **Description** |
|------------|----------------|
| **Soft Delete** | Toggle to maintain deleted data record |
| **Retain Last Non-Deleted Values** | Preserves most recent non-deleted record, even as other records are marked as deleted or become inactive. |

#### Delta Stream Target Clustering Options

![Append Target Clustering Options](https://github.com/coalesceio/Deferred-Merge/assets/169126315/6b168402-3bec-47d9-9870-be72e871f404)

| **Option** | **Description** |
|------------|----------------|
| **Cluster key** | **True**: Specify clustering column and allow expressions**False**: No clustering implemented |
| **Allow Expressions Cluster Key**| Aadd an expression to the specified cluster key|

#### Delta Stream Scheduling Options

If development mode is set to false then Scheduling Options can be used to configure how and when the task will run.

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Delta Stream Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Delta Stream Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Delta Stream Limitations

> 🚧 **Appyling Transformation**
> This node can't apply transformations to the columns for this node type.

### Delta Stream Deployment

#### Delta Stream Parameters

It includes an environment parameter that allows you to specify a different warehouse to run a task in different environments.
The parameter name is `targetTaskWarehouse`, and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Scheduling Options config "Select Warehouse on which to run the task" will be used when creating the task.

```json
{"targetTaskWarehouse": "DEV ENVIRONMENT"}
```

When set to any value other than `DEV ENVIRONMENT`, the node will attempt to create the task using a Snowflake warehouse with the specified value.
For example, with the setting below for the parameter in a QA environment, the task will execute using a warehouse named `compute_wh`.

```json
{"targetTaskWarehouse": "compute_wh"}
```

#### Delta Stream Initial Deployment

When deployed for the first time into an environment, executes the following stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Executes CREATE OR REPLACE statement to create Stream in target environment |
| **Create Target Table** | Creates destination table for processed data storage |
| **Target Table Initial Load** | Populates the target table with the existing data from the source object. This step ensures that the target table starts with a complete set of data before any further changes are applied |
| **Create Hybrid View** | Provides access to the most up-to-date data by combining the initial data load with any subsequent changes captured by the stream. The hybrid view ensures that users always have access to the latest version of the data without waiting for batch updates.|
| **Create Task** | Creates merge operation task for stream changes |
| **Resume Task** | Activates the created task for scheduled execution |
| **Apply Table Clustering** | Alters table with cluster key if enabled |
| **Resume Recluster Clustering** | Enables periodic reclustering to maintain optimal clustering |

##### Delta Stream Predecessor Task Deployment

When deploying with predecessor tasks, executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task to add task into DAG |
| **Create Task** | Creates task for loading target table |

> 📘 **Task DAG Note**
>
> For tasks in a DAG, include Task Dag Resume Root node type to resume root node after dependent task creation. Tasks use CREATE OR REPLACE only, with no ALTER capabilities.

##### Delta Stream Redeployment

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| **Create Stream if not exists** | Re-Create Stream at existing offset |
| **Create or Replace** | Create Stream |
| **Create at existing stream** | Re-Create Stream at existing offset |

##### Delta Stream Table Redeployment

The following column/table changes trigger ALTER statements:

* Table name changes
* Column drops
* Column data type alterations
* New column additions

Executes these stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Table Operations** | Executes appropriate ALTER statements for schema changes |
| **Target Initial Load** | Executes load based on configuration:- If initial load enabled + "Create or Replace": Uses INSERT OVERWRITE- All other scenarios: Uses INSERT INTO |

##### Delta Stream View Redeployment

Stream or table changes trigger Hybrid View recreation.

##### Delta Stream Task Redeployment

Stream or table changes trigger:

1. Task recreation
2. Task resumption

> 🚧 Redeployment Behavior
>
> Redeployment with changes in Stream/Table/Task properties will result in execution of all steps mentioned in inital deployment.

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Delta Stream Undeployment

When node is deleted, executes:

* Drop Stream
* Drop Table/Transient Table  
* Drop View
* Drop Current Task
* 
### Redeployment with no changes
 
If the nodes are redeployed with no changes compared to previous deployment, then no stages are executed

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| SIM or DSM | Table | 1. Warning(if applicable) 2. Drop/Alter(depending on Create Target As config) 3. Create(if applicable) |
| Table | Table | 1. Warning (if applicable) 2. Drop 3. Create|
| Any other Task | Table | 1. Warning (if applicable) 2. Drop 3. Create |
| Any Other | Table | 1. Warning (if applicable)2. Drop  3. Create |

**Note:** SIM and DSM nodes contain a **Create Target As** configuration similar to **Deferred Merge - Append** and **Delta Stream**. When switching from SIM/DSM, the system performs an **Alter** if this configuration matches the desired state, or a **Drop and Create** if it differs. For all other task or table types, this configuration is absent and is treated as "blank" in the current state, triggering a mandatory **Drop and Create** to correctly initialize the Deferred Merge table properties.

Please review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Stream for Directory Table (and vice versa) | Metadata columns are not automatically synchronized. Specific directory columns (e.g., `relative_path`, `size`, `md5`) are not added when switching to Directory Table, nor are they removed when switching back to a standard Stream. |
| 11 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added when there's a node type switch |
| 12 | Deferred Merge - Append and Delta Stream | Any Other(and vice versa) | System columns are not automatically managed, They are neither removed nor added when there's a node type switch. These must be manually dropped or added before redeployment. |

--------------

## Code

### Deferred Merge - Append Stream

* [Node definition](https://github.com/coalesceio/Deferred-Merge/blob/main/nodeTypes/DeferredMerge-AppendStream-300/definition.yml)
* [Create Template](https://github.com/coalesceio/Deferred-Merge/blob/main/nodeTypes/DeferredMerge-AppendStream-300/create.sql.j2)

### Deferred Merge - Delta Stream 

* [Node definition](https://github.com/coalesceio/Deferred-Merge/blob/main/nodeTypes/DeferredMerge-DeltaStream-301/definition.yml)
* [Create Template](https://github.com/coalesceio/Deferred-Merge/blob/main/nodeTypes/DeferredMerge-DeltaStream-301/create.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.0.3 | April 08, 2026 | NM-209 - Task Node Enhancements - Notification Integrations, Execute As User, and Advanced Task Para |
| 2.0.2 | March 17, 2026 | NM-220 Added Node Type Switch support in advance deploy templates |
| 2.0.1 | November 13, 2025 | NM-190 Empty SQL Statement issue fixed and Scheduling Options group bug fix |
| 2.0.0 | July 31, 2025 | Deploy phases at appropriate stages |
| 1.1.4 | March 17, 2025 | Missing Parameter Warning Added |

---

## Dynamic Tables

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/dynamic-tables</code> <CopyButton textToCopy='@coalesce/snowflake/dynamic-tables' />
Supported Platform: 
Latest Version: 2.1.8 (June 09, 2026)

auto-refreshdeclarative pipelinesdynamic tablesincrementallag specificationmaterializedreal-time

## Overview
Dynamic tables simplify data engineering in Snowflake by providing a reliable, cost-effective, and automated way to transform data.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/dynamic-tables</code> <CopyButton textToCopy='@coalesce/snowflake/dynamic-tables' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Snowflake - Dynamic Tables - Brief Summary

- **Dynamic Table Work**
Serve as intermediate processing layers within a data pipeline. These nodes handle complex transformations, data cleaning, and filtering before data reaches final consumption. They optimize performance by breaking down logic into manageable, automated steps.

- **Dynamic Table Dimension**
Provide essential business context, such as attributes for customers, products, or locations. These nodes use declarative logic to ensure descriptive data is automatically synchronized with source changes, allowing for seamless slicing and dicing of metrics in reports.

- **Dynamic Table Latest Record version**
Ensure data integrity by automatically surfacing the most recent state of a record. By handling deduplication and versioning natively, these nodes provide a "single source of truth" for the current status of business entities, eliminating the need for complex manual cleanup queries.

**Summary:**
Together, these node types leverage Snowflake’s declarative orchestration to provide a low-maintenance, automated framework for real-time data modeling. They ensure that business insights are built on fresh, accurate, and high-performance data structures.

----

## Nodetypes Config Matrix

| Category | Feature | DT Work | DT Dimension | DT Latest Record |
| :--- | :--- | :---: | :---: | :---: |
| **Create** | Create As Dynamic Table | ✅ | ✅ | ✅ |
 **Create** | Create As Dynamic Transient Table | ✅ | ✅ | ✅ |
| **Create** | Warehouse | ✅ | ✅ | ✅ |
| **Create** | Advanced Warehouse Selection | ✅ | ✅ | ✅ |
| **Create** |  Initialization Warehouse Size | ✅ | ✅ | ✅ |
| **Create** | Initialize | ✅ | ✅ | ✅ |
| **Create** | Cluster Key | ✅ | ✅ | ✅ | 
| **Refresh** | Refresh Warehouse size | ✅ | ✅ | ✅ |
| **Refresh** | Lag Specification (Time/Period) | ✅ | ✅ | ✅ |
| **Refresh** | Downstream Lag | ✅ | ✅ | ✅ |
| **Refresh** | Refresh Mode | ✅ | ✅ | ✅ |
| **Refresh** | Backfill Options | ✅ | ✅ | ✅ |
| **Logic** | Distinct / Group By All | ✅ | ⬜ | ⬜ |
| **Logic** | Table Key(s) / Business Key | ⬜ | ✅ | ✅ |
| **Logic** | Record Versioning Logic | ⬜ | ✅ | ✅ |
| **Logic** | Sequence / Ordering Column | ⬜ | ✅ | ✅ |
| **Logic** | Timestamp-track Data Load | ⬜ | ✅ | ✅ |
| **Options** | Copy Grants | ✅ | ✅ | ✅ |
| **Options** | Immutability Constraint | ✅ | ✅ | ✅ |
| **Others** | Enable Tests | ✅ | ✅ | ✅ |
| **Others** | Pre-SQL / Post-SQL | ✅ | ✅ | ✅ |

---

# Dynamic Tables Package

Package includes:

* [Dynamic Table Work](#dynamic-table-work)
* [Dynamic Table Dimension](#dynamic-table-dimension)
* [Dynamic Table Latest Record Version](#dynamic-table-latest-record-version)

---

## Dynamic Table Work

The Coalesce Dynamic Table Work UDN is a versatile node that allows you to develop and deploy a single Dynamic Table Work or a DAG of Dynamic Tables in Snowflake.

[Dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about) are a new table type offered by Snowflake that allow data teams to use SQL statements to declaratively define the results of data pipelines. Dynamic tables simplify the process of creating and managing data pipelines by streamlining data transformations without having to manage Streams and Tasks. 

### Dynamic Table Work Node Configuration

The Dynamic Table Work has three configuration groups:

* [Node Properties](#dynamic-table-work-node-properties)
* [Dynamic Table Options](#dynamic-table-work-options) 
* [General Options](#dynamic-table-work-general-options)
* [Advanced Options](#dynamic-table-work-advanced-options)

#### Dynamic Table Work Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Dynamic Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Dynamic Table Work Options

| **Option** | **Description** |
|------------|----------------|
| **Warehouse** | - (Required when Advance Warehouse is disabled) Name of warehouse used to refresh the Dynamic Table.|
| **Advance Warehouse Selection** |A toggle that enables size-based warehouse configuration for Dynamic Tables- **Refresh Warehouse**: Selects the warehouse (by size) used for regular refresh operations.- **Initialization Warehouse**: Selects the warehouse (by size) used during the initial creation or backfill of the Dynamic Table. - When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**. |
| **Downstream** | (Required) True/False toggle:- **True**: Refresh on demand when dependent tables need refresh- **False**: Set Lag Specification for refresh schedule |
| **Lag Specification** | Only if Downstream is False. Review [Snowflakes Dynamic Tables Refresh](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh) to understand how to specify the target lag. Set refresh schedule with:- **Time Value**: Frequency of the refresh- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh Mode** | Specifies refresh type:- **BLANK('')**: If the blank option is selected, the default behavior will trigger an INCREMENTAL refresh.- **AUTO**: Default incremental refresh. If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | Initial refresh behavior:- **BLANK('')**: If the blank option is selected, the default behavior will be ON_CREATE.- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |

#### Dynamic Table Work Advanced Options

| **Option** | **Description** |
|------------|----------------|
| **Copy grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created.Useful during replication.[More info on replication here](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Immutability Constraint** |True/False toggle:- True: Applies an IMMUTABLE condition to the Dynamic Table, preventing changes to data that matches the defined rule- False: No immutability is enforced; data can be updated normally |
| **Immutable Where Expression** | Visible when Immutability Constraint is enabled.- SQL condition used to identify rows that are considered immutable (no longer change).- This expression must reference valid dynamic table columns and should be deterministic. |
| **Enable Backfill** | - Visible only when Immutability Constraint is enabled.- True/False toggle:- True: Displays Backfill Options group. Refer [Snowflake Dynamic Tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-create) to understand how Immutability and Backfill features work.|

#### Dynamic Table Work Backfill Options

| **Option** | **Description** |
|------------|----------------|
| **Backfill Source Schema**| (Optional) Schema name of the backfill table.- If provided, the backfill table is read from this schema.- If left blank, the current node’s schema is used by default. |
| **Backfill Source Table**| Specifies a source table used to load historical data into the Dynamic Table.|
|**Time Travel**| Visible when Backfill option in enabled. True/False toggle to create the table with Time Travel options. |
|**Time Travel Type**| If Time Travel parameters AT/BEFORE are specified, data from the backfill table is copied at the specified time. |
| **Time Travel Reference** | Dropdown to specify how Snowflake should time travel the backfill source data:- **OFFSET**: Uses a relative time offset from the current time.- **STATEMENT**: Uses a specific Snowflake query ID to time travel the data to the moment that query was executed. |
| **Time Travel Value** | Value depends on the selected Time Travel Reference:- **OFFSET**: Provide a negative integer value (for example: -60, -120, -1440) representing time in minutes before the current time.- **STATEMENT**: Provide a valid query ID of a completed DML from the query history that falls within the table’s time travel retention period. |

#### Dynamic Table Work General Options

![dynamic table1 1_general](https://github.com/coalesceio/Dynamic-Table-Nodes/assets/7216836/2c663f13-8e7c-43c0-a7e0-c4abc1edfa14)

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True/False toggle to return DISTINCT rows |
| **Group By All** | True/False toggle to add non-aggregated columns to GROUP BY |
| **Multi Source** | True/False toggle for combining multiple sources via UNION or UNION ALL |
| **Create As** | Choose 'dynamic table' or 'transient dynamic table' |
| **Cluster key** | True/False toggle for clustering:- **True**: Specify clustering column and optional expressions- **False**: No clustering |
| **Allow Expressions Cluster Key**| When cluster key is set to true. Allows to add an expression to the specified cluster key| 

### Dynamic Table Work Deployment

#### Dynamic Table Work Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

```json
{
    "targetDynamicTableWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

```json
{
    "targetDynamicTableWarehouse": "compute_wh"
}
```

#### Advance Warehouse selection(Optional)
* Advanced Warehouse Selection allows using **separate warehouses for initialization and refresh** based on workload size.
* Users can select different sizes for refresh vs initialization to optimize cost/performance

#### Prerequisite

* Advanced Warehouse must be **enabled**.
* The required warehouses must already exist in Snowflake.
* Corresponding warehouse parameters must be defined in the deployment environment.
  
#### Example of parameter initialization
  ```json
{
    "warehouseSizesDict": {
        "xsDynamicTableWarehouse": "dev_wh_xs",
        "sDynamicTableWarehouse": "dev_wh_s",
        "lDynamicTableWarehouse": "dev_wh_l"
    },
  "targetDynamicTableWarehouse": "DEV ENVIRONMENT",
}
```
> **Note:** `dev_wh_xs`, `dev_wh_s`, and `dev_wh_l` are example warehouse names. Users must replace these values with the names of the Snowflake warehouses they have already created and want to use in their environment.

### Warehouse sizes supported in node config UI and corresponding parameter details

| Size        | Environment Parameter                                   |
| ----------- | ------------------------------------------------------- |
| X-Small     | `xsDynamicTableWarehouse`                               |
| Small       | `sDynamicTableWarehouse`                                |
| Medium      | `mDynamicTableWarehouse`                                |
| Large       | `lDynamicTableWarehouse`                                |
| X-Large     | `xlDynamicTableWarehouse`                               |
| 2X-Large    | `2xlDynamicTableWarehouse`                              |
| 3X-Large    | `3xlDynamicTableWarehouse`                              |
| 4X-Large    | `4xlDynamicTableWarehouse`                              |
| 5X-Large    | `5xlDynamicTableWarehouse`                              |
| 6X Large    | `6xlDynamicTableWarehouse`                              |

#### Notes

* These parameters are **required only when Advanced Warehouse is enabled**.
* When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**.

> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters 

### Dynamic Table Work Initial Deployment

When deployed for the first time into an environment the Dynamic Table Work node will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Dynamic Table/Dynamic Transient Table** | This stage will execute a `CREATE OR REPLACE` statement and create a Dynamic Table in the target environment. |

### Deploying a DAG of Dynamic Tables

When a DAG of related Dynamic Tables are deployed together Coalesce will deploy the Dynamic Tables in the order that the Dynamic Tables are ordered.

#### Dynamic Table Work Redeployment

After initial deployment, subsequent deployments may alter or recreate the Dynamic Table.

#### Altering the Dynamic Table Work

The following config changes trigger ALTER statements:

1. Warehouse name
2. Downstream setting  
3. Lag specification
4. Immutability Constraint
5. Advance Warehouse

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Also if the location of the node, node name, column level description, and table level description results in an `ALTER` statement, whereas other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

### Changing Materialization Type

If the materialization type changes in dynamic table config options, the following steps gets executed:

1. **Drop table**
2. **Create Work dynamic table**
3. **Apply Table Clustering(if cluster key option is provided)**
4. **Resume Recluster Table(if cluster key option is provided)**

### Recreating the Dynamic Table

If anything changes other than the configuration options specified in [Altering the Dynamic Table](#altering-the-dynamic-table) then the Dynamic Table will be recreated by running a `CREATE OR REPLACE `statement.

If the changes in node results in recreating the Dynamic table,then following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Table is dropped before recreating in case the node name or location is changed |
| **Create Work Dynamic table/Dynamic transient table**| Dynamic table is created|

### Redeploying a DAG of Dynamic Tables Work

If an entire DAG of Dynamic Tables has been deployed and changes are made to a deployed Dynamic Table Coalesce will only redeploy Dynamic Tables that have changed metadata.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dynamic Tables Work Undeployment

A table will be dropped if all of these are true:

* The Dynamic Work Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher level environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Dynamic Table** | Removes table from target environment |

---

## Dynamic Table Dimension

The Coalesce Dynamic Table Dimension UDN is a versatile node that allows you to develop and deploy a single Dynamic Table Dimension or a DAG of Dynamic Tables in Snowflake.

[Dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about) are a new table type offered by Snowflake that allow data teams to use SQL statements to declaratively define the results of data pipelines. Dynamic tables simplify the process of creating and managing data pipelines by streamlining data transformations without having to manage Streams and Tasks.

### Dimension Node Configuration

The Dynamic Table Dimension has four configuration groups:

* [Node Properties](#dimension-node-properties)
* [Dynamic Table Options](#dimension-table-options)
* [Dimension Options](#dimension-options)
* [General Options](#dimension-general-options)
* [Advanced Options](#dimension-advanced-options)

#### Dimension Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension Table Options

| **Option** | **Description** |
|------------|----------------|
| **Warehouse** | (Required when Advance Warehouse is disabled) Name of warehouse used to refresh the Dynamic Table |
| **Advance Warehouse Selection** |A toggle that enables size-based warehouse configuration for Dynamic Tables- **Refresh Warehouse**: Selects the warehouse (by size) used for regular refresh operations.- **Initialization Warehouse**: Selects the warehouse (by size) used during the initial creation or backfill of the Dynamic Table. - When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**.|
| **Downstream** | (Required) True/False toggle:- **True**: Refresh on demand when dependent tables need refresh- **False**: Set Lag Specification for refresh schedule |
| **Lag Specification** | Only if Downstream is False. Review [Snowflakes Dynamic Tables Refresh](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh) to understand how to specify the target lag. Set refresh schedule with:- **Time Value**: Frequency of refresh for a given Time Period.- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh Mode** | Specifies refresh type:- **BLANK('')**: If the blank option is selected, the default behavior will trigger an INCREMENTAL refresh.- **AUTO**: Default incremental refresh. If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | Initial refresh behavior:- **BLANK('')**: If the blank option is selected, the default behavior will be ON_CREATE.- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |

#### Dimension Options

![Dimension options](https://github.com/coalesceio/Dynamic-Table-Nodes/assets/169126315/b3f8d788-b8a8-4884-b9bd-2ff85039d9df)

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | (Required) Business key columns for Dimension key formation |
| **Record versioning** | (Required) Type of column for history maintenance:- Datetime column- Date and Time column- Integer column |
| **Timestamp** | Required if Datetime column chosen for Record versioning.**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |
| **Sequence** | Required if Integer column chosen for Record versioning|
| **Timetamp-track data load**| Required if Integer column chosen for Record versioning**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |
| **Date/Timestamp Columns** | Required if Date and Time columns chosen for Record versioning**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |

#### Dimension General Options

![dynamictable1 1-dimgeneral](https://github.com/coalesceio/Dynamic-Table-Nodes/assets/7216836/dda1c53c-a3da-4701-8449-0023db1ac44a)

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Choose 'dynamic table' or 'transient dynamic table' |
| **Cluster key** | True/False toggle for clustering:- **True**: Specify clustering column and optional expressions- **False**: No clustering |
| **Allow Expressions Cluster Key**| When cluster key is set to true. Allows to add an expression to the specified cluster key|

#### Dimension Advanced Options

| **Option** | **Description** |
|------------|----------------|
| **Copy grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created.Useful during replication.[More info on replication here](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Immutability Constraint** |True/False toggle:- True: Applies an IMMUTABLE condition to the Dynamic Table, preventing changes to data that matches the defined rule- False: No immutability is enforced; data can be updated normally |
| **Immutable Where Expression** | Visible when Immutability Constraint is enabled.- SQL condition used to identify rows that are considered immutable (no longer change).- This expression must reference valid dynamic table columns and should be deterministic. |
| **Enable Backfill** | - Visible only when Immutability Constraint is enabled.- True/False toggle:- True: Displays Backfill Options group. Review [Snowflake Dynamic Tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-create) to understand how Immutability and Backfill features work.|

#### Dynamic Table Dimension Backfill Options

| **Option** | **Description** |
|------------|----------------|
| **Backfill Source Schema**| (Optional) Schema name of the backfill table.- If provided, the backfill table is read from this schema.- If left blank, the current node’s schema is used by default. |
| **Backfill Source Table**| Specifies a source table used to load historical data into the Dynamic Table.|
|**Time Travel**| Visible when Backfill option in enabled. True/False toggle to create the table with Time Travel options. |
|**Time Travel Type**| If Time Travel parameters AT/BEFORE are specified, data from the backfill table is copied at the specified time. |
| **Time Travel Reference** | Dropdown to specify how Snowflake should time travel the backfill source data:- **OFFSET**: Uses a relative time offset from the current time.- **STATEMENT**: Uses a Snowflake query ID to time travel data, but is not supported for Dynamic Table Dimension backfill. |
| **Time Travel Value** | Value depends on the selected Time Travel Reference:- **OFFSET**: Provide a negative integer value (for example: -60, -120, -1440) representing time in minutes before the current time.-  **STATEMENT**: Provide a valid query ID of a completed DML from the query history that falls within the table’s time travel retention period. |

### Dimension Deployment

#### Dimension Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

```json
{
    "targetDynamicTableWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

```json
{
    "targetDynamicTableWarehouse": "compute_wh"
}
```

#### Advance Warehouse selection(Optional)
* Advanced Warehouse Selection allows using **separate warehouses for initialization and refresh** based on workload size.
* Corresponding warehouse parameters must be defined in the deployment environment.

#### Prerequisite

* Advanced Warehouse must be **enabled**.
* The required warehouses must already exist in Snowflake.
* Corresponding warehouse parameters must be defined in the deployment environment.
  
#### Example of parameter initialization
  ```json
{
    "warehouseSizesDict": {
        "xsDynamicTableWarehouse": "dev_wh_xs",
        "sDynamicTableWarehouse": "dev_wh_s",
        "lDynamicTableWarehouse": "dev_wh_l"
    },
  "targetDynamicTableWarehouse": "DEV ENVIRONMENT",
}
```
> **Note:** `dev_wh_xs`, `dev_wh_s`, and `dev_wh_l` are example warehouse names. Users must replace these values with the names of the Snowflake warehouses they have already created and want to use in their environment.

### Warehouse sizes supported in node config UI and corresponding parameter details

| Size        | Environment Parameter                                   |
| ----------- | ------------------------------------------------------- |
| X-Small     | `xsDynamicTableWarehouse`                               |
| Small       | `sDynamicTableWarehouse`                                |
| Medium      | `mDynamicTableWarehouse`                                |
| Large       | `lDynamicTableWarehouse`                                |
| X-Large     | `xlDynamicTableWarehouse`                               |
| 2X-Large    | `2xlDynamicTableWarehouse`                              |
| 3X-Large    | `3xlDynamicTableWarehouse`                              |
| 4X-Large    | `4xlDynamicTableWarehouse`                              |
| 5X-Large    | `5xlDynamicTableWarehouse`                              |
| 6X Large    | `6xlDynamicTableWarehouse`                              |

#### Notes

* These parameters are **required only when Advanced Warehouse is enabled**.
* When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**.

> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters 
 
### Dimension Initial Deployment

When deployed for the first time into an environment the Dynamic Table Work node will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Dynamic Table/Dynamic Transient Table** | This stage will execute a `CREATE OR REPLACE` statement and create a Dynamic Table in the target environment. |

### Deploying a DAG of Dimension Tables

When a DAG of related Dynamic Tables are deployed together Coalesce will deploy the Dynamic Tables in the order that the Dynamic Tables are ordered.

#### Dimension Redeployment

After initial deployment, subsequent deployments may alter or recreate the Dynamic Table.

#### Altering the Dimension Table

The following config changes trigger ALTER statements:

1. Warehouse name
2. Downstream setting  
3. Lag specification
4. Immutability Constraint
5. Advance Warehouse

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Also if the location of the node, node name, column level description, and table level description results in an `ALTER` statement, whereas other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

### Changing Materialization Type 

If the materialization type changes in dynamic table config options, the following steps gets executed:

1. **Drop table**
2. **Create dimension table**
3. **Apply Table Clustering(if cluster key option is provided)**
4. **Resume Recluster Table(if cluster key option is provided)**

### Recreating the Dimension Table

If anything changes other than the configuration options specified in [Altering the Dimension Table](#altering-the-dimension-table) then the Dynamic Table will be recreated by running a `CREATE OR REPLACE`statement.

If the changes in node results in recreating the Dynamic table, then following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Table is dropped before recreating in case the node name or location is changed |
| **Create Work Dynamic table/Dynamic transient table**| Dynamic table is created|

### Redeploying a DAG of Dimension

If an entire DAG of Dynamic Tables has been deployed and changes are made to a deployed Dynamic Table Coalesce will only redeploy Dynamic Tables that have changed metadata.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dynamic Table Dimension Undeployment

A table will be dropped if all of these are true:

* The Dynamic Dimension Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher level environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Dynamic Table** | Removes table from target environment |

---

## Dynamic Table Latest Record Version

The Coalesce Dynamic Table Latest Record Version UDN is a versatile node that allows you to develop and deploy a single Dynamic Table Work or a DAG of Dynamic Tables with only the latest version of rows in Snowflake.

[Dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about. ) are a new table type offered by Snowflake that allow data teams to use SQL statements to declaratively define the results of data pipelines. Dynamic tables simplify the process of creating and managing data pipelines by streamlining data transformations without having to manage Streams and Tasks.

### Latest Record Version Node Configuration

The Dynamic Table Dimension has four configuration groups:

* [Node Properties](#latest-record-version-node-properties)
* [Dynamic Table Options](#latest-record-version-options)
* [Dimension Options](#latest-record-version-dimension-options)
* [General Options](#latest-record-version-general-options)
* [Advanced Options](#latest-record-version-advanced-options)

#### Latest Record Version Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Latest Record Verion Table Options

| **Option** | **Description** |
|------------|----------------|
| **Warehouse** | (Required when Advance Warehouse is disabled) Name of warehouse used to refresh the Dynamic Table |
| **Advance Warehouse Selection** |A toggle that enables size-based warehouse configuration for Dynamic Tables- **Refresh Warehouse**: Selects the warehouse (by size) used for regular refresh operations.- **Initialization Warehouse**: Selects the warehouse (by size) used during the initial creation or backfill of the Dynamic Table. - When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**.|
| **Downstream** | (Required) True/False toggle:- **True**: Refresh on demand when dependent tables need refresh- **False**: Set Lag Specification for refresh schedule |
| **Lag Specification** | Only if Downstream is False. Review [Snowflakes Dynamic Tables Refresh](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh) to understand how to specify the target lag. Set refresh schedule with:- **Time Value**: Frequency of refresh for a given Time Period.- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh Mode** | Specifies refresh type:- **BLANK('')**: If the blank option is selected, the default behavior will trigger an INCREMENTAL refresh.- **AUTO**: Default incremental refresh. If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | Initial refresh behavior:- **BLANK('')**: If the blank option is selected, the default behavior will be ON_CREATE.- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |

#### Latest Record Version Options

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | (Required) Business key columns for Dimension key formation |
| **Record versioning** | (Required) Type of column for history maintenance:- Datetime column- Date and Time column- Integer column |
| **Timestamp** | Required if Datetime column chosen for Record versioning.**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |
| **Sequence** | Required if Integer column chosen for Record versioning|
| **Timetamp-track data load**| Required if Integer column chosen for Record versioning**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |
| **Date/Timestamp Columns** | Required if Date and Time columns chosen for Record versioning**Note:If multiple columns are chosen.The first timestamp column chosen is considered for versioning order** |

#### Latest Record Version General Options

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Choose 'dynamic table' or 'transient dynamic table' |
| **Cluster key** | True/False toggle for clustering:- **True**: Specify clustering column and optional expressions- **False**: No clustering |
| **Allow Expressions Cluster Key**| When cluster key is set to true. Allows to add an expression to the specified cluster key|

#### Latest Record Version Advanced Options

| **Option** | **Description** |
|------------|----------------|
| **Copy grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created.Useful during replication.[More info on replication here](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Immutability Constraint** |True/False toggle:- True: Applies an IMMUTABLE condition to the Dynamic Table, preventing changes to data that matches the defined rule- False: No immutability is enforced; data can be updated normally |
| **Immutable Where Expression** | Visible when Immutability Constraint is enabled.- SQL condition used to identify rows that are considered immutable (no longer change).- This expression must reference valid dynamic table columns and should be deterministic. |
| **Enable Backfill** | - Visible only when Immutability Constraint is enabled.- True/False toggle:- True: Displays Backfill Options group. Review [Snowflake Dynamic Tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-create) to understand how Immutability and Backfill features work.|

#### Latest Record Version Backfill Options

| **Option** | **Description** |
|------------|----------------|
| **Backfill Source Schema**| (Optional) Schema name of the backfill table.- If provided, the backfill table is read from this schema.- If left blank, the current node’s schema is used by default. |
| **Backfill Source Table**| Specifies a source table used to load historical data into the Dynamic Table.|
|**Time Travel**| Visible when Backfill option in enabled. True/False toggle to create the table with Time Travel options. |
|**Time Travel Type**| If Time Travel parameters AT/BEFORE are specified, data from the backfill table is copied at the specified time. |
| **Time Travel Reference** | Dropdown to specify how Snowflake should time travel the backfill source data:- **OFFSET**: Uses a relative time offset from the current time.- **STATEMENT**: Uses a Snowflake query ID to time travel data, but is not supported for Dynamic Table Dimension backfill. |
| **Time Travel Value** | Value depends on the selected Time Travel Reference:- **OFFSET**: Provide a negative integer value (for example: -60, -120, -1440) representing time in minutes before the current time.-  **STATEMENT**: Provide a valid query ID of a completed DML from the query history that falls within the table’s time travel retention period. |

### DAG of Dynamic Table Latest Record Version

When designing DAG of Dynamic tables, you should specify the target lag. Review [Understanding dynamic table refresh - Snowflake](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh)

### Latest Record Version Deployment

#### Latest Record Version Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

```json
{
    "targetDynamicTableWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

```json
{
    "targetDynamicTableWarehouse": "compute_wh"
}
```

#### Advance Warehouse selection(Optional)
* Advanced Warehouse Selection allows using **separate warehouses for initialization and refresh** based on workload size.
* Corresponding warehouse parameters must be defined in the deployment environment.

#### Prerequisite

* Advanced Warehouse must be **enabled**.
* The required warehouses must already exist in Snowflake.
* Corresponding warehouse parameters must be defined in the deployment environment.
  
#### Example of parameter initialization
  ```json
{
    "warehouseSizesDict": {
        "xsDynamicTableWarehouse": "dev_wh_xs",
        "sDynamicTableWarehouse": "dev_wh_s",
        "lDynamicTableWarehouse": "dev_wh_l"
    },
  "targetDynamicTableWarehouse": "DEV ENVIRONMENT",
}
```
> **Note:** `dev_wh_xs`, `dev_wh_s`, and `dev_wh_l` are example warehouse names. Users must replace these values with the names of the Snowflake warehouses they have already created and want to use in their environment.

### Warehouse sizes supported in node config UI and corresponding parameter details

| Size        | Environment Parameter                                   |
| ----------- | ------------------------------------------------------- |
| X-Small     | `xsDynamicTableWarehouse`                               |
| Small       | `sDynamicTableWarehouse`                                |
| Medium      | `mDynamicTableWarehouse`                                |
| Large       | `lDynamicTableWarehouse`                                |
| X-Large     | `xlDynamicTableWarehouse`                               |
| 2X-Large    | `2xlDynamicTableWarehouse`                              |
| 3X-Large    | `3xlDynamicTableWarehouse`                              |
| 4X-Large    | `4xlDynamicTableWarehouse`                              |
| 5X-Large    | `5xlDynamicTableWarehouse`                              |
| 6X Large    | `6xlDynamicTableWarehouse`                              |

#### Notes

* These parameters are **required only when Advanced Warehouse is enabled**.
* When using Advanced Warehouse, the **`targetDynamicTableWarehouse` parameter is not required**.

> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters 
 
### Latest Record Version Initial Deployment

When deployed for the first time into an environment the Dynamic Table Work node will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Dimension Dynamic Table/Dynamic Transient Table** | This stage will execute a `CREATE OR REPLACE` statement and create a Dynamic Table in the target environment. |

### Deploying a DAG of Latest Record Version Tables

When a DAG of related Dynamic Tables are deployed together Coalesce will deploy the Dynamic Tables in the order that the Dynamic Tables are ordered.

#### Latest Record Version Redeployment

After initial deployment, subsequent deployments may alter or recreate the Dynamic Table.

#### Altering the Latest Record Version Table

The following config changes trigger ALTER statements:

1. Warehouse name
2. Downstream setting  
3. Lag specification
4. Immutability Constraint
5. Advance Warehouse

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Also if the location of the node, node name, column level description, and table level description results in an `ALTER` statement, whereas other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

### Changing Materialization Type

If the materialization type changes in dynamic table config options, the following steps gets executed:

1. **Drop transient dimension table**
2. **Create transient dimension table**
3. **Apply Table Clustering(if cluster key option is provided)**
4. **Resume Recluster Table(if cluster key option is provided)**

### Recreating the Latest Record Version Table

If anything changes other than the configuration options specified in Altering the Latest Record Version Table, then the Dynamic Table will be recreated by running a `CREATE OR REPLACE`statement.

If the changes in node results in recreating the Dynamic table, then following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/transient table** | Table is dropped before recreating in case the node name or location is changed |
| **Create Work Dynamic table/Dynamic transient table**| Dynamic table is created|

### Redeploying a DAG of Latest Record Version

If an entire DAG of Dynamic Tables has been deployed and changes are made to a deployed Dynamic Table Coalesce will only redeploy Dynamic Tables that have changed metadata.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Latest Record Version Undeployment

A table will be dropped if all of these are true:

* The Dynamic Latest Record Version Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher level environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Dynamic Table** | Removes table from target environment |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Dynamic Table | Dynamic Table | Follows existing redeployment stages |
| Dynamic Transient Table | Dynamic Transient Table | Follows existing redeployment stages |
| Any Other | Dynamic Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Dynamic Transient Table | 1. Warning (if applicable)2. Drop  3. Create |

Please review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |

--------------

## Code

### Dynamic Table Work Code

* [Node definition](https://github.com/coalesceio/Dynamic-Table-Nodes/tree/main/nodeTypes/DynamicTableWork-347/definition.yml)
* [Create Template](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/nodeTypes/DynamicTableWork-347/create.sql.j2)

### Dynamic Table Dimension Code

* [Node definition](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/nodeTypes/DynamicTableDimension-350/definition.yml)
* [Create Template](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/nodeTypes/DynamicTableDimension-350/create.sql.j2)

### Dynamic Table Latest Record Version Code

* [Node definition](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/nodeTypes/DynamicTableLatestRecordVersion-351/definition.yml)
* [Create Template](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/nodeTypes/DynamicTableLatestRecordVersion-351/create.sql.j2)

### Macros

* [Macros](https://github.com/coalesceio/Dynamic-Table-Nodes/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.1.8 | June 09, 2026 | Validate Select fix with "'ns' is undefined" in the Dynamic Table package is done. |
| 2.1.7 | February 26, 2026 | Node Type Switching supported in Advance Deploy templates |
| 2.1.6 | January 05, 2026 | Added Advance Warehouse, Immutability Constraint and Backfill Features |
| 2.1.5 | December 26, 2025 | Fix for Missing Handler for Incoming Node Type Transitions (View/Table → Dynamic Table) |
| 2.1.4 | November 07, 2025 | Mocro update and blank option added to Refresh Mode and Initialize |
| 2.1.3 | October 06, 2025 | Dynamic Iceberg Table moved to Iceberg Tables Package |
| 2.1.2 | September 26, 2025 | Integer versioning column support, transform support for tstamp columns,multiple tstamp notsupported |
| 2.1.0 | August 13, 2025 | Advanced option Copy grants added to all dynamic table node types |
| 1.1.11 | June 06, 2025 | Deployment fix-single source to multi-source |
| 1.1.10 | April 22, 2025 | Adding data type definition to dynamic table node types. |
| 1.1.9 | March 27, 2025 | Fix done for dynamic dimension to use hash key as table key |
| 1.1.8 | March 07, 2025 | Force Users to Enter Parameters When Required |
| 1.1.7 | February 24, 2025 | Redeployment fixes for join clause |

---

## External Data Package(Package)

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/external-data-package</code> <CopyButton textToCopy='@coalesce/snowflake/external-data-package' />
Supported Platform: 
Latest Version: 2.1.3 (March 30, 2026)

apiazure blobcloud storagecsvdata exportdata ingestionexcelexternal stagefile loadinggcsjdbcjsonparquetrest apis3schema inferenceseed

## Overview
Provides tools for loading and managing external data in Snowflake, including schema inference, data copying, and unloading, API integration, and parsing of Excel files.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/external-data-package</code> <CopyButton textToCopy='@coalesce/snowflake/external-data-package' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# External Data Package

The Coalesce External Data Package includes:

* [CopyInto](#CopyInto)
* [Snowpipe](#Snowpipe)
* [External Table](#external-tables)
* [Inferschema](#inferschema)
* [CopyUnload](#CopyUnload)
* [API](#API)
* [API-NODEUPDATE](#API-NODEUPDATE)
* [JDBC LOAD](#jdbc-load)
* [Parse Excel](#parse-excel)
* [Parse Json](#parse-json)
* [Gitseed](#gitseed)
* [Code](#code)

# External Data Package - Business Summary

## Overview

The External Data Package is a comprehensive toolkit for managing external data ingestion and processing in Snowflake through Coalesce. It provides organizations with multiple pathways to load, transform, and manage data from various sources into their Snowflake environment.

---

## Key Business Capabilities

### 1. Flexible Data Loading Options

- **CopyInto**: Batch load data from staged files into Snowflake tables with full control over file formats, error handling, and data validation
- **Snowpipe**: Enable continuous, automated data ingestion for near real-time analytics without manual intervention
- **GitSeed**: Version-controlled data seeding directly from Git repositories, enabling reproducible data environments across development, testing, and production

### 2. External Data Integration

- **External Tables**: Query data stored in cloud storage (AWS, GCP, Azure) without moving it into Snowflake, reducing storage costs
- **API Node**: Connect to external REST APIs to pull data from third-party systems and SaaS applications
- **JDBC Load**: Integrate with traditional databases (SQL Server, Oracle, etc.) for hybrid data architectures

### 3. Intelligent Schema Management

- **InferSchema**: Automatically detect file structures and column definitions, reducing manual configuration and accelerating time-to-value
- Supports case-sensitive column handling for maintaining data fidelity across systems

### 4. Advanced Processing

- **Parse Excel/JSON**: Handle large, complex files that exceed standard size limits
- **CopyUnload**: Export Snowflake data back to cloud storage for sharing or archival purposes

---

## Business Value

### Reduced Time-to-Insight
Automated schema detection and continuous ingestion minimize manual setup

### Cost Optimization
External tables and selective data loading reduce storage costs

### Risk Mitigation
Built-in error handling, validation, and version control ensure data quality

### Scalability
Cloud-native architecture handles data volumes from megabytes to petabytes

### Compliance
Secure credential management and audit trails support regulatory requirements

---

 CopyInto 

### CopyInto Node Configuration

The Copy-Into node type the following configurations available:

* [Node Properties](#copy-into-node-properties)
* [General Options](#copy-into-general-options)
* [Source Data](#copy-into-file-location)
* [File Format](#copy-into-file-format)
* [Extended CSV File Format Options](#extended-csv-file-format-options)
* [Copy Options](#copy-into-copy-options)

CopyInto - Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### Key points to use CopyInto Node

* CopyInto node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* CopyInto node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to Inferschema to know more on how to use the node and add Copy-Into on top of it.
* The data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config or reload parameter
* The path or subfolder name inside stage where the file is located can be specified using config 'Path or subfolder'.Do not prefix or suffix '/' in path name.Example,one level 'SUBFOLDER',two levels 'SUBFOLDER/INNERFOLDER'.
* Blank options added to Copy-Into and CSV file format options to revert any specific option

### Use CopyInto to load columns with insensitive case
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button with DDLCaseInsensitive toggle ON to execute create table based on inferred schema
* Table is created with column names in default case followed by Snowflake(UPPERCASE)
* Re-Sync columns to have all the columns with default case in mapping grid,so that the nodes added upstream follow the same.

### Use CopyInto node with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema
* This is mainly a test to make sure create will work
* Hit Run button to execute DML

![Resync](https://github.com/user-attachments/assets/de3c4ce2-370e-43d0-a010-fc6131cd8669)

If the above works, it should be deployable as is.  Deploy will simply take the columns and execute a create table.

  CopyInto - General Options 

**InferSchema-True**

![CopyInto](https://github.com/user-attachments/assets/a537faac-bb91-4a00-b055-4612934d95b9)

**InferSchema-False**

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Select from the options to create as Table or Transient Table- **Transient table**  -**Table** |
| **TruncateBefore(Disabled when Inferschema is true)** | True / False toggle that determines whether or not a table is to be truncated before reloading - **True**: Table is truncated and Copy-Into statement is executed to reload the data into target table- **False**: Data is loaded directly into target table and no truncate action takes place. |
| **InferSchema** | True / False toggle that determines whether or not to infer the columns of file before loading - **True**: The node is created with the inferred columns- **False**: No infer table step is executed |
| **DDL Case Insensitive**| True/False toggle that determines whether to preserve the case of columns from source file - **True**: The create and data load preserves the case of columns- **False**: The columns used in create and data load are case insensitive[How to use the toggle](#use-copyinto-to-load-columns-with-insensitive-case) |
      
  CopyInto - Source Data 

**InferSchema-true**

![image](https://github.com/user-attachments/assets/2f549fe3-648f-4132-bcdf-2ab92f6554bd)

**InferSchema-false**

![image](https://github.com/user-attachments/assets/c1071dff-da18-45fc-9e43-1dd5cdd03329)

##### Internal or External Stage

| **Setting** | **Description** |
|---------|-------------|
| **Coalesce Storage Location of Stage** | A storage location in Coalesce where the stage is located.|
| **Stage Name (Required)** | Internal or External stage where the files containing data to be loaded are staged|
| **File Name(s)(Ex:a.csv,b.csv)** | Specifies a list of one or more files names (separated by commas) to be loaded |
| **Path or subfolder** | Not mandatory.Specifies the path or subfolders inside the stage where the file is located.Ensure that '/' is not pre-fixed before or after the subfolder name|
| **File Pattern (Ex:'.*hea.*[.]csv')**| A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match |

![CopyInto-file location2](https://github.com/user-attachments/assets/0c5949c4-0286-4250-91cf-38325e2c312a)

##### External location
| **Setting** | **Description** |
|---------|-------------|
| **External URI** | Enter the URI of the External location. This URI can point to either a private or public bucket/container.|
| **Storage Integration** | Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity. This field is **not mandatory** for publicly accessible buckets or when another authentication method (such as presigned URLs or public access) is used. |

 CopyInto - File Format 

**File format definition-File format name**

![Copy-Into format](https://github.com/user-attachments/assets/9e4e8f7e-29be-4209-a08b-cb19fadccf05)

**File format definiton-File format values**

![copy-into-file-format](https://github.com/user-attachments/assets/2a737c5f-3bb3-471a-9365-bf3251677415)

##### File format config-Inferschema

* It is not mandatory to create a file format in snowflake to infer the structure of the file
* If the file format definition is set to 'File format values',a temporary file format is created based on the config values and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and is of file type csv,a temporary file format is created to adapt the same to infer the structure and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and the file types except csv,the same file format is used for inferring.No temporary file formts are created.
* The temporary file format created follows the naing convention ==>`TEMP_{{file_type}}_{{node name}}`.Ensure that any predefined file formats created in snowflake does not have the same file format name.
  
##### File Format Definition - File Format Name

| **Setting** | **Description** |
|---------|-------------|
|* **File Format Name**|Specifies an existing named file format to use for loading data into the table|
|* **File Type**| CSV  JSON  ORC  AVRO  PARQUET  XML |

##### File Format Definition - File Format Values

| **Setting** | **Description** |
|---------|-------------|
|**File Format Values**|Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Record delimiter**-Characters that separate records in an input file**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Field optionally enclosed by**- Character used to enclose strings(octal notation preferred)**Number of header lines to skip**- Number of lines at the start of the file to skip**Parse Header(InferSchema-true)**-Boolean that specifies whether to use the first row headers in the data files to determine column names.**Trim Space**- Boolean that specifies whether to remove white space from fields  **Replace invalid characters**- Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Encoding**- Specifies the character set of the source data when loading data into a table **Date format**- String that defines the format of date values in the data files to be loaded. **Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded.[Refer to extended options here](#extended-CSV-file-format-options)|
|**JSON**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character **Trim Space** - Boolean that specifies whether to remove white space from fields**Strip Outer Array**- Boolean that instructs the JSON parser to remove outer brackets [ ] **Date format**- String that defines the format of date values in the data files to be loaded**Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded|
|**ORC**|**Trim Space** - Specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**AVRO**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**PARQUET**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character**Using vectorised scanner** - Boolean that specifies whether to use a vectorized scanner for loading Parquet files|
|**XML**|**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|

 Extended CSV File Format Options 

| **Setting** | **Description** |
|---------|-------------|
|**CSV**|**Skip blank lines**- Boolean that specifies to skip any blank lines encountered in the data files  **Nulls for empty field**- When loading data, specifies whether to insert SQL NULL for empty fields in an input file**Skip byte order mark**-Boolean that specifies whether to skip the BOM (byte order mark), if present in a data file.**Parsing error for column mismatch**-Boolean that specifies whether to generate a parsing error if the number of delimited columns (i.e. fields) in an input file does not match the number of columns in the corresponding table.**Multiple lines**-Boolean that specifies whether multiple lines are allowed.If MULTI_LINE is set to FALSE and the specified record delimiter is present within a CSV field, the record containing the field will be interpreted as an error.**Binary format**-Defines the encoding format for binary input or output.  **Replace values with NULL**-o specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. **Escape optionally enclosed by**-A singlebyte character string used as the escape character for unenclosed field values only.Octal notation is preferred.**Escape enclosed/unenclosed values**-A singlebyte character string used as the escape character for enclosed or unenclosed field values.Octal notation is preferred|

 CopyInto - Copy Options 

| **Setting** | **Description** |
|---------|-------------|
|**On Error Behavior**|-String (constant) that specifies the error handling for the load operation|
|                     |* CONTINUE* SKIP_FILE* SKIP_FILE_num* SKIP_FILE_num%* ABORT_STATEMENT* BLANK OPTION|
|**Specify the number of errors that can be skipped**|-Required when On Error Behavior is either `SKIP_FILE_num` or `SKIP_FILE_num%`.Specify the number of errors that can be skipped.|
|**Size Limit**|-Number (> 0) that specifies the maximum size (in bytes) of data to be loaded for a given COPY statement|
|**Purge Behavior**|-Boolean that specifies whether to remove the data files from the stage automatically after the data is loaded successfully|
|**Return Failed Only**|-Boolean that specifies whether to return only files that have failed to load in the statement result|
|**Force**|-Boolean that specifies to load all files, regardless of whether they’ve been loaded previously and have not changed since they were loaded|
|**Load Uncertain Files**|-Boolean that specifies to load files for which the load status is unknown. The COPY command skips these files by default|
|**Enforce Length**|-Boolean that specifies whether to truncate text strings that exceed the target column length|
|**Truncate Columns**|-Boolean that specifies whether to truncate text strings that exceed the target column length|

### CopyInto - System Columns

The set of columns which has source data and file metadata information.

* **SRC** - The data from the file is loaded into this variant column.
* **LOAD_TIMESTAMP** - Current timestamp when the file gets loaded.
* **FILENAME** - Name of the staged data file the current row belongs to. Includes the full path to the data file.
* **FILE_ROW_NUMBER** - Row number for each record in the staged data file.
* **FILE_LAST_MODIFIED** - Last modified timestamp of the staged data file the current row belongs to
* **SCAN_TIME** - Start timestamp of operation for each record in the staged data file. Returned as TIMESTAMP_LTZ.

### CopyInto Deployment

#### CopyInto Deployment Parameters

The CopyInto node typeincludes an environment parameter that allows you to specify if you want to perform a full load or a reload based on the load type when you are performing a Copy-Into operation.

The parameter name is `loadType` and the default value is ``.

MDXSAFE_PLACEHOLDER_0_ENDPLACEHOLDER

When the parameter value is set to `Reload`, the data is reloaded into the table regardless of whether they’ve been loaded previously and have not changed since they were loaded.As a alternate option we can use TruncateBefore option in node config

#### CopyInto Initial Deployment
When deployed for the first time into an environment the Copy-into node of materialization type table will execute the below stage:

| Deployment Behavior  | Load Type | Stages Executed |
|--|--|--|
| Initial Deployment | ``|Create Table/Transient Table
| Initial Deployment | Reload|Create Table/Transient Table

### CopyInto Redeployment

#### Altering the CopyInto Tables

There are few column or table changes like Change in table name,Dropping existing column, Alter Column data type,Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column | Edit table description**: Alter table statement is executed to perform the alter operation.

#### Copy-Into change in materialization type

When the materialization type of Copy-Into node is changed from table to transient table or viceversa,the below stages are executed:

* **Drop table/transient table**
* **Create transient table/table**
* 
### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### CopyInto Undeployment

If the CopyInto node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop table/transient table**: Target table in Snowflake is dropped

Snowpipe 

The Coalesce Snowpipe node is a node that performs two operations. It can be used to load historical data using     
 [CopyInto](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table). Also the Snowpipe node can be used to create a pipe to auto ingest files from AWS, GCP, or Azure.

[Snowpipe](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro) enables loading data from files as soon as they’re available in a stage. 

This means you can load data from files in micro-batches, making it available to users within minutes, rather than manually executing COPY statements on a schedule to load larger batches.

### Key points to use Snowpipe Node

* Snowpipe node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* Snowpipe node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to Inferschema to know more on how to use the node and add Copy-Into on top of it.
* The data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config or reload parameter
*The path or subfolder name inside stage where the file is located can be specified using config 'Path or subfolder'.Do not prefix or suffix '/' in path name.Example,one level 'SUBFOLDER',two levels 'SUBFOLDER/INNERFOLDER'.

### Use Snowpipe node with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema
* This is mainly a test to make sure create will work
* Hit Run button to execute DML

### Snowpipe Node Configuration

The Snowpipe node type the following configurations available:

* [Node Properties](#snowpipe-node-properties)
* [General Options](#snowpipe-general-options)
* [Snowpipe Options](#snowpipe-snowpipe-options)
* [File Location](#snowpipe-file-location)
* [File Format](#snowpipe-file-format)
* [Extended CSV File Format Options](#snowpipe-extended-CSV-file-format-options)
* [Copy Options](#snowpipe-copy-options)

**InferSchema-true**
![image](https://github.com/user-attachments/assets/fec7326a-71aa-49b4-a71c-f91174f47777)

**InferSchema-false**
![image](https://github.com/user-attachments/assets/8b2ed94f-9497-4bc3-82f9-ce883205536d)

Snowpipe Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 General Options 

| **Setting** | **Description** |
|-------------|-----------------|
| **Create As** | Dropdown that helps us to create a table or Transient table to load data from external stage.  * Table * Transient Table|
|**InferSchema**|True / False toggle that determines whether or not to infer the columns of file before loading * True -The node is created with the inferred columns* False -No infer table step is executed|
| **DDL Case Insensitive**| True/False toggle that determines whether to preserve the case of columns from source file - **True**: The create and data load preserves the case of columns- **False**: The columns used in create and data load are case insensitive[How to use the toggle](#use-copyinto-to-load-columns-with-insensitive-case) |

 Snowpipe Options 

![snowpipe-snowpipe-options](https://github.com/user-attachments/assets/7e341bb3-4a36-4d4a-b451-5cde6805dad6)

| **Setting** | **Description** |
|-------------|-----------------|
|**Enable Snowpipe**| Drop down that helps us to create a pipe to auto ingest files from external stage or validate the Copy-Into statement.* Enable Snowpipe - Load data auto ingesting files from AWS, Azure, or GCP. Choose your**SNS service** Toggle to check if we need to use Notification Service.If false,default Snowflake notification is used**Cloud Provider.** * AWS - AWS SNS Topic. Specifies the Amazon Resource Name (ARN) for the SNS topic for your S3 bucket.* Azure - Integration. Specifies the existing notification integration used to access the storage queue.* GCP -  Integration. Specifies the existing notification integration used to access the storage queue.* Test Copy Statement - To validate the Copy-into statement before we use it to create PIPE|
|**Load historical data**|Loads the historic data into the target table by executing a COPY_INTO statement|

 Snowpipe File Location 

**InferSchema-true**
![image](https://github.com/user-attachments/assets/ae16545e-4a01-47f6-bf1b-66f3385f21d1)

**InferSchema-false**
![snowpipe-file-location](https://github.com/user-attachments/assets/70189562-bb34-49ba-9735-db63f34b271c)

| **Setting** | **Description** |
|---------|-------------|
| **Coalesce Storage Location of Stage** | A storage location in Coalesce where the stage is located.|
| **Stage Name (Required)** | Internal or External stage where the files containing data to be loaded are staged|
| **File Name(s)(Ex:'a.csv','b.csv')** | Enabled when 'Enable Snowpipe' under Snowpipe Options is toggled off. Specifies a list of one or more files names (separated by commas) to be loaded. For example, `'a.csv','b.csv'`|
| **Path or subfolder** | Not mandatory.Specifies the path or subfolders inside the stage where the file is located.Ensure that '/' is not pre-fixed before or after the subfolder name|
| **File Pattern (Ex:'.*hea.*[.]csv')**| A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match. For example, `*hea.*[.]csv'`|
  
 Snowpipe File Format 

**File format definition-File format name**

![image](https://github.com/user-attachments/assets/acb50407-40ff-4bd1-aeb5-e855b9b4da3f)

**File format definition-File format values**

![Snowpipe-file-format](https://github.com/user-attachments/assets/ae640901-16ce-4ed2-8e6c-efae278d3942)

##### File format config-Inferschema

* It is not mandatory to create a file format in snowflake to infer the structure of the file
* If the file format definition is set to 'File format values',a temporary file format is created based on the config values and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and is of file type csv,a temporary file format is created to adapt the same to infer the structure and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and the file types except csv,the same file format is used for inferring.No temporary file formts are created.
* The temporary file format created follows the naing convention ==>`TEMP_&#x7B;&#x7B;file_type&#x7D;&#x7D;_&#x7B;&#x7B;node name&#x7D;&#x7D;`.Ensure that any predefined file formats created in snowflake does not have the same file format name.
* Blank options have been added for few config options if u want to exclude any file format option

##### File Format Definition - File Format Name

| **Setting** | **Description** |
|---------|-------------|
|* **File Format Name**|Specifies an existing named file format to use for loading data into the table|
|* **File Type**| CSV  JSON  ORC  AVRO  PARQUET  XML |

##### File Format Definition - File Format Values

| **Setting** | **Description** |
|---------|-------------|
|**File Format Values**|Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Record delimiter**-Characters that separate records in an input file.Octal notation preferred.**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Field optionally enclosed by**- Character used to enclose strings**Number of header lines to skip**- Number of lines at the start of the file to skip**Parse Header(InferSchema-true)**-Boolean that specifies whether to use the first row headers in the data files to determine column names. **Trim Space**- Boolean that specifies whether to remove white space from fields  **Replace invalid characters**- Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Encoding**- Specifies the character set of the source data when loading data into a table **Date format**- String that defines the format of date values in the data files to be loaded. **Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded.[Refer to extended options here](#snowpipe-extended-CSV-file-format-options)|
|**JSON**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Trim Space** - Boolean that specifies whether to remove white space from fields**Strip Outer Array**- Boolean that instructs the JSON parser to remove outer brackets [ ].**Date format**- String that defines the format of date values in the data files to be loaded**Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded|
|**ORC**|**Trim Space** - Specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**AVRO**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
**PARQUET**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character**Using vectorised scanner** - Boolean that specifies whether to use a vectorized scanner for loading Parquet files|
|**XML**|**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|

 Snowpipe Extended CSV File Format Options 

| **Setting** | **Description** |
|---------|-------------|
|**CSV**|**Skip blank lines**- Boolean that specifies to skip any blank lines encountered in the data files **Nulls for empty field**- When loading data, specifies whether to insert SQL NULL for empty fields in an input file**Skip byte order mark**-Boolean that specifies whether to skip the BOM (byte order mark), if present in a data file.**Parsing error for column mismatch**-Boolean that specifies whether to generate a parsing error if the number of delimited columns (i.e. fields) in an input file does not match the number of columns in the corresponding table.**Multiple lines**-Boolean that specifies whether multiple lines are allowed.If MULTI_LINE is set to FALSE and the specified record delimiter is present within a CSV field, the record containing the field will be interpreted as an error.**Binary format**-Defines the encoding format for binary input or output.  **Replace values with NULL**-o specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. **Escape optionally enclosed by**-A singlebyte character string used as the escape character for unenclosed field values only**Escape enclosed/unenclosed values**-A singlebyte character string used as the escape character for enclosed or unenclosed field values.|

 Snowpipe Copy Options 

| **Setting** | **Description** |
|---------|-------------|
|**On Error Behavior**|String (constant) that specifies the error handling for the load operation* CONTINUE* SKIP_FILE* SKIP_FILE_num * Specify the number of errors that can be skipped.* SKIP_FILE_num%* Specify the number of errors that can be skipped* BLANK OPTION|
|**Enforce Length**|Boolean that specifies whether to truncate text strings that exceed the target column length|
|**Truncate Columns**|Boolean that specifies whether to truncate text strings that exceed the target column length|

### Snowpipe System Columns

The set of columns which has source data and file metadata information.

* **SRC** - The data from the file is loaded into this variant column.
* **LOAD_TIMESTAMP** - Current timestamp when the file gets loaded.
* **FILENAME** - Name of the staged data file the current row belongs to. Includes the full path to the data file.
* **FILE_ROW_NUMBER** - Row number for each record in the staged data file.
* **FILE_LAST_MODIFIED** - Last modified timestamp of the staged data file the current row belongs to
* **SCAN_TIME** - Start timestamp of operation for each record in the staged data file. Returned as TIMESTAMP_LTZ.

### Key points to use Snowpipe node
* Snowpipe node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* Snowpipe node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to Inferschema to know more on how to use the node and add Snowpipe node on top of it.

### Use Snowpipe node with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema and pipe to ingest data
  
![image](https://github.com/user-attachments/assets/7b8a2161-6457-483b-9c66-1e27af0c72f3)

If the above works, it should be deployable as is.  Deploy will simply take the columns and execute a create table and snowpipe

### Snowpipe Deployment

#### Snowpipe Deployment Parameters

The Snowpipe includes twp environment parameters.The parameter `loadType` allows you to specify if you want to perform a full load or a reload based on the load type 
when you are performing a Copy-Into operation.
The another parameter `targetIntegration` allows you to specify the Integration name to be used for auto ingesting files from Clous providers

The parameter name is `loadType` and the default value is ``.

MDXSAFE_PLACEHOLDER_1_ENDPLACEHOLDER

When the parameter value is set to `Reload`, the data is reloaded into the table regardless of whether they’ve been loaded previously 
and have not changed since they were loaded.

The parameter name is targetIntegration and the default value is DEV ENVIRONMENT.

When set to DEV ENVIRONMENT the value entered in the Snowpipe Options config Integration is used for ingesting files from Azure/GCP Cloud providers.

MDXSAFE_PLACEHOLDER_2_ENDPLACEHOLDER

When set to any value other than DEV ENVIRONMENT the node will use the specified value for Integration.

For example, the Snowpipe node will use the specific value for auto ingestion.

MDXSAFE_PLACEHOLDER_3_ENDPLACEHOLDER
The parameter name is targetAWSSNSTopic and the default value is DEV ENVIRONMENT.

When set to DEV ENVIRONMENT the value entered in the Snowpipe Options config AWS SNS Topic is used for ingesting files from AWS Cloud providers where a customer prefers SNS service.

MDXSAFE_PLACEHOLDER_4_ENDPLACEHOLDER

When set to any value other than DEV ENVIRONMENT the node will use the specified value for Integration.

For example, the Snowpipe node will use the specific value for auto ingestion.

MDXSAFE_PLACEHOLDER_5_ENDPLACEHOLDER

#### Snowpipe Initial Deployment

When deployed for the first time into an environment the Snowpipe node will execute the below stages depending on if Enable Snowpipe is enabled,'Load Historical Data' is enabled and the `loadType` parameter.

| Deployment Behavior  | Enable Snowpipe | Historical Load | Load Type | Stages Executed |
|--|--|---|--|--|
| Initial Deployment | Enable Snowpipe |true| ``|Create Table  Historical full load using CopyInto  Create Pipe  Alter Pipe 
| Initial Deployment | Enable Snowpipe |true| Reload|Create table  Truncate Target table  Historical full load using CopyInto  Create Pipe  Alter Pipe
| Initial Deployment | Enable Snowpipe |false| Reload or Empty|Create table  Truncate Target table  Create Pipe
| Initial Deployment | Test Copy Statement |false| Reload or Empty|Create table  Test Copy Statement-No pipe creation

### Snowpipe Redeployment

#### Altering the Snowpipe node

There are few column or table changes like Change in table name, Dropping existing column,  Alter Column data type, Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the target Table in the target environment.Any table level changes or node config changes results in recreation of pipe

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column| Edit table description |**: Alter table statement is executed to perform the alter operation.
* **Truncate target table**:The target table is truncated in case the Load Type parameter is set to 'Reload'
* **Historical full load using CopyInto**:Historical data are loaded if 'Load Historical' toggle is on.
* **Create Pipe**: Pipe is recreated if enable snowpipe option is true
* **Alter Pipe**:Pipe is refreshed if 'Load Historical' toggle is on.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Snowpipe Undeployment

If the Snowpipe node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

This is executed in two stages:

* **Drop Table**: Target table is dropped
* **Drop Pipe**: Pipe is dropped

## External Tables

The Coalesce External Table nodes create a new external table in the current/specified schema or replaces an existing external table.

An [external table](https://docs.snowflake.com/en/sql-reference/sql/create-external-table#examples) reads data from a set of one or more files in a specified external stage which can point to AWS,GCP or Azure cloud providers.

### External Tables Node Configuration

The External table node type has four configuration groups:

* [Node Properties](#external-tables-node-properties)
* [General Operations](#external-tables-general-options)
* [File Location](#external-tables-file-location)
* [File Format](#external-tables-file-format)
* [Additional Options](#external-tables-additional-options)

 External Tables Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 General Options 

| **Settings** | **Description** |
|--------------|-----------------|
|**InferSchema**|True / False toggle that determines whether or not to infer the columns of file before loading * True -The node is created with the inferred columns* False -No infer table step is executed|
| **DDL Case Insensitive**| True/False toggle that determines whether to preserve the case of columns from source file - **True**: The create and data load preserves the case of columns- **False**: The columns used in create and data load are case insensitive.[How to use the toggle](#use-copyinto-to-load-columns-with-insensitive-case) |
|**Partition by toggle**|When set to true,we get to choose partition column from dropdown.A partition column must evaluate as an expression that parses the path and/or filename information in the METADATA$FILENAME.By default the partitioning is done by FILENAME column|
|**Partition Column dropdown(Partition by toggle true)**|Get to choose the partition column with appropriate trnasformation from dropdown |

 External Tables Node File Location 

**InferSchema-true**

![image](https://github.com/user-attachments/assets/e3d02f1f-55f4-4c79-9f8c-2f8776874e6d)

**InferSchema-false**

![Filelocation](https://github.com/prj2001-udn/External-Data-Package/assets/169126315/48b7a0dd-301c-4077-857c-e37fd234bbf8)

| **Settings** | **Description** |
|--------------|-----------------|
|**Coalesce Stage Storage Location of Stage(Required)**|A storage location in Coalesce where the stage is located|
|**Stage Name (Required)**|Internal or External stage where the files containing data to be loaded are staged|
| **File Name(s)(Ex:'a.csv','b.csv')** | Enabled when InferSchema toggle is true. Specifies a list of one or more files names (separated by commas) to be loaded. For example, `'a.csv','b.csv'`|
|**File Pattern**|A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match. For example, `*hea.*[.]csv'`|

External Tables  File Format

**InferSchema-true**

![image](https://github.com/user-attachments/assets/14fac636-3968-4ae7-9cdf-6ba0bcb079c0)

**InferSchema-false**

![fileformat](https://github.com/prj2001-udn/External-Data-Package/assets/169126315/82945bb5-ba5d-46dd-b66d-3d6c10ff77d9)

##### File Format Definition - File Format Name
  
| **Settings** | **Description** |
|--------------|-----------------|
|**Coalesce Storage Location of File Format**|Location in Coalesce pointing to the database and schema,the file format resides.Mandatory when File Format Name is chosen|
|**File Format Name**|Specifies an existing named file format to use for loading data into the table|
|**File Type**|* CSV* JSON* ORC* AVRO* PARQUET|

##### File Format Definition - File Format Values

| **Settings** | **Description** |
|--------------|-----------------|
|**File Format Values**|-Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Record delimiter**-Characters that separate records in an input file**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Field optionally enclosed by**- Character used to enclose strings**Number of header lines to skip**- Number of lines at the start of the file to skip **Skip blank lines**- Boolean that specifies to skip any blank lines encountered in the data files **Trim Space**- Boolean that specifies whether to remove white space from fields  **Replace invalid characters**- Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Date format**- String that defines the format of date values in the data files to be loaded. **Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**-String that defines the format of timestamp values in the data files to be loaded|
|**JSON**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Trim Space** - Boolean that specifies whether to remove white space from fields**Strip Outer Array**- Boolean that instructs the JSON parser to remove outer brackets [ ].**Date format**- String that defines the format of date values in the data files to be loaded**Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded|
|**ORC**|**Trim Space** - Specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**AVRO**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
|**PARQUET**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
|**XML**|**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|

 External Tables Additional Options

| **Settings** | **Description** |
|--------------|-----------------|
|**Auto refresh**|-True/False toggle that specifies whether Snowflake should enable triggering automatic refreshes of the external table metadata when new or updated data files are available in the named external stage * False - Auto refresh does not take place* True - Auto refresh takes place. Setting this to true, give you the option to choose a Cloud Provider from AWS, Azure, and GCP. * AWS - AWS SNS Topic, Enabled only when Cloud Provider is AWS and auto refresh is true.Required only when configuring AUTO_REFRESH for Amazon S3 stages using Amazon Simple Notification Service (SNS). * GCP - Integration. Enabled when Cloud Provider is GCP/Azure. Specifies the existing notification integration used to access the storage queue * Azure - Integration. Enabled when Cloud Provider is GCP/Azure. Specifies the existing notification integration used to access the storage queue|

### System Columns
The set of columns which has source data and file metadata information.

* **VALUE** - The data from the file is loaded into this variant column
* **FILENAME** - Name of the staged data file the current row belongs to. Includes the full path to the data file.
* **PARTITION_COLUMN** - A column with default transformation split_part(metadata$filename,'/',2) is added to partition external table.

### Key points to use External table node
* External table node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* External node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to Inferschema to know more on how to use the node and add External node on top of it.
* A column with default transformation split_part(metadata$filename,'/',2) is added to partition external table.The user can rename the column and modify the transformation as per their requirement.Also,we can use any other column with appropriate transformation for partitioning external table

### Use External table with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema and pipe to ingest data

![image](https://github.com/user-attachments/assets/b04d2d66-eaa6-4f06-8282-e3b6bd4cac09)

If the above works, it should be deployable as is.  Deploy will simply take the columns and execute a create table and snowpipe

### External table Deployment

### Initial Deployment

When deployed for the first time into an environment the External table node will execute the below stage:

* **Create External Table** - This will execute a CREATE OR REPLACE statement and create a table in the target environment.

### Redeployment

#### Recreating the External table

The subsequent deployment of External table with changes in config options will recreate the table.

The following stages are executed:

* **Create External Table**
  
### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Undeployment

If the External table node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop External Table**

 InferSchema 

The Coalesce InferSchema UDN is a versatile node infers schema of the file in internal or external stage and dynamically creates the target table of the same name as inferschema node.

[InferSchema](https://docs.snowflake.com/en/sql-reference/functions/infer_schema) in Snowflake automatically detects the file metadata schema in a set of staged data files that contain semi-structured data and retrieves the column definitions.

### Key points InferSchema 

* A sample file in the internal or external stage.
* An existing fileformat to parse the file
* The file format is also expected to be in the same location as stage
* The mapping grid after creating the inferred node can be Re-Synced with the exact structure of the table/transient table using the Re-Sync Columns button in the mapping grid

### InferSchema Node Configuration

The InferSchema node type has two configuration groups:

* [InferSchema Node Properties](#inferschema-node-properties)
* [InferSchema Source Data](#inferschema-source-data)
* [InferSchema File format Options](#inferschema-file-format-options)
  
Go to the node and select the **Config tab** to see the Node Properties, Dynamic Table Options and General Options.

 InferSchema Node Properties 

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |
  
 InferSchema Source Data 

| **Settings** | **Description** |
|-------------|-----------------|
|**Create As**|Select from the options to create as Table or Transient Table* Transient Table* Table|
|**Coalesce Storage Location of Stage(Required)**|A storage location in Coalesce where the stage is located|
|**Stage Name (Required)**|Internal or External stage where the files containing data to be loaded are staged|
|**File Names (Required)**|Use commas to seperate multiple files. File whose metatdata is to be inferred|
|**Coalesce Storage Location of File Format**|Location in Coalesce pointing to the database and schema,the file format resides|
|**File Format Name (Required)**|Name of the file format object that describes the data contained in the staged files.It is expected in the same location as Stage|
|**Redeployment Behavior**|**CREATE OR REPLACE**: Dynamically creates target table based on the inferred schema from file in staging area**ALTER EXISTING TABLE**: Dynamically alters inferred table by comparing the inferred schema of the same file (with changes if any)and created table**DROP EXISTING TABLE**: Drops the table inferred|

 InferSchema File Format Options 

##### File format config-Inferschema

* It is not mandatory to create a file format in snowflake to infer the structure of the file
* If the file format definition is set to 'File format values',a temporary file format is created based on the config values and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and is of file type csv,a temporary file format is created to adapt the same to infer the structure and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and the file types except csv,the same file format is used for inferring.No temporary file formts are created.
* The temporary file format created follows the naing convention ==>`TEMP_&#x7B;&#x7B;file_type&#x7D;&#x7D;_&#x7B;&#x7B;node name&#x7D;&#x7D;`.Ensure that any predefined file formats created in snowflake does not have the same file format name.

##### File Format Definition - File Format Name
  
| **Settings** | **Description** |
|--------------|-----------------|
|**Coalesce Storage Location of File Format**|Location in Coalesce pointing to the database and schema,the file format resides.Mandatory when File Format Name is chosen|
|**File Format Name**|Specifies an existing named file format to use for loading data into the table|
|**File Type**|* CSV* JSON* ORC* AVRO* PARQUET|

##### File Format Definition - File Format Values

| **Settings** | **Description** |
|--------------|-----------------|
|**File Format Values**|-Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Record delimiter**-Characters that separate records in an input file**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Field optionally enclosed by**- Character used to enclose strings**Parse Header**-Boolean that specifies whether to use the first row headers in the data files to determine column names **Encoding**- Specifies the character set of the source data when loading data into a table [Refer this link for extended csv format options](#infer-extended-CSV-file-format-options) |
|**JSON**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Trim Space** - Boolean that specifies whether to remove white space from fields**Strip Outer Array**- Boolean that instructs the JSON parser to remove outer brackets [ ].|
|**ORC**|**Trim Space** - Specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**AVRO**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
|**PARQUET**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
|**XML**|**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|

 InferSchema Extended CSV File Format Options 

| **Setting** | **Description** |
|---------|-------------|
|**CSV**|**Skip blank lines**- Boolean that specifies to skip any blank lines encountered in the data files  **Trim Space**- Boolean that specifies whether to remove white space from fields  **Replace invalid characters**- Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Nulls for empty field**- When loading data, specifies whether to insert SQL NULL for empty fields in an input file**Skip byte order mark**-Boolean that specifies whether to skip the BOM (byte order mark), if present in a data file.**Parsing error for column mismatch**-Boolean that specifies whether to generate a parsing error if the number of delimited columns (i.e. fields) in an input file does not match the number of columns in the corresponding table.**Multiple lines**-Boolean that specifies whether multiple lines are allowed.If MULTI_LINE is set to FALSE and the specified record delimiter is present within a CSV field, the record containing the field will be interpreted as an error.**Binary format**-Defines the encoding format for binary input or output.  **Replace values with NULL**-o specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. **Escape optionally enclosed by**-A singlebyte character string used as the escape character for unenclosed field values only**Escape enclosed/unenclosed values**-A singlebyte character string used as the escape character for enclosed or unenclosed field values.|

### InferSchema Usage

#### Option1-Using API-NODEUPDATE
* Add a InferSchema node, for example `INFER_JSON` and hit create.'Infer and Create table' stage runs and creates a table with the same name as InferSchema node.
* Use [API-NODEUPDATE](#API-NODEUPDATE) node to update the columns of the infer node created in the above step.
* Now we can add a Copy-Into,Snowpipe or External table node on top of the inferred table to load staged files.
* Once we get the required metadata columns in our downstream node(Copy-Into,Snowpipe or External table),bulk edit the source in the node to be blank.Also delete the from clause in the Join tab.These information are not required for Copy-Into,Snowpipe or External table to  load data from files.
* Delete the API-NODEUPDATE node ,we created in step2 as we have updated the columns required for the downstream nodes and the data for the same are derived from files.
* Eventually API-NODEUPDATE nodes are not required to be deployed.It is sufficient to deploy the Copy-into,Snowpipe or External table nodes which holds the data from staged files

#### Option2-Using Re-Sync Columns button
* Add a InferSchema node, for example `INFER_JSON` and hit create.'Infer and Create table' stage runs and creates a table with the same name as InferSchema node.
* Use Re-Sync Columns button in the mapping grid to update the columns of the infer node created in the above step.
* Now we can add a Copy-Into,Snowpipe or External table node on top of the inferred table to load staged files.

  ![image](https://github.com/user-attachments/assets/e3f1c823-942e-49ef-9c56-5338f3973f97)

### InferSchema Deployment

#### InferSchema Initial Deployment

When deployed for the first time into an environment the InferSchema node will execute the stage:

* Stage executed: **Infer and Create target table**

#### InferSchema Redeployment

**Redeployment Behavior: Create or Replace**

| Redeployment Behavior | Stage Executed |
|---|---|
| Create or Replace | Infer and Create target table

If any of the Source Data options like Stage storage location, Stage name or filename are modified.
Then you can redeploy the Infer Schema node with redeployment behaviour “Create or Replace”.

> 📘 Info
> 
> You can go back to the browser and Re-sync columns of Inferred table, re-execute Copy-Into and redeploy

**Redeployment Behavior: Alter Existing Table**

| Redeployment Behavior | Stage Executed |
|---|---|
|Alter existing table| Infer and Alter target table

If all Source Data options remain same and only there are changes in the existing file structure, you can redeploy the Infer Schema node with redeployment behaviour “Alter existing table”.

**Redeployment Behavior: Drop Existing Table**

| Redeployment Behavior | Stage Executed |
|---|---|
|Drop existing table |Drop inferred table

If you want to drop the inferred table you can redeploy the Infer Schema node with redeployment behaviour “Drop existing table”.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed
  
### InferSchema Undeployment

If the InferSchema node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then no action takes place.

CopyUnload 

The Coalesce Copy Unload node unloads the table into internal stage or external stage location in various file formats supported by Snowflake

A [Copy Unload](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location#examples) node loads data from a table (or query) into one or more files in the internal or external location mentioned

### Copy Unload Node Configuration

The Copy Unload node type has four configuration groups:

* [Node Properties](#copy-unload-node-properties)
* [File Location](#copy-unload-file-location)
* [File Format](#copy-unload-file-format)
* [Copy Options](#copy-unload-copy-options)

 Copy Unload Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 Copy Unload Node File Location 

![copy-unload-file](https://github.com/coalesceio/External-Data-Package/assets/169126315/02698cc6-1d16-4e0c-b0fa-c8d614a6e2ed)

| **Setting** | **Description** |
|---------|-------------|
|**Coalesce Stage Storage Location of Stage(Required)**| A storage location in Coalesce where the stage is located|
|**Stage (Required)**| Location in Snowflake(internal stage) or external stage or external location where the data files are unloaded|
|**Partition by (Optional)**| Unload operation splits the table rows based on the partition expression and determines the number of files to create based on the number of unique values in a particular column (only sinlge column name expected)|
|**Allow Expressions in partition by clause**| A regular expression pattern string, enclosed in single quotes, for example: ('date=' || to_varchar(dt, 'YYYY-MM-DD') || '/hour=' || to_varchar(date_part(hour, ts))- Concatenates labels and column values to output meaningful filenames|

Copy Unload  File Format

![FileFormat](https://github.com/coalesceio/External-Data-Package/assets/169126315/53434835-c071-4132-b1f0-d0c41810216f)

##### File Format Definition - File Format Name

| **Setting** | **Description** |
|---------|-------------|
|**Coalesce Storage Location of File Format**| File format location in snowflake that is mapped as storage location in Coalesce|
|**File Format Name**|Specifies an existing named file format in Snowflake to use for unloading data into the table|
|**File Type**|* CSV  * JSON  * PARQUET|

##### File Format Definition - File Format Values

| **Setting** | **Description** |
|---------|-------------|
|**File Format Values**|-Provides different file format options for the File Type chosen to unload|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**: String (constant) that specifies the current compression algorithm for the data files to be unloaded**Record delimiter**: Characters that separate records in an unloaded file**Field delimiter**: One or more singlebyte or multibyte characters that separate fields in an unloaded file**Field optionally enclosed by**: Character used to enclose strings. (Default is \042)**File Extension**: String that specifies the extension for files unloaded to a stage. Accepts any extension**Date Format**: String that defines the format of date values in the unloaded data files.(Default is AUTO)**Time Format**: String that defines the format of time values in the unloaded data files.(Default is AUTO)**Timestamp Format**: String that defines the format of timestamp values in the unloaded data files.(Default is AUTO)|
|**JSON**|**Compression**: String (constant) that specifies the current compression algorithm for the data files to be unloaded**File extension** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**PARQUET**|**Compression**: String (constant) that specifies the current compression algorithm for the data files to be unloaded|

 Copy Unload Copy Options

![CopyOptions](https://github.com/coalesceio/External-Data-Package/assets/169126315/4a1d32ec-a898-40ba-93e7-94194bcae02d)

| **Setting** | **Description** |
|---------|-------------|
|**Overwrite Flag**|Toogle overwrites existing files with matching names, if any, in the location where files are unloaded|
|**Header**|Toggle to be enabled if the file created requires a header| 
|**Single File Flag**|Toggle generates a single file when true, else generates multiple files if partition by enabled|
|**Max File Size (MB)**|Number (> 0) that specifies the upper size limit (in bytes) of each file to be generated in parallel per thread. Note that the actual file size and number of files unloaded are determined by the total amount of data and number of nodes available for parallel processing|
|**Include Query ID**|When true provides unique identifier for unloaded files by including a universally unique identifier (UUID) in the filenames of unloaded data files| 
|**Detailed Output**|When true includes a row for each file unloaded to the specified stage. Columns show the path and name for each file, its size, and the number of rows that were unloaded to the file|

  ### Copy Unload Deployment

  Deployment not applicable for this node

API 

The Coalesce API Node loads external data which is external to snowflake with the help of EXTERNAL ACCESS INTEGRATION Name which has a network rule which allows access to external network locations using procedure handler

### API Node Configuration

The API Node Configuration type has two configuration groups:

* [Node Properties](#api-node-properties)
* [Options](#API-node-Options)

 API Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 API Node Options 

![Options]

| **Setting** | **Description** |
|---------|-------------|
|**Snowflake EXTERNAL ACCESS INTEGRATION Name (Required)**|EXTERNAL ACCESS INTEGRATION Name has a network rule which allows access to external network locations external to snowflake using procedure|
|**Method(Required)**|HTTP Request methods Get,Put,Post**Get**: API Call to retrieve data**Put**: API Call to update existing data**Post**: API Call to create new data|  
|**URI(Required)**|Uniform Resoure Identifier to be provided to locate and interact with resources within a specific API|
|**Bearer/Token/API-key Authorization**|True/False toggle.If enabled provide Snowflake secret for API token.[How to pass Credentials for API](#Credentials-API)|
|**HTTPheaders(Ex:Authorization,x-api-key)**|Type of HTTPheader supported by the application|
|**Snowlake secret for API token**|Enabled if Bearer Authorisation is chosen.Provide the snowflake secret created for API token|
|**ClientID/Secret Authorization**|True/False toggle.If enabled provide Snowflake secret for API Client ID and another snowflake secret for API password.[How to pass Credentials for API](#Credentials-API)|
|**Snowflake secret for API Client ID**|Provide the snowflake secret created for a specific Client ID|
|**Snowflake secret for API Client password**|Provide the snowflake secret created for password of specific Client ID|
|**Forcibly add security credentials in header**| It is a security threat to add tokens/credentials in headers.If you want to proceed aware of the risk enable 'Forcibly add security credentials in header|
|**Headers**|headers are used to provide additional context and information about the request|
|**Payload**|when making HTTP requests to a URI, the request may include a payload (also known as a body)|
|**Note:**|Payload applicable only to Put & Post API method calls|

Credentials-API
* Bearer/authorisation tokens need not be added to URL headers
* When using Bearer Authorisation,ensure that you prefix token with Bearer when you create Snowflake secret

* API-key authorisation keys need not be added to URL headers
  

* When using Client ID/Secret Authorisation,create secrets as below.

API-NODEUPDATE

### API-NODEUPDATE Node Configuration

The API-NODEUPDATE Node Configuration type has two configuration groups:

* [Node Properties](#api-nodeup-properties)
* [Options](#API-nodeup-Options)

 API-NODEUPDATE Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 API-NODEUPDATE Node Options 

![API-Options](https://github.com/user-attachments/assets/fc366fc0-bae2-4f16-aa89-dde5b701bb1c)

| **Setting** | **Description** |
|---------|-------------|
|**Snowflake EXTERNAL ACCESS INTEGRATION Name (Required)**| EXTERNAL ACCESS INTEGRATION Name has a network rule which allows access to external network locations external to snowflake using procedure|
|**Snowlake secret for Coalesce API token (Required)**|SNOWFLAKE SECRET to allow access to Coalesce API|
|**Workspace-Node details**|Information on the list of nodes for which columns need to be updated**Workspace ID**: This is the id of workspace where node belongs**Node name**: Name of the node whose columns needs to be updated**Storage Location (E.g DBName.SchemaName):**: Enter storage location of the table, with database name and schema|

JDBC LOAD

The Coalesce JDBC Load Node allows to connect, interact, and retrieve data from various database management systems into Snowflake

### JDBC Load Node Configuration

The JDBC Load Node Configuration type has two configuration groups:

* [Node Properties](#JDBC-load-node-properties)
* [Options](#JDBC-load-Options)

 JDBC Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 JDBC Load Options 

![JDBC_load_options](https://github.com/coalesceio/External-Data-Package/assets/169126315/5af13d5b-576a-44ce-ae5b-d02d7501cc7f)

| Name | Description|
|--|--|
|**JDBC Driver(Required)**|Required to establish the connection to external database|
|**JAR file external stage location(Required)**|Location of the external stage where JAR file is located|
|**JAR file name(Required)**|Name of JAR file located in external stage|
|**EXTERNAL ACCESS INTEGRATION(Required)**|EXTERNAL ACCESS INTEGRATION Name has a network rule which allows access to external network locations external to snowflake using procedure|
|**External database credentials**|Username, Password of the external Database in the form snowflake secret object|
|**JDBC URL(Required)**|URL of the database with port number (ex: "sqlserver-tests.database.windows.net:1433" )|
|**SQL(Required)**|SQL query to retrieve the required data|
|**Truncate Before**|When enabled,the table is truncated before data load| 

Parse Excel 

The Coalesce Parse Excel node parses an large excel file in stage location to json format on the target location

### Parse Excel Node Configuration

The Parse Excel node type has three configuration groups:

* [Node Properties](#parse-excel-node-properties)
* [Stored Procedure Options](#parse-excel-stored-procedure-options)
* [Excel File Processing Options](#parse-excel-file-processing-options)

 Parse Excel Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 Parse Excel Stored Procedure Options 

![StoredProcedureOptions](https://github.com/coalesceio/External-Data-Package/assets/169126315/ad7e21b6-1ea2-43c9-875a-5a1ce34c9828)

| **Setting** | **Description** |
|---------|-------------|
|**Stored Procedure Storage Location(Required)**|Location in Snowflake to be provided where Stored Procedure will be loaded by script|
|**Note**|Stored Procedure location and Storage Location (i.e. destination) must be in same Schema|

Excel File Processing Options

![ExcelFileProcessingOptions](https://github.com/coalesceio/External-Data-Package/assets/169126315/f612a1cb-f5fe-420f-9a38-d040f0151e6a)

| **Setting** | **Description** |
|---------|-------------|
|**Source File Storage Location(REQUIRED)**|Storage location name in Coalesce which points to the database and schema  where stage is located (source file has to be located under stages)|
|**Source File Stage(REQUIRED)**|Stage name in Snowflake where source file is located|
|**Source File Name(REQUIRED)**|Excel filename to be parsed|

### Initial Deployment

When deployed for the first time into an environment the Parse Excel node will execute the below stage:

* **Create OR Replace Table** - This will execute a CREATE OR REPLACE statement and create a table in the target environment.
* **Create OR Replace Stored Procedure** - This will create a Stored Procedure in the target environment 

### Redeployment

#### Recreating the Parse Excel

There are few column or table changes like Change in table name, Dropping existing column, Alter Column data type, Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the target Table in the target environment.Any table level changes or node config changes results in  recreation of Stored Procedure

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column| Edit table description** -  Alter table statement is executed to perform the alter operation.
* **Create OR Replace Stored Procedure** - This will create a Stored Procedure in the specified target environment  

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Undeployment

If the Parse Excel node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop Table**
* **Drop Procedure**

Parse Json 

The Coalesce parses large json file containing json array to the provided target location 

### Parse Json Node Configuration

The Parse Excel node type has three configuration groups:

* [Node Properties](#parse-json-node-properties)
* [UDTF Procedure Options](#parse-json-UDTF-procedure-options)
* [JSON File Processing Options](#parse-json-json-file-processing-options)

 Parse json Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 Parse JSON UDTF Procedure Options 

![UDTFProcedureOptions](https://github.com/coalesceio/External-Data-Package/assets/169126315/c319c9f4-6654-4c9b-9416-5a979279055f)

| **Setting** | **Description** |
|---------|-------------|
|**User Defined Table Function Location (Required)**|Location in Snowflake to be provided where UDTF will be loaded by script|
|**Note**| UDTF location and Storage Location (i.e. destination) must be in same Schema|

Parse JSON JSON File Processing Options

![JSONFileProcessingOptions](https://github.com/coalesceio/External-Data-Package/assets/169126315/b32ac841-6e29-4442-8474-8385cd9a0e97)

| **Setting** | **Description** |
|---------|-------------|
|**Source File Storage Location(REQUIRED)**|Schema name in Snowflake where source file is located (source file has to be located under stages)|
|**Source File Stage(REQUIRED)**|Stage name in Snowflake where source file is located|
|**Source File Name(REQUIRED)**|Json filename to be parsed|
|**JSON Array Name(REQUIRED)**|Array name of the json|

### Initial Deployment

When deployed for the first time into an environment the Parse Json node will execute the below stage:

* **Create OR Replace Table** - This will execute a CREATE OR REPLACE statement and create a table in the target environment.
* **Create OR Replace Function** - This will create a UDTF function in the target environment 

### Redeployment

#### Recreating the Parse Json

There are few column or table changes like Change in table name, Dropping existing column, Alter Column data type, Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the target Table in the target environment.Any table level changes or node config changes results in  recreation of Stored Procedure

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column| Edit table description** -  Alter table statement is executed to perform the alter operation.
* **Create OR Replace Stored Procedure** - This will create a Stored Procedure in the specified target environment  

 ### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Undeployment

If the Parse Json node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop Table**
* **Drop Procedure**

 GITSEED 

### Gitseed Node Configuration

The Gitseed node type the following configurations available:

* [Node Properties](#gitseed-node-properties)
* [General Options](#gitseed-general-options)
* [Git Repository Configuration](#gitrepo-file-location)
* [File Format](#copy-into-file-format)
* [Extended CSV File Format Options](#extended-csv-file-format-options)
* [Copy Options](#gitseed-copy-options)

Gitseed - Node Properties

There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the target table will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### Prerequisites

We require few setups done in Snowflake before we use the Gitseed node type

* Create Snowflake secret containing the credentials to use for authenticating with the remote Git repository.
* Create an API INTEGRATION that contains information about the remote Git repository such as allowed credentials and prefixes for target URLs.
* Using GIT Secret and GIT API INTEGRATION ,create a local clone of GIT Repo in Snowflake.
* Ideally the local clone of GIT repo in snowflake is refreshed with files or folders only after GIT fetch operation is performed.This can handled during node execution using GIT Fetch toggle in config or as a precursor in Snowflake

### Key points to use Gitseed Node

* Gitseed node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single variant column in target table.
* Gitseed node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to Inferschema to know more on how to use the node and add Gitseed on top of it.
* The data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config or reload parameter
* The path or subfolder name inside stage where the file is located can be specified using config 'Path or subfolder'.Do not prefix or suffix '/' in path name.Example,one level 'SUBFOLDER',two levels 'SUBFOLDER/INNERFOLDER'.
* Blank options added to Copy-Into and CSV file format options to revert any specific option

### Use Gitseed to load columns with insensitive case
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button with DDLCaseInsensitive toggle ON to execute create table based on inferred schema
* Table is created with column names in default case followed by Snowflake(UPPERCASE)
* Re-Sync columns to have all the columns with default case in mapping grid,so that the nodes added upstream follow the same.

### Use Gitseed node with InferSchema option
* Set Infer Schema toggle to true
* Hit Create button to Infer Schema
* To choose the file format configs,[refer link](#file-format-config-inferschema)
* Click on Re-Sync Columns button
* If all looks good, set Infer Schema button to false
* Hit Create button to execute create table based on inferred schema
* This is mainly a test to make sure create will work
* Hit Run button to execute DML

![Resync](https://github.com/user-attachments/assets/de3c4ce2-370e-43d0-a010-fc6131cd8669)

If the above works, it should be deployable as is.  Deploy will simply take the columns and execute a create table.

  Gitseed - General Options 

**InferSchema-True**

![CopyInto](https://github.com/user-attachments/assets/a537faac-bb91-4a00-b055-4612934d95b9)

**InferSchema-False**

| **Option** | **Description** |
|------------|----------------|
| **Create As** | Select from the options to create as Table or Transient Table- **Transient table**  -**Table** |
| **TruncateBefore(Disabled when Inferschema is true)** | True / False toggle that determines whether or not a table is to be truncated before reloading - **True**: Table is truncated and Copy-Into statement is executed to reload the data into target table- **False**: Data is loaded directly into target table and no truncate action takes place. |
| **InferSchema** | True / False toggle that determines whether or not to infer the columns of file before loading - **True**: The node is created with the inferred columns- **False**: No infer table step is executed |
| **DDL Case Insensitive**| True/False toggle that determines whether to preserve the case of columns from source file - **True**: The create and data load preserves the case of columns- **False**: The columns used in create and data load are case insensitive[How to use the toggle](#use-copyinto-to-load-columns-with-insensitive-case) |
      
  Git Repository Configuration 

**InferSchema-true**

**InferSchema-false**

| **Setting** | **Description** |
|-------------|-----------------|
|**GIT Fetch**| If enabled,an alter statement to fetch files from GIT repository is executed.Ideally the local clone of GIT repo in snowflake is refreshed with files or folders only after GIT fetch operation is performed|
| **Coalesce Storage Location of Snowflake GIT repository** | A storage location in Coalesce where the Snowflake GIT repo resides.[Pre-requisites for GIT repo setup in Snowflake](#Prerequisites)|
| **Snowflake GIT repository (Required)**| Local Clone of the remote GIT repo in Snowflake where the files containing data to be loaded are staged|
| **GIT Branch (Ex:main) (Required)**| Specifies the GIT branch where the file is located.Ensure that '/' is not pre-fixed before or after the subfolder name|
| **GIT Folder / Path (Ex:seeds)**| Specifies the GIT folder inside the branch where the file is located.Ensure that '/' is not pre-fixed before or after the subfolder name|
| **File Name(s)(Ex:a.csv,b.csv)** | Specifies a list of one or more files names (separated by commas) to be loaded |
| **File Pattern (Ex:'.*hea.*[.]csv')**| A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match |

 Gitseed - File Format 

**File format definition-File format name**

![Copy-Into format](https://github.com/user-attachments/assets/9e4e8f7e-29be-4209-a08b-cb19fadccf05)

**File format definiton-File format values**

![copy-into-file-format](https://github.com/user-attachments/assets/2a737c5f-3bb3-471a-9365-bf3251677415)

##### File format config-Inferschema

* It is not mandatory to create a file format in snowflake to infer the structure of the file
* If the file format definition is set to 'File format values',a temporary file format is created based on the config values and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and is of file type csv,a temporary file format is created to adapt the same to infer the structure and dropped on successfully inferring the structure of the table
* If the file format definition is set to 'File format name' and the file types except csv,the same file format is used for inferring.No temporary file formts are created.
* The temporary file format created follows the naing convention ==>`TEMP_&#x7B;&#x7B;file_type&#x7D;&#x7D;_&#x7B;&#x7B;node name&#x7D;&#x7D;`.Ensure that any predefined file formats created in snowflake does not have the same file format name.
  
##### File Format Definition - File Format Name

| **Setting** | **Description** |
|---------|-------------|
|* **File Format Name**|Specifies an existing named file format to use for loading data into the table|
|* **File Type**| CSV  JSON  ORC  AVRO  PARQUET  XML |

##### File Format Definition - File Format Values

| **Setting** | **Description** |
|---------|-------------|
|**File Format Values**|Provides file format options for the File Type chosen|
|**File Type**|Each file type has different configurations available|
|**CSV**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Record delimiter**-Characters that separate records in an input file**Field delimiter**- One or more singlebyte or multibyte characters that separate fields in an input file**Field optionally enclosed by**- Character used to enclose strings(octal notation preferred)**Number of header lines to skip**- Number of lines at the start of the file to skip**Parse Header(InferSchema-true)**-Boolean that specifies whether to use the first row headers in the data files to determine column names.**Trim Space**- Boolean that specifies whether to remove white space from fields  **Replace invalid characters**- Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.**Encoding**- Specifies the character set of the source data when loading data into a table **Date format**- String that defines the format of date values in the data files to be loaded. **Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded.[Refer to extended options here](#extended-CSV-file-format-options)|
|**JSON**|**Compression**- String (constant) that specifies the current compression algorithm for the data files to be loaded**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character **Trim Space** - Boolean that specifies whether to remove white space from fields**Strip Outer Array**- Boolean that instructs the JSON parser to remove outer brackets [ ] **Date format**- String that defines the format of date values in the data files to be loaded**Time format**- String that defines the format of time values in the data files to be loaded**Timestamp format**- String that defines the format of timestamp values in the data files to be loaded|
|**ORC**|**Trim Space** - Specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**AVRO**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|
|**PARQUET**|**Trim Space** - Boolean that specifies whether to remove white space from fields**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character**Using vectorised scanner** - Boolean that specifies whether to use a vectorized scanner for loading Parquet files|
|**XML**|**Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character|

 Extended CSV File Format Options 

| **Setting** | **Description** |
|---------|-------------|
|**CSV**|**Skip blank lines**- Boolean that specifies to skip any blank lines encountered in the data files  **Nulls for empty field**- When loading data, specifies whether to insert SQL NULL for empty fields in an input file**Skip byte order mark**-Boolean that specifies whether to skip the BOM (byte order mark), if present in a data file.**Parsing error for column mismatch**-Boolean that specifies whether to generate a parsing error if the number of delimited columns (i.e. fields) in an input file does not match the number of columns in the corresponding table.**Multiple lines**-Boolean that specifies whether multiple lines are allowed.If MULTI_LINE is set to FALSE and the specified record delimiter is present within a CSV field, the record containing the field will be interpreted as an error.**Binary format**-Defines the encoding format for binary input or output.  **Replace values with NULL**-o specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. **Escape optionally enclosed by**-A singlebyte character string used as the escape character for unenclosed field values only.Octal notation is preferred.**Escape enclosed/unenclosed values**-A singlebyte character string used as the escape character for enclosed or unenclosed field values.Octal notation is preferred|

 Gitseed - Copy Options 

| **Setting** | **Description** |
|---------|-------------|
|**On Error Behavior**|-String (constant) that specifies the error handling for the load operation|
|                     |* CONTINUE* SKIP_FILE* SKIP_FILE_num* SKIP_FILE_num%* ABORT_STATEMENT* BLANK OPTION|
|**Specify the number of errors that can be skipped**|-Required when On Error Behavior is either `SKIP_FILE_num` or `SKIP_FILE_num%`.Specify the number of errors that can be skipped.|
|**Size Limit**|-Number (> 0) that specifies the maximum size (in bytes) of data to be loaded for a given COPY statement|
|**Purge Behavior**|-Boolean that specifies whether to remove the data files from the stage automatically after the data is loaded successfully|
|**Return Failed Only**|-Boolean that specifies whether to return only files that have failed to load in the statement result|
|**Force**|-Boolean that specifies to load all files, regardless of whether they’ve been loaded previously and have not changed since they were loaded|
|**Load Uncertain Files**|-Boolean that specifies to load files for which the load status is unknown. The COPY command skips these files by default|
|**Enforce Length**|-Boolean that specifies whether to truncate text strings that exceed the target column length|
|**Truncate Columns**|-Boolean that specifies whether to truncate text strings that exceed the target column length|

### Gitseed - System Columns

The set of columns which has source data and file metadata information.

* **SRC** - The data from the file is loaded into this variant column.
* **LOAD_TIMESTAMP** - Current timestamp when the file gets loaded.
* **FILENAME** - Name of the staged data file the current row belongs to. Includes the full path to the data file.
* **FILE_ROW_NUMBER** - Row number for each record in the staged data file.
* **FILE_LAST_MODIFIED** - Last modified timestamp of the staged data file the current row belongs to
* **SCAN_TIME** - Start timestamp of operation for each record in the staged data file. Returned as TIMESTAMP_LTZ.

### Gitseed Deployment

#### Gitseed Deployment Parameters

The Gitseed node typeincludes an environment parameter that allows you to specify if you want to perform a full load or a reload based on the load type when you are performing a Copy-Into operation.

The parameter name is `loadType` and the default value is ``.

MDXSAFE_PLACEHOLDER_6_ENDPLACEHOLDER

When the parameter value is set to `Reload`, the data is reloaded into the table regardless of whether they’ve been loaded previously and have not changed since they were loaded.As a alternate option we can use TruncateBefore option in node config

#### Gitseed Initial Deployment
When deployed for the first time into an environment the Copy-into node of materialization type table will execute the below stage:

| Deployment Behavior  | Load Type | Stages Executed |
|--|--|--|
| Initial Deployment | ``|Create Table/Transient Table
| Initial Deployment | Reload|Create Table/Transient Table

### Gitseed Redeployment

#### Altering the Gitseed Tables

There are few column or table changes like Change in table name,Dropping existing column, Alter Column data type,Adding a new column if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

The following stages are executed:

* **Rename Table| Alter Column | Delete Column | Add Column | Edit table description**: Alter table statement is executed to perform the alter operation.

#### Gitseed change in materialization type

When the materialization type of Copy-Into node is changed from table to transient table or viceversa,the below stages are executed:

* **Drop table/transient table**
* **Create transient table/table**
   
### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Gitseed Undeployment

If the CopyInto node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

* **Drop table/transient table**: Target table in Snowflake is dropped
  
## Code

### InferSchema

* [Node definition](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Inferschema-299/definition.yml)
* [Create Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Inferschema-299/create.sql.j2)
* [Run Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Inferschema-299/run.sql.j2)

### CopyInto

* [Node definition](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/CopyInto-324/definition.yml)
* [Create Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/CopyInto-324/create.sql.j2)
* [Run Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/CopyInto-324/run.sql.j2)

### Snowpipe

* [Node definition](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Snowpipe-323/)
* [Create Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Snowpipe-323/)
* [Run Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/Snowpipe-323/)

### External Tables 

* [Node definition](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/ExternalTable-298/definition.yml)
* [Create Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/ExternalTable-298/create.sql.j2)
* [Run Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/ExternalTable-298/run.sql.j2)

### Copy Unload

* [Node definition](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/CopyUnload-302)
* [Create Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/CopyUnload-302)
* [Run Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/CopyUnload-302)

### API

* [Node definition](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/API-303)
* [Create Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/API-303)
* [Run Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/API-303)

### JDBC

* [Node definition](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/JDBCLoad-327)
* [Create Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/JDBCLoad-327)
* [Run Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/JDBCLoad-327)

### Parse Excel

* [Node definition](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseExcel-346)
* [Create Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseExcel-346)
* [Run Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseExcel-346)

### Parse Json

* [Node definition](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseJSON-347)
* [Create Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseJSON-347)
* [Run Template](https://github.com/coalesceio/External-Data-Package/tree/86f1d8f019493de644094a2139a8e1a3a0a510be/nodeTypes/ParseJSON-347)

### Gitseed

* [Node definition](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/GitSeed-606/definition.yml)
* [Create Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/GitSeed-606/create.sql.j2)
* [Run Template](https://github.com/coalesceio/External-Data-Package/blob/main/nodeTypes/GitSeed-606/run.sql.j2)

### Macros
 * [Macros/Node configs](https://github.com/coalesceio/External-Data-Package/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.1.3 | March 30, 2026 | Copy-Into,Snowpipe,Inferschema unenclosed_field fix |
| 2.1.2 | February 05, 2026 | API-key authentication support for API node type |
| 2.1.1 | January 13, 2026 | Git seed node type added |
| 2.0.5 | December 31, 2025 | Adding Snowflake Secrets support to API node for secure credential management |
| 2.0.3 | November 06, 2025 | Conditional STORAGE_INTEGRATION changes added in Copy Into Node Type |

---

## Functional Node Types(Package)

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/functional-node-types</code> <CopyButton textToCopy='@coalesce/snowflake/functional-node-types' />
Supported Platform: 
Latest Version: 4.3.3 (April 23, 2026)

calendar tabledate dimensionmatch_recognizepattern matchingpivotqualifyunpivotwindow functions

## Overview
A package of functional nodes specific to common types of transformations or data sets.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/functional-node-types</code> <CopyButton textToCopy='@coalesce/snowflake/functional-node-types' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Functional Node Types

The Coalesce Functional Node Types Package includes:

* [Date Dimension](#date-dimension)
* [Time Dimension](#Time-dimension)
* [Pivot](#Pivot)
* [Unpivot](#Unpivot)
* [Match Recognize](#match-recognize)
* [View-Qualify advanced deploy](#view-qualify-advanced-deploy)
* [Recursive CTE](#recursive-cte)
* [Code](#code)

---

## Functional node types - Brief Summary

This is a package of functional nodes designed for common types of data transformations. coalesce It includes eight node types:

### Dimension Tables

- **Date Dimension** — generates a calendar table with attributes like day, month, quarter, fiscal year, and holiday flags, configurable by start date and number of days to generate.
- **Time Dimension** — generates a time-of-day reference table with attributes like hour, minute, second, AM/PM, and business/peak hour flags, at configurable granularity (hour, minute, or second).

### Reshaping Data

- **Pivot** — converts row values into columns (e.g., turning month + sales rows into jan_sales, feb_sales columns), supporting single and multi-column pivots with dynamic structure inference.
- **Unpivot** — the reverse operation, rotating wide columns back into rows for a normalized structure.

Advanced SQL Patterns

- **Match Recognize** — leverages Snowflake's MATCH_RECOGNIZE clause to detect patterns in sequential data, useful for fraud detection, session tracking, and clickstream analysis.
- **View-Qualify (Advanced Deploy)** — creates Snowflake views with an optional QUALIFY filter on window functions, similar to how HAVING works for aggregates.
- **Recursive CTE** — supports hierarchical/self-referential queries (e.g., org charts, tree structures) using Snowflake's recursive common table expressions.

---

## Date Dimension

The Coalesce Date Dimension Table provides a comprehensive breakdown of date-related attributes, enabling efficient handling of date operations across various
use cases. The table typically includes columns such as day, month, year, day of the week, week of the year, quarter, and flags like day is weekday
or weekend. Additional columns like fiscal year, fiscal quarter, holiday indicators can also be included, depending on the requirements.

### Date Dimension Node Configuration

The Date Dimension node type has two configuration groups:

* [Node Properties](#date-dimension-node-properties)
* [Options](#date-options)
* [Additional options](#additional-options)

![Fact_config](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/45d22ea5-32ca-49f5-a464-b266cb29b516)

#### Date Dimension Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

##### Date Options

| **Setting** | **Description** |
|---------|-------------|
| **Starting Date**| A date from where the date values should be added in the date table.Default is :DATEADD(DAY, -730, CURRENT_DATE)|
| **Number of Days To Generate ** | Numeric value indicating how many days' records should be generated from the Starting Date. |
| **Generated Date Column Name** |Metadata column name used in the SQL generated for inserting records into the table. |

#### Additional Options

You can create the node as:

* [Table](#date-table-create-as-table)
* [View](#date-table-create-as-view)
* [Transient Table](#date-table-create-as-transient-table)

##### Date Dimension Create as Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Populate on deploy**| Toggle:True/FalseWhen enabled data is populated along with table creation during deployment.[More info on Populate on Deploy](#populate-on-deploy)|
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention if enabled |
| **Business key** | Required column for Type 1 Dimensions |
| **Default String Value** | If Insert Zero Key Record toggle is True then add a default value for columns with datatype string |
| **Default Surrogate Key Value** | If Insert Zero Key Record toggle is True then add a default value for surrogate key column|
| **Default Date Value (Date Format DD-MM-YYYY)** | If Insert Zero Key Record toggle is True then add a default value for date key column in the format DD-MM-YYYY|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Date Dimension Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Override Create SQL** | Toggle: True/False**True**: View is created by overriding the SQL**False**: Nodetype defined create view SQL will execute |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Date Dimension Create as Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Transient Table|
| **Truncate Before** | Toggle: True/FalseThis determines whether a table will be overwritten each time a task executes. **True**: Uses INSERT OVERWRITE**False**: Uses INSERT to append data |
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention if enabled |
| **Business key** | Required column for Type 1 Dimensions |
| **Default String Value** | If Insert Zero Key Record toggle is True then add a default value for columns with datatype string |
| **Default Surrogate Key Value** | If Insert Zero Key Record toggle is True then add a default value for surrogate key column|
| **Default Date Value (Date Format DD-MM-YYYY)** | If Insert Zero Key Record toggle is True then add a default value for date key column in the format DD-MM-YYYY|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Date Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

![work_join](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/7acff10b-8845-4c44-851a-b7a0bc7eaf41)

> 📘 **Specify Group by and Order by Clauses**
>
> Best Practice is to specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

### Date Dimension Deployment

#### Date Dimension Initial Deployment

When deployed for the first time into an environment the Date node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Date Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Date View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Date Dimension Redeployment

After the Date node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Date Table or recreating the Date table.

#### Altering the Date Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in business keys
2. Changes in join clauses
3. Transformations made at column level
4. Changing DML options like Truncate, Insert Zero key

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Truncate \| Insert Zero Key \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Date Dimension Recreating the Views

The subsequent deployment of Date node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with updated definition |

#### Date Dimension Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Date table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Date view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Date table/transient table |

> 📘 **Materialization Date Dimension**
>
> When the materialization type of Date node is changed from table/transient table to View and use Override Create SQL for view creation. This ensures that the following change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

![CreateSQL](https://github.com/coalesceio/Coalesce-Base-Node-Types---Advanced-Deploy/assets/7216836/0296abf8-0747-4ae8-8478-0782e5e2e545)

### Populate on Deploy
#### Initial deployment

|**Populate ondeploy**| **Insert Zero Key** |**Stages Executed**|
|---------------------|---------------------|-------------------|
|True                 | False               |Create table/transient table Insert data on deploy|
|True                 | True                |Create table/transient table Insert Zero key records Insert data on deploy|

#### Redeployment
The stages Insert Zero key and Insert data on deploy are executed if there are changes in transformation,data type,nullability,zero key changes and default values,column addition,date related config values.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Date Dimension Deploy Undeployment

If a Date Dimension Node of materialization type table/view/transient table are deleted from a Datespace, that Datespace is committed to Git and that commit deployed to a higher level environment then the DateTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Time Dimension

The Coalesce Time Dimension Table provides a comprehensive breakdown of Time-related attributes, enabling efficient handling of Time operations across various
use cases. The table typically includes columns such as TIME_KEY, HOUR_24, HOUR_12, MINUTE, SECOND, AM_PM. Additional columns like TIME_OF_DAY, HOUR_BAND, DAY_NIGHT, FLAG_BUSINESS_HOURS, PEAK_OFF_PEAK can also be included, depending on the requirements.

### Time Dimension Node Configuration

The Time Dimension node type has two configuration groups:

* [Node Properties](#Time-dimension-node-properties)
* [Time Dimension Options](#Time-options)
* [Additional options](#additional-options)

#### Time Dimension Node Properties

| **Setting** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

##### Time Dimension Options

| **Setting** | **Description** |
|---------|-------------|
| **Generated Time Column Name** |Metadata column name used in the SQL generated for inserting records into the table. |
| **Granularity** |Dropdown: Hour, Minute, Second. Determines the number of records generated: Hour: 24 rows, Minute: 1,440 rows, Second: 86,400 rows. Default: Minute
| **Generated Time Column Name** |Metadata column name used in SQL generation. Default: "TIME_COL"
| **Business Hours Start** |Time value for FLAG_BUSINESS_HOURS calculation. Default: 08:00:00
| **Business Hours End** |Time value for FLAG_BUSINESS_HOURS calculation. Default: 17:00:00
| **Peak Hours Start** |Time value for PEAK_OFF_PEAK calculation. Default: 09:00:00
| **Peak Hours End** |Time value for PEAK_OFF_PEAK calculation. Default: 17:00:00

#### Additional Options

You can create the node as:

* [Table](#Time-table-create-as-table)
* [View](#Time-table-create-as-view)
* [Transient Table](#Time-table-create-as-transient-table)

##### Time Dimension Create as Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Table|
| **Populate on deploy**| Toggle:True/FalseWhen enabled data is populated along with table creation during deployment.[More info on Populate on Deploy for time](#populate-on-deploy-for-time)|
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention if enabled |
| **Default String Value** | If Insert Zero Key Record toggle is True then add a default value for columns with datatype string. Default: UNKNOWN |
| **Default Business Key Value** | If Insert Zero Key Record toggle is True then add a default value for business key column. Default: 99:99:99|
| **Default Time Value (Time Format HH:MM:SS)** | If Insert Zero Key Record toggle is True then add a default value for Time key column in the format HH:MM:SS. Default: 00:00:00|
| **Business key** | Required column for Type 1 Dimensions |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Time Dimension Create as View

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| View|
| **Override Create SQL** | Toggle: True/False**True**: View is created by overriding the SQL**False**: Nodetype defined create view SQL will execute |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

##### Time Dimension Create as Transient Table

| **Setting** | **Description** |
|---------|-------------|
| **Create As**| Transient Table|
| **Populate on deploy**| Toggle:True/FalseWhen enabled data is populated along with table creation during deployment.[More info on Populate on Deploy for time](#populate-on-deploy-for-time)|
| **Business key** | Required column for Type 1 Dimensions |
| **Insert Zero Key Record** | Toggle: True/FalseInsert Zero Key Record to Dimention if enabled |
| **Default String Value** | If Insert Zero Key Record toggle is True then add a default value for columns with datatype string. Default: UNKNOWN |
| **Default Business Key Value** | If Insert Zero Key Record toggle is True then add a default value for business key column. Default: 99:99:99|
| **Default Time Value (Time Format HH:MM:SS)** | If Insert Zero Key Record toggle is True then add a default value for Time key column in the format HH:MM:SS. Default: 00:00:00|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Pre-SQL**| SQL to execute before data insert operation |
| **Post-SQL** | SQL to execute after data insert operation |

### Time Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

### Time Dimension Deployment

#### Time Dimension Initial Deployment

When deployed for the first time into an environment the Time node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Time Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Time View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Time Dimension Redeployment

After the Time node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Time Table or recreating the Time table.

#### Altering the Time Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage upTimes. A few cases are listed below:

1. Changes in business keys
2. Changes in join clauses
3. Transformations made at column level
4. Changing DML options like Insert Zero key

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata UpTime \| Business Keys \| Truncate \| Insert Zero Key \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Time Dimension Recreating the Views

The subsequent deployment of Time node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates a new view with upTimed definition |

#### Time Dimension Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Time table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Time view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Time table/transient table |

> 📘 **Materialization Time Dimension**
>
> When the materialization type of Time node is changed from table/transient table to View and use Override Create SQL for view creation. This ensures that the following change is made in the stage function in Create SQL tab so that the order of deployment is maintained.

### Populate on Deploy for Time
#### Initial deployment

|**Populate ondeploy**| **Insert Zero Key** |**Stages Executed**|
|---------------------|---------------------|-------------------|
|True                 | False               |• Create table/transient table • Insert data on deploy|
|True                 | True                |• Create table/transient table • Insert data on deploy • Insert Zero key records|

#### Redeployment
The stages Insert Zero key and Insert data on deploy are executed if there are changes in transformation,data type,nullability,zero key changes and default values,column addition,Time related config values.

|**S/No**|**Scenario**       |**Populate on Deploy ON**|**Populate on Deploy OFF**|
|--------|-------------------|----------------------|-----------------------|
|1 |Initial deployment |• Create Table • Populate |• Create Table|
|2 |Materialization change | • Drop Current Table  • Create Table Populate | • Drop Current Table  • Create Table |
|3 |Granularity change |• Repopulate | • Metadata Update |
|4 |Business start/end change|• Repopulate|• Metadata Update |
|5 |Peak start/end change|• Repopulate|• Metadata Update |
|6 |Transform logic changes|• Repopulate|• Metadata Update |
|7 |Time column name change|• Repopulate|• Metadata Update |
|8 |Adding a column|• Alter – Add Column • Repopulate|• Alter – Add Column |
|9 |Datatype change |• Alter – Attribute Change • Repopulate |• Alter – Attribute Change |
|10 |Default value change |• Alter – Attribute Change • Repopulate |• Alter – Attribute Change |
|11 |Nullability change |• Alter – Attribute Change • Repopulate |• Alter – Attribute Change |
|12 |Zero key-only changes |• Merge Update Zero Key|• Metadata Update |
|13 |Run Changes |• Display Message |• Populate Data |
|14 |Description changes (table/column) |• Alter – Attribute Change  |• Alter – Attribute Change |
|15 |Rename table |• Alter Table - Rename |• Alter Table - Rename |
|16 |Move table |• Alter Table - Rename |• Alter Table - Rename |
|17 |Rename column |• Alter Table – Rename Column |• Alter Table – Rename Column |
|18 |Drop column |• Alter Table – Drop Column |• Alter Table – Drop Column |
|19 |Pre-SQL / Post-SQL changes |• Metadata Update |• Metadata Update |
|20 |Test enabled changes |• Metadata Update |• Metadata Update |
|21 |Business key changes |• Metadata Update |• Metadata Update |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Time Dimension Deploy Undeployment

If a Time Dimension Node of materialization type table/view/transient table are deleted from a Timespace, that Timespace is committed to Git and that commit deployed to a higher level environment then the TimeTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Pivot

Pivoting ia crucial feature of data transformation.The [Pivot node](https://docs.snowflake.com/en/sql-reference/constructs/pivot) in Coalesce transforms a table by turning the unique values from one column in the input expression into multiple columns and aggregating results where required on any remaining column values. This operation is specified in the `FROM` clause after the table name or subquery.  

It is especially useful for converting narrow tables, such as one with columns for `empid`, `month`, and `sales`, 
into wider tables, for example, `empid`, `jan_sales`, `feb_sales`, and `mar_sales`.

### Pivot Node Configuration

Pivot has three configuration groups: 

* [Node Properties](#pivot-node-properties)
* [General Options](#pivot-general-options)
* [Pivot Options](#pivot-options)

  ![image](https://github.com/user-attachments/assets/edd67d5d-7216-429a-a292-2fe4980d1a9e)

### Pivot limitations

* Currently, the PIVOT semantic doesn’t allow multiple aggregations
* A pivot query that doesn’t use dynamic pivot can return output with duplicate columns. We recommend avoiding output with duplicate columns. A dynamic pivot query deduplicates duplicate columns.
* A pivot query that doesn’t use dynamic pivot might fail if it attempts to CAST a VARIANT column to a different data type. Dynamic pivot queries don’t have this limitation.
  
#### Pivot Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Pivot Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Pivot General Options

![image](https://github.com/user-attachments/assets/3e607c04-f5c8-40ed-8da8-018a5455f520)

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' or 'transient table' |
| **Pivot mode**| Choose ['Simplified'](#simplified-pivot-mode),'Standard'|
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Simplified Pivot Mode

* In Simplified Pivot Mode,we load the data into single DATA variant column
* We have to add a stage node on top of it to flatten it further into columns.

#### Simplified Pivot Options

| **Pivot column**|Pivot column(Dropdown) Pivot column(textbox)The column from the source table or subquery that will be aggregated and turned into new columns.|
|**Value Column**|-Value Column(Dropdown)  -Value Column(textbox)  Values you want to populate in the new columns.|
|**Aggregate Functions**|Aggregation you want to apply, like AVG, COUNT, MAX, MIN, and SUM.|

##### Simplified filter options

| **Filter Column**| Choose column to add any filter condition|
| **Filter Operator**|Choose the operation `=`,`<`,`>`|
| **Filter Value**|Choose the comparison value for filter condition|

#### Standard Pivot Options

##### Single Pivot Column

![image](https://github.com/user-attachments/assets/498aafd3-dba4-4d6e-92e7-ede9b54fef6a)

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Pivot table** | Toggle: True/False  True,it is the first run and the pivot table structure is yet to be determinedFalse,when the pivot table is created and generated columns have been Re-synced in Coalesce|
| **Pivot column**|Pivot column(Dropdown) Pivot column(textbox)The column from the source table or subquery that will be aggregated and turned into new columns.|
| **Single value column** |Toggle: True Determines which if analysis of single or multiple value columns to be added.Value column is the column from the source table or subquery that contains the values from which column names will be generated. |
|**Value Column**|-Value Column(Dropdown)  -Value Column(textbox)  Values you want to populate in the new columns.|
|**Aggregate Functions**|Aggregation you want to apply, like AVG, COUNT, MAX, MIN, and SUM.|
|**Subquery -PIVOT column values**|Not mandatory.A sql query is expected.When a query is mentioned,pivot happens on all values found in the subquery|
|**Filter Column Values(comma separated list of column values-Ex 'Q1','Q2')**|Not mandatory.Specified list of column values for the pivot column|
|**Exclude Columns**|Not mandatory.To specifically exclude columns from a pivot query|
|**Default value for NULL**|Replace all NULL values in the pivot result with the specified default value. The default value can be any scalar expression that does not depend on the pivot and aggregation column|

##### Multiple Pivot Columns

![image](https://github.com/user-attachments/assets/7a9eded6-0ca3-4ed3-9541-b25c943cca51)

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Pivot table** | Toggle: True/False True,it is the first run and the pivot table structure is yet to be determined.False,when the pivot table is created and generated columns have been Re-synced in Coalesce|
| **Pivot column**|Pivot column(Dropdown)  Pivot column(textbox) The column from the source table or subquery that will be aggregated and turned into new columns.|
|**Pivot operation on same column values**|Toggle:True/False - True If pivot is to applied to same pivot column values for multiple value columns- False If pivot is to applied to differnt pivot column values for each value column|
| **Single value column** |Toggle:False Determines which if analysis of single or multiple value columns to be done.Value column is the column from the source table or subquery that contains the values from which column names will be generated. |
|**Value Column**|-Value Column(Dropdown)  -Value Column(textbox) Values you want to populate in the new columns.-Aggregate FunctionsAggregation you want to apply, like AVG, COUNT, MAX, MIN, and SUM.-Column Values  Enabled if the Pivot operation on same column values is false|
|**Filter Column Values(comma separated list of column values-Ex 'Q1','Q2')**|Specified list of column values for the pivot column|
|**Default value for NULL**|Replace all NULL values in the pivot result with the specified default value. The default value can be any scalar expression that does not depend on the pivot and aggregation column|

### Pivot node Usage

* Add a Pivot node on top of source node
* Add the pivot columns,value columns ,aggregation operation from config
* When you choose the pivot and value dropdown,ensure that the textbox alongside the dropdown is entered with Column name.This textBox information is required once the pivot table structure is synced into Coalesce.
* The toggle 'Infer Structure of Pivot Data' is required to be true when the node is created for the first time.
* The toggle 'Single value column' is set to false, if you want a multi-dimensional pivot
* Once the pivot table is created,the 'Re-Sync Columns' can be used to sync the structure of pivot table into Coalesce mapping grid.
  ![image](https://github.com/user-attachments/assets/2f39a419-1662-41e3-9fda-eaaf08c8b1d3)
* After Re-sync,recreate the table with 'Infer Structure of Pivot Data' set to false
* If the above works, it should be deployable as is. Deploy will simply take the columns and execute a create table.
* Hit run to insert data into table keeping the 'Infer Structure of Pivot Data' set to false
  
### Pivot Deployment

### Points to note for deployment
* Create table with ‘Infer PIVOT structure’ toggle enabled
* Re-Sync columns to the mapping grid
* Deploy with ‘Infer PIVOT structure’ toggle set to false
* Repeat the above steps if you see changes in column of table during redeployment.It is fine to skip for change in materialization type,change in target location or change in node name
* Ensure the new columns added or dropped are part of the inferred PIVOT structure and not added/dropped directly in the mapping grid.The deployment will succeed but insert will fail
> 📘 **Deployment**
>
> Ensure 'Infer Pivot structure' set to false before deployment

#### Pivot Initial Deployment

When deployed for the first time into an environment the Pivot node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Pivot Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Pivot View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Pivot Redeployment

After the Pivot node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Pivot Table or recreating the Pivot table.
Pivot
#### Altering the  Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in join clauses
2. Transformations made at column level
5. Default NULL values - Config changes

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Transformation \| Default NULL\| Join** | A dummy statement would execute with specific changes listed in comments|

**Note:** A few configuration changes are not recommended without re-inferring the table before deployment. For more details, please refer to this document - [Metadata Prevention](https://docs.google.com/document/d/1wAh_b_7HIqEEn4Q5lXduO_Li_9MrL4KZ/edit?usp=sharing&ouid=105543507530226126437&rtpof=true&sd=true)

#### Pivot Recreating the Views

The subsequent deployment of Pivot node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Pivot View** | Creates a new view with upPivotd definition |

#### Pivot Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Pivot table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Pivot view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Pivot table/transient table |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Pivot Deploy Undeployment

If a Pivot Node of materialization type table/view/transient table are deleted from a workspace, that workspace is committed to Git and that commit deployed to a higher level environment then the PivotTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Unpivot

The [Unpivot node](https://docs.snowflake.com/en/sql-reference/constructs/unpivot#examples) in Coalesce rotates a table by transforming columns into rows. 
UNPIVOT is not exactly the reverse of PIVOT because it cannot undo aggregations made by PIVOT.

This operator can be used to transform a wide table (e.g. empid, jan_sales, feb_sales, mar_sales) into a narrower table (e.g. empid, month, sales).

### Unpivot limitations

* It cannot reverse aggregations performed by PIVOT
* It requires that all columns have the same data type.In case if the columns from source have diffrent data types,ensure the data types are type casted in an upstream node before adding a UNPIVOT node.
* UNPIVOT cannot be used in dynamic tables or stored procedures
* Ensure that your data is structured and formatted correctly, as any inconsistencies may affect the unpivoting process. It's important to check for any missing values, duplicate entries, or data types that are not compatible with the unpivot function.

### Unpivot Node Configuration

Unpivot has three configuration groups: 

* [Node Properties](#unpivot-node-properties)
* [General Options](#unpivot-general-options)
* [Unpivot Options](#unpivot-options)

#### Unpivot Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Pivot Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

 ![image](https://github.com/user-attachments/assets/edd67d5d-7216-429a-a292-2fe4980d1a9e)

#### Unpivot general Options

![image](https://github.com/user-attachments/assets/3e607c04-f5c8-40ed-8da8-018a5455f520)

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' or 'transient table' |
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Unpivot Options

![image](https://github.com/user-attachments/assets/45d4e9ae-8da3-46cf-b6bf-37ff693c8fa9)

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Pivot table** | Toggle: True/False True,it is the first run and the pivot table structure is yet to be determined.False,when the pivot table is created and generated columns have been Re-synced in Coalesce|
| **Value-Coulmn** |Column that will hold the values from the unpivoted columns |
| **Name-column** | Column that will hold the names of the unpivoted columns  |
| **Column-list**| The names of the columns in the source table or subquery that will be rotated into a single pivot column|
| **Include NULLS**| Specifies whether to include or exclude rows with NULLs|

### Unpivot node Usage

* Add a Unpivot node on top of source node
* Add the Unpivot column list ,value column,name column in config
* When you choose the Unpivot and value dropdown,ensure that the textbox alongside the dropdown is entered with Column name.This textBox information is required once the Unpivot table structure is synced into Coalesce.
* The toggle 'Infer Structure of Unpivot Data' is required to be true when the node is created for the first time.
* The toggle 'Single value column' is set to false, if you want a multi-dimensional Unpivot
* Once the Unpivot table is created,the 'Re-Sync Columns' can be used to sync the structure of Unpivot table into Coalesce mapping grid.
* After Re-sync,recreate the table with 'Infer Structure of Unpivot Data' set to false
  ![image](https://github.com/user-attachments/assets/79282085-e9b7-41e1-8bd2-68fdf98eeb00)
* If the above works, it should be deployable as is. Deploy will simply take the columns and execute a create table.
* Hit run to insert data into table keeping the 'Infer Structure of Pivot Data' set to false

#### Unpivot Initial Deployment

### Points to note for deployment
* Create table with ‘Infer UNPIVOT structure’ toggle enabled
* Re-Sync columns to the mapping grid
* Deploy with ‘Infer UNPIVOT structure’ toggle set to false
* Repeat the above steps if you see changes in column of table during redeployment.It is fine to skip for change in materialization type,change in target location or change in node name
* Ensure the new columns added or dropped are part of the inferred UNPIVOT structure and not added/dropped directly in the mapping grid.The deployment will succeed but insert will fail

> 📘 **Deployment**
>
> Ensure 'Infer Unpivot structure' set to false before deployment

When deployed for the first time into an environment the Unpivot node of materialization type table or view will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Unpivot Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Unpivot View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Unpivot Redeployment

After the Unpivot node with materialization type table/transient table/view has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Unpivot Table or recreating the Unpivot table.
Unpivot
#### Altering the  Table and Transient Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

1. Changing table names
2. Dropping existing columns
3. Altering column data types
4. Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/ Alter Column/ Delete Column/ Add Column/Edit table description** | Alter table statement is executed to perform the alter operation |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates. A few cases are listed below:

1. Changes in join clauses
2. Transformations made at column level
5. Default NULL values - Config changes

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Transformation \| Default NULL\| Join** | A dummy statement would execute with specific changes listed in comments|

**Note:** A few configuration changes are not recommended without re-inferring the table before deployment. For more details, please refer to this document - [Metadata Prevention](https://docs.google.com/document/d/1wAh_b_7HIqEEn4Q5lXduO_Li_9MrL4KZ/edit?usp=sharing&ouid=105543507530226126437&rtpof=true&sd=true)

#### Unpivot Recreating the Views

The subsequent deployment of Unpivot node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Unpivot View** | Creates a new view with upUnpivotd definition |

#### Unpivot Drop and Recreate View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Unpivot table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Unpivot view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Unpivot table/transient table |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Unpivot Deploy Undeployment

If a Unpivot Node of materialization type table/view/transient table are deleted from a Unpivotspace, that Unpivotspace is committed to Git and that commit deployed to a higher level environment then the UnpivotTable in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view** | Removes the table or view from the environment |

## Match Recognize

The Match Recognize node type uses Snowflake's SQL MATCH_RECOGNIZE clause to identify and process patterns in datasets(https://docs.snowflake.com/en/sql-reference/constructs/match_recognize). It helps identify events, trends, and anomalies by analyzing rows in sequence. It is useful for tasks like fraud detection, session tracking, and clickstream analysis, with options to customize patterns for different needs.

### Match Recognize Node Configuration

Match Recognize has three configuration groups: 

* [Node Properties](#Match-Recognize-node-properties)
* [General Options](#Match-Recognize-general-options)
* [Match Recognize Options](#Match-Recognize-options)

#### Match Recognize Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Match Recognize Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Match Recognize General Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' or 'transient table' |
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |

#### Match Recognize Infer Column Option

| **Options** | **Description** |
|-------------|-----------------|
| **Infer structure of Match Recognize table** | Toggle: True/False  True,it is the first run and the Match Recognize table structure is yet to be determinedFalse,when the Match Recognize table is created and generated columns have been Re-synced in Coalesce|

#### Match Recognize Partitioning & Ordering

| **Options** | **Description** |
|-------------|-----------------|
| **Partition By** | Toggle: True/False  True,Enable the Column Dropdown and Textbox to add columns for partitioningFalse,Disable the Dropdown and Textbox to add columns|
| **Match Recognize Column**|Match Recognize column(Dropdown) Match Recognize column(textbox) The column from the source table or subquery that will be added in Partition By Clause of the Match Recognize Query|
| **Order By** | Toggle: True/False  True,Enable the Column Dropdown and Textbox to add columns for adding in order by clauseFalse,Disable the Dropdown and Textbox to add columns|
| **Match Recognize Column**|Match Recognize column(Dropdown) Match Recognize column(textbox)The column from the source table or subquery that will be added in Order By Clause of the Match Recognize Query|

#### Match Recognize Output Specification

##### Measures
| **Options** | **Description** |
|-------------|-----------------|
| **Expression** | Expression Textbox  A function name to be added as part of major for e.g MATCH_NUMBER,COUNT etc|
| **Column Name (OR Value to pass to Expression)**|Column Textbox Acolumn name or the value to the function mentioned in the expression of measures of the Match Recognize Query|
| **Alias of Expression Column Name** | Alias Expression Textbox  Alias of the expression  of measures of the Match Recognize Query|

#### Match Recognize Match Control
| **Options** | **Description** |
|-------------|-----------------|
| **Rows Per Match** | (Dropdown)  Specifies which rows are returned for a successful match.Select ONE ROW PER MATCH , ALL ROWS PER MATCH or BLANK if not required
| **All Rows Per Match** | (Dropdown)  Returns a row for each row that is part of the match. Select SHOW EMPTY MATCHES, OMIT EMPTY MATCHES , WITH UNMATCHED ROWS or BLANK if not required 
| **After Match Skip** | (Dropdown)  Specifies where to continue after a match. Select PAST LAST ROW , TO NEXT ROW, TO or BLANK (if not required )
| **To** | (Dropdown)  Continue matching at the first or last (default) row that was matched to the given symbol. Select  FIRST or LAST.
| **After match skip variable name** | (Textbox)  Add an alias name for the after match skip clause.

#### Match Recognize Pattern & Conditions
| **Options** | **Description** |
|-------------|-----------------|
| **Pattern** | (Textbox)  The pattern defines a valid sequence of rows that represents a match. The pattern is defined like a regular expression. (regex) and is built from symbols, operators, and quantifiers.

##### Define
| **Options** | **Description** |
|-------------|-----------------|
| **Expression** | (Textbox)  Defining symbols (also known as “pattern variables”) are the building blocks of the pattern.A symbol is defined by an expression.The 
| **Column Name** | (Textbox)  It is the symbol name of the expression

#### Match Recognize Select Query Options
| **Options** | **Description** |
|-------------|-----------------|
| **Select Query Functions** | Toggle: True/False  True,Enable Expression and columns to add aggregate functions to Select QueryFalse,Disable the option|
| **Expression** | Expression Textbox  A function name to be added as part of major for e.g AVG,COUNT etc|
| **Column Name (OR Value to pass to Expression)**|Column Textbox Acolumn name or the value to the function mentioned in the expression of select quey of the Match Recognize Query|
| **Alias of Expression Column Name** | Alias Expression Textbox  Alias of the expression  of select query of the Match Recognize Query|
| **Select Query Order By** | Toggle: True/False  True,Enable the Column Dropdown and Textbox to add columns for adding in order by clause of select queryFalse,Disable the Dropdown and Textbox to add columns|
| **Select Query Order By Column**|Select Query Order By column(Dropdown) Select Query Order By column(textbox).The column from the source table or subquery that will be added in Order By Clause of the Select Query|

### Match Recognize node Usage

* Add a Match Recognize node on top of source node
* Add the Match Recognize options from config
* When you choose the Match Recognize and value dropdown,ensure that the textbox alongside the dropdown is entered with Column name.This textBox information is required once the Match Recognize table structure is synced into Coalesce.
* The toggle 'Infer Structure of Match Recognize Data' is required to be true when the node is created for the first time.
* Once the Match Recognize table is created,the 'Re-Sync Columns' can be used to sync the structure of Match Recognize table into Coalesce mapping grid.
* For further Match Recognize operations,keep the 'Infer Structure of Match Recognize Data' set to false
  
### Match Recognize Deployment

### Points to note for deployment
* Create table with ‘Infer Structure of Match Recognize Data’ toggle enabled
* Re-Sync columns to the mapping grid
* Deploy with ‘Infer Structure of Match Recognize Data’ toggle set to false
* Repeat the above steps if you see changes in column of table during redeployment.It is fine to skip for change in materialization type,change in target location or change in node name
* Ensure the new columns added or dropped are part of the inferred Match Recognize structure and not added/dropped directly in the mapping grid.The deployment will succeed but insert will fail.

### Match Recognize Initial Deployment
When deployed for the first time into an environment the Match Recognize node of materialization type table or view or transient table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Match Recognize Table/transient table/view** | This will execute a CREATE OR REPLACE statement and create a Match Recognize table in the target environment |

#### Match Recognize Table Redeployment

When the Match Recognize node is redeployed with any changes in table or config changes result in re-creating the node

The below stage is executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Match Recognize Table/transient table/view** | This will execute a CREATE OR REPLACE statement and create a Match Recognize table in the target environment |

#### Match Recognize Table Deploy Drop and Recreate Work View/Table/Transient Table

| **Change** | **Stages Executed** |
|------------|-------------------|
| **View to table/transient table** |  Drop view  Create or Replace Match Recognize table/transient table |
| **Table/transient table to View** |  Drop table/transient table Create Match Recognize view |
| **Table to transient table or vice versa** |  Drop table/transient table Create or Replace Match Recognize table/transient table |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Match Recognize Tables Undeployment
If a Match Recognize Node of materialization type table/view/transient table are deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Match Recognize node in the target environment will be dropped.

This is executed in below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop table/view/transient table** | Removes the table or view from the environment |

## View-Qualify advanced deploy

The Coalesce View-Qualify advanced deploy UDN is a versatile node that allows you to develop and deploy a View in Snowflake with an added QUALIFY filter.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.In a SELECT statement, the QUALIFY clause filters the results of window functions.

[QUALIFY](https://docs.snowflake.com/en/sql-reference/constructs/qualify) does with window functions what HAVING does with aggregate functions and GROUP BY clauses.

### View Node Configuration

The View node type has two configuration groups:

* [Node Properties](#view-node-properties)
* [Options](#view-options)

![View-qualify NC](https://github.com/user-attachments/assets/6cf9f488-488f-4b0c-bf77-ab1b56f40b6c)

#### View Node Properties

| **Properties** | **Description** |
|----------|-------------|
| **Storage Location** | Storage Location where the WORK will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |

#### View Options

| **Options** | **Description** |
|---------|-------------|
| **QUALIFY filter** | Toggle: True/False**True**: Quaify filter on a window function is added to the view definition and the configs related to qualify filter are displayed **False**: A simple view is created |
| **Window functions** | A drop down with the list of window functions is visible.Window function to be used in QULAIFY filter predicate is chosen.  |
| **Window function column name**|The window fucntion is applied to this specific column.This is not required for window functions like 'ROW_NUMBER','RANK','DENSE_RANK','PERCENT_RANK','NTILE'|
| **Constant value-desired number of buckets**|Enabled whn the window function chosen is "NTILE"|
| **Partition By** | Toggle: True/False**True**: Lists the columns to be used for partitioning window function **False**: Partition by column drop down is invisible.Check [preferences](#preferences) for more info |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause**False**: Sort column and sort order drop down are invisible.Check [preferences](#preferences) for more info |
| **Operator** | A drop with the list of comparison operators are listed down |
| **Compare return value of function with window column** | Toggle: True/False**True**: The return value of window function is compared with the column chosen for window function **False**: Compared with an expected value entered in the config below|
| **Expected value**| The expected value possibly an integer to compare the results of window function|

![View-Qialify options](https://github.com/user-attachments/assets/67a7d373-d437-4eaf-b30b-2deda610b4ba)

### Preferences

* PARTITION BY is optional.You can omit PARTITION BY if you want to treat the entire result set as one partition.
* ORDER BY is also optional.But it is required for ranking or cumulative functions where order matters (e.g., ROW_NUMBER(), RANK(), LAG(), LEAD())
* ORDER BY is optional for functions like AVG(), SUM(), etc., unless you want cumulative behavior.
  
### View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

> 📘 **Specify Group by Clauses**
>
> Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

### View Deployment

#### View Initial Deployment

When deployed for the first time into an environment the View node will execute the Create View stage.

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a View in the target environment |

#### View Redeployment

The subsequent deployment of View node with changes in node name or node location results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

Changes in QUALIFY filter or view definition results in recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create View** | Creates new view with updated definition |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

#### View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop View** | Removes the view from the environment |

## Recursive CTE

A (Recursive Common Table Expression CTE)[https://docs.snowflake.com/en/user-guide/queries-cte#recursive-ctes-and-hierarchical-data] in Snowflake is a powerful SQL feature that allows you to query hierarchical or self-referential data by referencing itself. This is particularly useful for tasks like traversing organizational hierarchies, processing tree structures, or calculating cumulative totals.

### Recursive CTE node Usage

#### Adding Recursive CTE node on a source

* Add a Recursive CTE node on top of source(Ex:Employees)
  
  
* Keep the multi-source toggle ON and add three sources-Anchor clause,Recursive clause and CTE clause.Multiple sources in the context of this node does not mean mapping to three different sources.All three clauses refer to single source or a join of sources.

  
* The sources need not be mapped to Recursive or CTE clauses.From clauses need not be generated in join tab.
* Ensure source columns are mapped to appropriate source tables/views in recursive clause
  
* Use the join tab to add termination condition for recursive clause else there are chances for recursive cte to go on an infinite loop
  
* In case of applying aggregation to CTE,there might be a need to exclude columns from the CTE.A (mapping column)[#mapping-column] "Column Specification" is mapping grid which when set to 'N' for a specific column,it is excluded from CTE and target as well.

* In case if multiple tables are joined in anchor clause,then ensure you specify the anchor table name in config.
  
#### Adding a Recursive CTE with no source

All the above steps specified above applies here as well except source columns need not be mapped to source tables/view in recursive clause.

### Recursive CTE limitations

* We can join multiple sources in anchor clause but union of multiple sources in anchor clause is not supported
* A termination condition is required in recursive clause to avoid infinite loop
* A cyclic data heirarchy might make Recursive CTE run until it succeeds or times out

### Recursive CTE Node Configuration

Recursive CTE has three configuration groups: 

* [Node Properties](#recursive-cte-properties)
* [Options](#recursive-general-options)

#### Recursive CTE Node Properties

| **Property** | **Description** |
|--------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Pivot Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Recursive general Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Choose 'table', 'view' or 'transient table' |
| **Truncate** | True/False toggle to enable or disable truncating the output columns |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Multiple sources joined in Anchor clause**| Toggle: True/FalseIf enabled we need to specify which among the joined tables act as Anchor table|
| **Anchor table name**| Specify the anchor table name if multiple tables are joined in anchor clause|
| **Anchor and CTE join column**| Specify the column names to be joined from anchor and CTE clause.Note:Enabled if the node type has a source.In case of a sequence generation or date series generation,this option will not be dispalyed|

### Mapping Column
* A mapping column "Column Specification" is added to mapping grid if there is a requirement to skip any column only from CTE and eventuallt the target table
* In order to skip a column,set "Column Specification" to 'N'
* In the context of Recursive CTE node type,ony a value of 'N' is accepted by "Column Specification".
  
### Recursive CTE Deployment

#### Recursive CTE Initial Deployment

When deployed for the first time into an environment the Recursive CTE node of materialization type table will execute the below stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment |
| **Create Target View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment |

#### Recursive CTE Redeployment

After the Recursive CTE node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Recursive CTE Table or recreating the Recursive CTE table.

#### Altering the Recursive CTE Tables

A few types of column or table changes will result in an ALTER statement to modify the Recursive CTE Table in the target environment, whether these changes are made individually or all together:

* Changing table names
* Dropping existing columns
* Altering column data types
* Adding new columns

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table\| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation |
| **Swap Cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Recreating the Recursive CTE Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

* Join clause
* Adding transformation
* Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Table** | Creates a new table |

#### Recreating the Recursive CTE Views

The subsequent deployment of Recursive CTE node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Removes existing view |
| **Create View** | Creates new view with updated definition |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Recursive CTE Undeployment

If a Recursive CTE Node of materialization type table is deleted from a Workspace, that Recursive CTEspace is committed to Git and that commit deployed to a higher level environment then the Recursive CTE Table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

If a Recursive CTE Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Recursive CTE View in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Delete View** | Drops the existing Recursive CTE view from target environment |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Table | Table | Follows existing redeployment stages |
| Transient Table | Transient Table | Follows existing redeployment stages |
| View | View | Follows existing redeployment stages |
| Any Other | Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Transient Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | View | 1. Warning (if applicable)2. Drop  3. Create |

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added when there's a node type switch |

--------------

## Code

### Date Table Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Date-404/definition.yml)
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Date-404/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Date-404/run.sql.j2) |

### Time Dimension Table Code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/TimeDimension-603/definition.yml)
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/TimeDimension-603/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/TimeDimension-603/run.sql.j2) |

### Pivot code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Pivot-409/definition.yml)|
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Pivot-409/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Pivot-409/run.sql.j2)

### Unpivot code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Unpivot-399/definition.yml)|
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Unpivot-399/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/Unpivot-399/run.sql.j2)

### Match Recognize code

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/MatchRecognize-410/definition.yml)|
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/MatchRecognize-410/create.sql.j2) |
| **Run Template** | [run.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/MatchRecognize-410/run.sql.j2)

### View-Qualify advanced deploy

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/View-Qualifyadvanceddeploy-459/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/View-Qualifyadvanceddeploy-459/create.sql.j2)|

### Recursive CTE

| **Component** | **Link** |
|--------------|-----------|
| **Node definition** | [definition.yml](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/RecursiveCTE-586/definition.yml) |
| **Create Template** | [create.sql.j2](https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/RecursiveCTE-586/create.sql.j2) |
| **Run Template** | [run.sql.js] (https://github.com/coalesceio/functional-node-types/blob/main/nodeTypes/RecursiveCTE-586/run.sql.j2) |

[Macros](https://github.com/coalesceio/functional-node-types/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 4.3.3 | April 23, 2026 | Enclosing Pivot and Unpivot columns within quotes |
| 4.3.2 | April 09, 2026 | NM-220 Added Node Type Switch support in advance deploy templates |
| 4.3.1 | March 12, 2026 | Simplified Pivot mode added, get_clause() macro modified |
| 4.3.0 | December 12, 2025 | Time Dimension - New Node Type for time-of-day analytics |
| 4.2.3 | December 10, 2025 | 'Populate on deploy' toggle enabled |
| 4.2.2 | November 18, 2025 | Aggregation support for Recursive CTE |

---

## Iceberg Tables

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/iceberg-tables</code> <CopyButton textToCopy='@coalesce/snowflake/iceberg-tables' />
Supported Platform: 
Latest Version: 3.3.0 (May 27, 2026)

apache icebergdata lakeexternal catalogiceberginteroperabilitylakehouseopen table formatpolaris

## Overview
The Iceberg Tables Package provides tools to manage and deploy Iceberg tables in Snowflake or integrate with externally managed catalogs.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/iceberg-tables</code> <CopyButton textToCopy='@coalesce/snowflake/iceberg-tables' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Iceberg Tables Package – Business Summary

The Iceberg Tables Package provides a standardized framework for creating and managing Apache Iceberg–based tables within Snowflake environments. Built on the open table format from Apache Iceberg and integrated with Snowflake, the package enables organizations to store data in external cloud storage while maintaining Snowflake’s performance, governance, and SQL capabilities.

This package supports multiple ingestion and processing patterns, including batch loading (Copy Into), automated micro-batch ingestion (Snowpipe), externally cataloged Iceberg tables, and dynamic Iceberg tables for incremental transformations. It allows businesses to work directly with data lake storage such as AWS S3, Azure Blob Storage without moving data into native Snowflake storage.

By combining open data lake storage with Snowflake’s compute, optimization, and lifecycle management features, the Iceberg Tables Package enables scalable, cost-efficient, and flexible data architectures. It is ideal for organizations that want to modernize data lake environments, support near real-time ingestion, implement incremental data pipelines, and maintain ACID compliance while leveraging open standards.

-------

## Iceberg Tables Package includes

* [Snowflake Iceberg table](#snowflake-iceberg-table)
* [External Iceberg table](#external-iceberg-table)
* [External Iceberg REST Table](#external-iceberg-rest-table)
* [Copy-Into Iceberg table](#copy-into-iceberg-table)
* [Snowpipe Iceberg table](#snowpipe-iceberg-table)
* [Dynamic Iceberg Work table](#Dynamic-Iceberg-Work-table)
* [Dynamic Iceberg Dimension table](#Dynamic-Iceberg-Dimension-table)
* [Dynamic Iceberg Latest Record Version table](#Dynamic-Iceberg-Latest-Record-Version-table)
* [Code](#code)

## Snowflake Iceberg Table

An Iceberg table uses the Apache Iceberg open table format specification, which provides an abstraction layer on data files stored in open formats. [Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg) for Snowflake combine the performance and query semantics of regular Snowflake tables with external cloud storage that you manage. They are ideal for existing data lakes that you can't, or choose not to, store in Snowflake.

An Iceberg table that uses Snowflake as the Iceberg catalog provides full Snowflake platform support with read and write access. The table data and metadata are stored in external cloud storage, which Snowflake accesses using an external volume. Snowflake handles all life-cycle maintenance, such as compaction, for the table.

### Snowflake Iceberg Table Prerequisites

* The Role mentioned in the Workspace and Environment properties of Coalesce should be ACCOUNTADMIN in order to successfully create an Iceberg table.
* An EXTERNAL VOLUME is expected to be created in Snowflake at the Storage Location chosen in the Node properties.
* In case of creating a Snowflake Iceberg table with structured column types like OBJECT, MAP, or ARRAY, ensure the data type is updated with the appropriate structure. For example, if the source Snowflake table has OBJECT data type, then the data type of the same column in the Iceberg table node added on top is expected to be structured type OBJECT (age string, id number) based on the data it has.

### Snowflake Iceberg Node Configuration

The Snowflake Iceberg table has two configuration groups:

* [Snowflake Iceberg Node Properties](#snowflake-iceberg-node-properties)
* [Snowflake Iceberg Options](#snowflake-iceberg-options)

#### Snowflake Iceberg Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Snowflake Iceberg Options

| **Option** | **Description** |
|--------|-------------|
| **Type of Catalog** | Specify the type of catalog:**Snowflake**  **Polaris** |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in snowflake as a prerequisite |
| **Base location name** | The path to a directory where Snowflake can write data and metadata files for the table. Specify a relative path from the table's EXTERNAL_VOLUME location |
| **Catalog integration** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. This option is enabled if the type of catalog is Polaris |
| **Cluster key** | True/False to determine whether the Iceberg table is to be clustered or not***True** - Allows you to specify the column based on which clustering is to be done.* **Allow Expressions Cluster Key** - True allows to add an expression to the specified cluster key* **False** - No clustering done |

### Snowflake Iceberg Deployment

#### Snowflake Iceberg Initial Deployment

When deployed for the first time into an environment the node will execute the following stage:

| **Stage** | **Description** |
|-------|-------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment |

#### Snowflake Iceberg Redeployment

Any changes in config options will result in recreating the table during redeployment.

* `EXTERNAL VOLUME`
* `Base location name`
* `Node Properties`
* `Column Name`

| **Stage** | **Description** |
|-------|-------------|
| **Create or Replace Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment |

##### Snowflake Iceberg Alter Structured Data Type

Changes in structured data type columns excluding any other config changes results in `ALTER` statements execution.

| **Stage** | **Description** |
|-------|-------------|
| **Alter Iceberg Table** | This stage will execute `ALTER ICEBERG TABLE` statement and alters structured data types like OBJECT, MAP or ARRAY |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

## Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

#### Snowflake Iceberg Undeployment

If a Snowflake Iceberg table is dropped from the workspace and commited to Git results in table dropped from target environment.

| **Stage** | **Description** |
|-------|-------------|
| **Drop Iceberg Table** | Removes the Iceberg table from the target environment |

## External Iceberg Table

External Iceberg table node type is an external Iceberg table node type wrapped with task functionality.

An Iceberg table uses the Apache Iceberg open table format specification, which provides an abstraction layer on data files stored in open formats. [Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg) for Snowflake combine the performance and query semantics of regular Snowflake tables with external cloud storage that you manage. They are ideal for existing data lakes that you cannot, or choose not to, store in Snowflake.

An Iceberg table that uses an external catalog provides limited Snowflake platform support with read-only access. With this table type, Snowflake uses a catalog integration to retrieve information about your Iceberg metadata and schema.

You can use this option to create an Iceberg table registered in the AWS Glue Data Catalog or to create a table from Iceberg metadata files in object storage.

### External Iceberg Table Prerequisites

* The Role in the Workspace and Environment properties of Coalesce should be `ACCOUNTADMIN` in order to successfully create an Iceberg table. You can also grant `SYSADMIN` roles to `EXTERNAL VOLUME`, `CATALOG INTEGRATION` created.
* An `EXTERNAL VOLUME`, `CATALOG INTEGRATION` is expected to be created in Snowflake at the Storage Location chosen in the Node properties.
* To enable task success/failure notifications, an AWS SNS Notification Integration must be set up in Snowflake. This requires an AWS SNS Topic, an IAM Role with SNS publish permissions, and a Snowflake Notification Integration linked to the SNS topic. Refer to the [Snowflake SNS Integration Guide](https://docs.snowflake.com/en/user-guide/notifications/creating-notification-integration-amazon-sns) for setup steps.

### External Iceberg Table Configuration

* [External Iceberg Table Node Properties](#external-iceberg-table-node-properties)
* [External Iceberg Table Options](#external-iceberg-table-options)
* [External Iceberg Table Scheduling Options](#external-iceberg-table-task-scheduling-options)
* [External Iceberg Table System Columns](#external-iceberg-table-system-columns)

### External Iceberg Table Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** (required) | Storage Location where the Iceberg Table will be created. |
| **Node Type** (required) | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** (required) | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment. |

### External Iceberg Table Options

| **Options** | **Description** |
|-------------|-----------------|
| **Type of Catalog** | Specify the type of catalog:- AWS Glue- Object Storage |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in Snowflake as a prerequisite. |
| **Catalog integration** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. |
| **Catalog namespace** | Optionally specifies the namespace (for example, `my_glue_database`) for the AWS Glue Data Catalog source. Option available if AWS Glue catalog is chosen. |
| **Catalog table name** | Name of the catalog table. Option available if AWS Glue catalog is chosen. |
| **Metadata filepath** | Specifies the relative path of the Iceberg metadata file to use for column definitions. Option available if Object Storage Catalog is chosen. |
| **Schedule refresh** | True or False toggle that determines whether a task will be created or if the SQL to be used in the task will execute DML as a Run action. Prior to creating a task, it is helpful to test the SQL the task will execute to make sure it runs without errors and returns the expected data.- **False** - A table will be created and SQL will execute as a Run action.- **True** - After sufficiently testing the SQL as a Run action, setting Schedule refresh Mode to true will wrap the SQL statement in a task with options specified in Scheduling Options.When Run is executed, a message appears prompting the user to wait for the refresh to occur. |

### External Iceberg Table Task Scheduling Options

If schedule refresh mode is set to true then Task Scheduling Options can be used to configure how and when the task will run.

| **Options** | **Description** |
|-------------|------------------------|
| **Scheduling Mode** | **Warehouse Task** - User managed warehouse will execute tasks**Serverless Task** - Utilize serverless compute to execute tasks |
| **Select Warehouse on which to run task** | Enter the name of the warehouse you want the task to run on without quotes*(Visible if Scheduling Mode is set to Warehouse Task)* |
| **Select initial serverless Warehouse size** | Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times*(Visible when Scheduling Mode is set to Serverless Task)* |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | **Minutes** - Allows you to specify a minute interval for running task**Cron** - Allows you to specify a CRON schedule for running task**Predecessor** - Allows you to specify a predecessor task to determine when a task should execute |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 3.1.0 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format.*|
| **Enter task schedule using Cron** | Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax*(Only visible when Task Schedule is set to Cron)* |
| **Enter predecessor tasks separated by a comma** | One or more task names that precede the task being created in the current node. Task names are case sensitive and should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor tasks separate the task names using a comma and no spaces*(Only visible when Task Schedule is set to Predecessor)* |
| **Enter root task name** | Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor tasks separate the task names using a comma and no spaces |

### External Iceberg Table Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

### External Iceberg Table Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### External Iceberg Table System Columns

**DATA** - Column added for deployment but column is not added to iceberg table as columns specifications are not required for External Iceberg tables.The exact columns are refreshed in the mapping grid using **Re-Sync Columns** button in the top right corner in mapping grid.The other option to refresh columns is by using [API-NODEUPDATE](https://github.com/coalesceio/External-Data-Package/blob/main/README.md#API-NODEUPDATE) node type.

### External Iceberg Table With Task Deployment Parameters

### Task Deployment Parameters

All parameters default to `DEV ENVIRONMENT`, which uses the value set in **Task Scheduling Options**. Set to any other value to override per environment.

| Parameter | Description |
|---|---|
| `targetTaskWarehouse` | Warehouse used to run the task |
| `targetTaskExecuteAsUser` | Snowflake user the task executes as |
| `targetTaskErrorIntegration` | SNS integration for error notifications |
| `targetTaskSuccessIntegration` | SNS integration for success notifications |

---

**Example — DEV ENVIRONMENT (uses Task Scheduling Options values)**
```json
{
  "targetTaskWarehouse": "DEV ENVIRONMENT",
  "targetTaskExecuteAsUser": "DEV ENVIRONMENT",
  "targetTaskErrorIntegration": "DEV ENVIRONMENT",
  "targetTaskSuccessIntegration": "DEV ENVIRONMENT"
}
```

**Example — Override for QA/Production environment**
```json
{
  "targetTaskWarehouse": "compute_wh",
  "targetTaskExecuteAsUser": "qa_user",
  "targetTaskErrorIntegration": "my_error_integration",
  "targetTaskSuccessIntegration": "my_success_integration"
}
```

### External Iceberg Table Initial Deployment

When deployed for the first time into an environment the External Iceberg Table node will execute three stages dependent on whether or not the task schedule relies on a predecessor task.

**External Iceberg Table No Predecessor Task Deployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |
| **Resume Task** | After the task has been created it needs to be resume so that the task runs on the schedule. |

**External Iceberg Table Predecessor Task Deployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |
| **Suspend Root Task** | To add a task into a DAG of task the root task needs to be put into a suspended state. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |

If a task is part of a DAG of tasks, the DAG needs to include a node type called **Task Dag Resume Root**. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task enabled nodes are CREATE OR REPLACE only though this is subject to change.

### External Iceberg Table Redeployment

If any changes in config options like external volume, base location, node properties, or column results in recreating the Iceberg table during redeployment.

#### Recreating the External Snowflake Iceberg Table

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |

#### External Iceberg Table Recreating the Task Redeployment

After the Task has deployed for the first time into a target environment, subsequent deployments with changes in task schedule, warehouse, or scheduling options will result in a `CREATE TASK`AND `RESUME TASK` statements being issued.

The following stages are executed.

**External Iceberg Table No Predecessor Task Redeployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |
| **Resume Task** | After the task has been created it needs to be resume so that the task runs on the schedule. |

**External Iceberg Table Predecessor Task Redeployment**

| **Stage** | **Description** |
|----------|---------------|
| **Suspend Root Task** | To add a task into a DAG of task the root task needs to be put into a suspended state. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### External Iceberg Table Undeployment

If a Snowflake iceberg table with task is dropped from the workspace and commited to Git results in table and task dropped from target environment.

The following stages are executed.

**External Iceberg Table No Predecessor Task Dropped**

| **Stage** | **Description** |
|----------|---------------|
| **Drop Iceberg Table** | Drop Iceberg Table |
| **Drop Current Task** | This stage will drop the task. |

**External Iceberg table Predecessor Task Dropped**

| **Stage** | **Description** |
|----------|---------------|
| **Drop Iceberg Table** | Drop Iceberg Table |
| **Suspend Root Task** | To drop a task from a DAG of task the root task needs to be put into a suspended state. |
| **Drop Task** | This stage will drop the task. |

---

## External Iceberg REST Table

External Iceberg REST Table node type provides an external Apache Iceberg table wrapped with task functionality.

An Iceberg table uses the Apache Iceberg open table format specification, which provides an abstraction layer on data files stored in open formats. [Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg) for Snowflake combine the performance and query semantics of regular Snowflake tables with external cloud storage that you manage. They are ideal for existing data lakes that you cannot, or choose not to, store in Snowflake.

An Iceberg table that uses an external catalog provides limited Snowflake platform support with read-only access. Snowflake added write support for externally managed Iceberg tables — including AWS Glue — in October 2025 (GA; Preview began July 2025). With this table type, Snowflake uses a catalog integration to retrieve information about your Iceberg metadata and schema and also write data to AWS Glue-managed Apache Iceberg tables from Coalesce.

You can use this option to create an Iceberg table registered in the AWS Glue Data Catalog and load data into them. Currently supports INSERT only loads.

### External Iceberg REST Table Prerequisites

* The Role in the Workspace and Environment properties of Coalesce should be `ACCOUNTADMIN` in order to successfully create an Iceberg table. You can also grant `SYSADMIN` roles to `EXTERNAL VOLUME`, `CATALOG INTEGRATION` created.
* A Snowflake Catalog Integration configured with `CATALOG_SOURCE = ICEBERG_REST` and `CATALOG_API_TYPE = AWS_GLUE` using SigV4 authentication.
* An AWS IAM role with read and write permissions on the target AWS Glue Data Catalog and the underlying S3 location.
* A Snowflake External Volume configured with `ALLOW_WRITES = TRUE`  pointing to the target S3 location.
* Either catalog-vended credentials or a storage integration configured for Snowflake to access the underlying S3 data.
* Use a Snowflake **catalog-linked database** to create and register the Iceberg table in AWS Glue during deployment with Create Mode.
* An AWS Glue Database created in the Glue Data Catalog, under which Iceberg tables will be registered and managed.
* To enable task success/failure notifications, an AWS SNS Notification Integration must be set up in Snowflake. This requires an AWS SNS Topic, an IAM Role with SNS publish permissions, and a Snowflake Notification Integration linked to the SNS topic. Refer to the [Snowflake SNS Integration Guide](https://docs.snowflake.com/en/user-guide/notifications/creating-notification-integration-amazon-sns) for setup steps.

### Key Features
 
- **Node-level table linking** — each node points to a specific Glue-managed Iceberg table
- **Two modes supported:**

    | Mode | Description |
    |---|---|
    | **Create with Schema Mode** | Node creates a new Iceberg table and registers it in AWS Glue Data Catalog at deploy time ([requires catalog-linked database](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-rest)) |
    | **Sync Mode** | Node links to an already existing Glue Iceberg table and writes to it |

### External Iceberg REST Table Configuration

* [External Iceberg REST Table Node Properties](#external-iceberg-rest-table-node-properties)
* [External Iceberg REST Table Options](#external-iceberg-rest-table-create-options)
* [External Iceberg REST Table Options](#external-iceberg-rest-table-load-options)
* [External Iceberg REST Table Scheduling Options](#external-iceberg-rest-table-task-scheduling-options)

### External Iceberg REST Table Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** (required) | Storage Location where the Iceberg REST Table will be created. |
| **Node Type** (required) | Name of template used to create node objects. |
| **Deploy Enabled** (required) | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment. |

### External Iceberg REST Table Create Mode Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Create as - iceberg table |
| **Type of Catalog** | Specify the type of catalog:- AWS Glue |
| **Create Table with Schema Mode** | Creates and registers the Iceberg table in a catalog-linked database (e.g. AWS Glue) |
| **Partition By** | Toggle which enables Iceberg table partitioning columns or expressions. |
| **Partition Configuration** | Define partitions using a **column** or a **transform expression** (e.g., `TRUNCATE(10, n_name)`, `BUCKET(6, n_nationkey)`). Partitions are applied in the order they are listed. Enabled only when partitioning is toggled on. |
| **Target File Size** | Specifies a target Parquet file size for the table. |
| **Replace Invalid Characters** | Specifies whether to replace invalid UTF-8 characters with the Unicode replacement character (�) in query results. |
| **Auto Refresh Metadata** | Table will use automated metadata refreshes if set to TRUE |

### External Iceberg REST Table Sync Mode Options

| **Options** | **Description** |
|-------------|-----------------|
| **Create As** | Create as - iceberg table |
| **Type of Catalog** | Specify the type of catalog:- AWS Glue |
| **Sync Mode** | Links to an existing Glue-managed Iceberg table and writes to it without recreating it. |
| **External Volume** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in Snowflake as a prerequisite. |
| **REST Catalog** | Specifies the identifier (name) of the [REST Catalog Integration - Glue](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-glue) for this table. |
| **Catalog Namespace** | Optionally specifies the namespace (for example, `my_glue_database`) for the AWS Glue Data Catalog source. Option available if AWS Glue catalog is chosen. |
| **Catalog Table Name** | Name of the catalog table. Option available if AWS Glue catalog is chosen. |
| **Replace Invalid Characters** | Specifies whether to replace invalid UTF-8 characters with the Unicode replacement character (�) in query results. |
| **Auto Refresh Metadata** | Table will use automated metadata refreshes if set to TRUE |

### External Iceberg REST Table Load Options

| **Setting** | **Description** |
|---------|-------------|
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Truncate Before** | Toggle: True/FalseDetermines whether the table is truncated before execution.**True**: Table is truncated before the DML operation runs.**False**: The DML operation runs without truncation. |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing.**False**: Group by All is visible. |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible, data grouped by all columns**False**: DISTINCT is visible |
| **Order By** | Toggle: True/False**True**: Sort column and sort order drop down are visible and are required to form order by clause. **False**: Sort options invisible |
| **Schedule Run** | *Available when **Sync Mode** is ON.*True or False toggle that determines whether a task will be created or if the SQL to be used in the task will execute DML as a Run action. Prior to creating a task, it is helpful to test the SQL the task will execute to make sure it runs without errors and returns the expected data.- **False** - A table will be created and SQL will execute as a Run action.- **True** - After sufficiently testing the SQL as a Run action, setting Schedule refresh Mode to true will wrap the SQL statement in a task with options specified in Scheduling Options.When Run is executed, a message appears prompting the user to wait for the load to occur. |

### External Iceberg REST Table Task Scheduling Options

If schedule refresh mode is set to true then Task Scheduling Options can be used to configure how and when the task will run.

| **Options** | **Description** |
|-------------|------------------------|
| **Scheduling Mode** | **Warehouse Task** - User managed warehouse will execute tasks**Serverless Task** - Utilize serverless compute to execute tasks |
| **Run Task Warehouse** | Enter the name of the warehouse you want the task to run on without quotes*(Visible if Scheduling Mode is set to Warehouse Task)* |
| **Select initial serverless Warehouse size** | Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times*(Visible when Scheduling Mode is set to Serverless Task)* |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | **Minutes** - Allows you to specify a minute interval for running task**Cron** - Allows you to specify a CRON schedule for running task**Predecessor** - Allows you to specify a predecessor task to determine when a task should execute |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 3.1.0 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format.*|
| **Enter task schedule using Cron** | Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax*(Only visible when Task Schedule is set to Cron)* |
| **Enter predecessor tasks separated by a comma** | One or more task names that precede the task being created in the current node. Task names are case sensitive and should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor tasks separate the task names using a comma and no spaces*(Only visible when Task Schedule is set to Predecessor)* |
| **Enter root task name** | Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor tasks separate the task names using a comma and no spaces |

### External Iceberg REST Table Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

### External Iceberg Table Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

### External Iceberg REST Table Notes

* **Resync Columns** button is applicable only for **Sync Mode**
* **Resync Columns** refreshes column metadata from Glue and may cause any existing transforms to disappear. It is recommended to define all transforms after the table is created and resync is done.
* **Task scheduling** is currently possible only for **Sync mode** and not during Create mode using catalog-linked databases.
* CREATE ICEBERG TABLE **with DEFAULT** is not supported in Catalog-Linked Databases.
* **Table description** is stored in Snowflake only and is not propagated to the Glue catalog. **Column descriptions**, however, are passed through to the Glue-linked Iceberg table and will be visible in AWS Glue Data Catalog.
* Partition info is Iceberg-native and stored in S3 metadata, so the Glue console won't display it like a regular partition column. 
* Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### External Iceberg REST Table With Task Deployment Parameters

### Task Deployment Parameters

All parameters default to `DEV ENVIRONMENT`, which uses the value set in **Config Options**. Set to any other value to override per environment.

| Parameter | Description |
|---|---|
| `targetTaskWarehouse` | Warehouse used to run the task |
| `targetTaskExecuteAsUser` | Snowflake user the task executes as |
| `targetTaskErrorIntegration` | SNS integration for error notifications |
| `targetTaskSuccessIntegration` | SNS integration for success notifications |
| `targetExternalVolume` | External volume used for the Iceberg table |
| `targetRESTCatalogIntegration` | REST Catalog integration used for the Iceberg table |
| `targetCatalogTableName` | The actual table name registered in AWS Glue catalog |
| `targetCatalogNamespace` | The Glue database/namespace where the Iceberg table lives |

---

**Example — DEV ENVIRONMENT (uses Task Scheduling Options values)**
```json
{
  "targetTaskWarehouse": "DEV ENVIRONMENT",
  "targetTaskExecuteAsUser": "DEV ENVIRONMENT",
  "targetTaskErrorIntegration": "DEV ENVIRONMENT",
  "targetTaskSuccessIntegration": "DEV ENVIRONMENT",
  "targetExternalVolume": "DEV ENVIRONMENT",
  "targetRESTCatalogIntegration": "DEV ENVIRONMENT",
  "targetCatalogTableName": "DEV ENVIRONMENT",
  "targetCatalogNamespace": "DEV ENVIRONMENT"
}
```

**Example — Override for QA/Production environment**
```json
{
  "targetTaskWarehouse": "compute_wh",
  "targetTaskExecuteAsUser": "qa_user",
  "targetTaskErrorIntegration": "my_error_integration",
  "targetTaskSuccessIntegration": "my_success_integration",
  "targetExternalVolume": "my_external_volume",
  "targetRESTCatalogIntegration": "my_rest_catalog_integration",
  "targetCatalogTableName": "my_catalog_table_name",
  "targetCatalogNamespace": "my_catalog_namespace"
}
```

### External Iceberg REST Table Initial Deployment

When deployed for the first time into an environment the External Iceberg Table node will execute three stages dependent on whether or not the task schedule relies on a predecessor task.

**External Iceberg Table No Predecessor Task Deployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |
| **Resume Task** | After the task has been created it needs to be resume so that the task runs on the schedule. |

**External Iceberg Table Predecessor Task Deployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |
| **Suspend Root Task** | To add a task into a DAG of task the root task needs to be put into a suspended state. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |

If a task is part of a DAG of tasks, the DAG needs to include a node type called **Task Dag Resume Root**. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task enabled nodes are CREATE OR REPLACE only though this is subject to change.

### External Iceberg Table Redeployment

If any changes in config options like external volume, base location, node properties, or column results in recreating the Iceberg table during redeployment.

#### Recreating the External Snowflake Iceberg Table

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment. |

#### External Iceberg Table Recreating the Task Redeployment

After the Task has deployed for the first time into a target environment, subsequent deployments with changes in task schedule, warehouse, or scheduling options will result in a `CREATE TASK`AND `RESUME TASK` statements being issued.

The following stages are executed.

**External Iceberg Table No Predecessor Task Redeployment**

| **Stage** | **Description** |
|----------|---------------|
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |
| **Resume Task** | After the task has been created it needs to be resume so that the task runs on the schedule. |

**External Iceberg Table Predecessor Task Redeployment**

| **Stage** | **Description** |
|----------|---------------|
| **Suspend Root Task** | To add a task into a DAG of task the root task needs to be put into a suspended state. |
| **Create Task** | This stage will create a task that will load the target table on the schedule specified. |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### External Iceberg Table Undeployment

If a Snowflake iceberg table with task is dropped from the workspace and commited to Git results in table and task dropped from target environment.

The following stages are executed.

**External Iceberg Table No Predecessor Task Dropped**

| **Stage** | **Description** |
|----------|---------------|
| **Drop Iceberg Table** | Drop Iceberg Table |
| **Drop Current Task** | This stage will drop the task. |

**External Iceberg table Predecessor Task Dropped**

| **Stage** | **Description** |
|----------|---------------|
| **Drop Iceberg Table** | Drop Iceberg Table |
| **Suspend Root Task** | To drop a task from a DAG of task the root task needs to be put into a suspended state. |
| **Drop Task** | This stage will drop the task. |

---

## Copy-Into Iceberg Table

Copy-Into Iceberg table node creates an Iceberg table where data is loaded from files in external stage using Copy-Into.

An Iceberg table uses the Apache Iceberg open table format specification, which provides an abstraction layer on data files stored in open formats. [Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg) for Snowflake combine the performance and query semantics of regular Snowflake tables with external cloud storage that you manage. They are ideal for existing data lakes that you cannot, or choose not to, store in Snowflake.

### Keep In Mind When Using Copy-Into Iceberg Table

* The Role in the Workspace and Environment properties of Coalesce should be `ACCOUNTADMIN` in order to successfully create an Iceberg table. You can also grant `SYSADMIN` roles to `EXTERNAL VOLUME`, `CATALOG INTEGRATION` created.
* An `EXTERNAL VOLUME`, `CATALOG INTEGRATION` is expected to be created in Snowflake at the Storage Location chosen in the Node properties.
* In case of creating a Snowflake Iceberg table with structured column types like `OBJECT`, `MAP` or `ARRAY`, ensure the data type is updated with the appropriate structure. For example, if the source Snowflake table has OBJECT data type, then the data type of the same column in the Iceberg table node added on top is expected to be structured type OBJECT (age string, id number) based on the data it has.
* CopyInto node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into a single string column in the target table. Ensure to change the data type of the column to structured object to derive the columns in further steps.
* CopyInto node can be added on top of an inferred table (table created by running the inferschema node) if you want to load data into specific columns as defined in the files. Refer to [Inferschema](https://github.com/coalesceio/External-Data-Package/blob/main/README.md#inferschema) to know more on how to use the node and add Copy-Into on top of it.
* The data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config or reload parameter.

### Copy-Into Iceberg Table Configuration

* [Copy-Into Iceberg Table Node Properties](#copy-into-iceberg-table-node-properties)
* [Copy-Into Iceberg Table Iceberg Options](#copy-into-iceberg-table-iceberg-options)
* [Copy-Into Iceberg Table Source Data](#copy-into-iceberg-table-source-data)
* [Copy-Into Iceberg - File Format](#copy-into-iceberg---file-format)
* [Copy-Into Iceberg Copy Options](#copy-into-iceberg-copy-options)
* [Copy-Into Iceberg System Columns](#copy-into-iceberg-system-columns)

#### Copy-Into Iceberg Table Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Copy-Into Iceberg Table Iceberg Options

| **Option** | **Description** |
|------------|-----------------|
| **Type of Catalog** | Specify the type of catalog: - **Snowflake**- **Polaris** |
| **Snowflake EXTERNAL VOLUME Name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in Snowflake as a prerequisite. |
| **Catalog Integration** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. This option is enabled if the type of catalog is Polaris. |
| **Base Location Name** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. |
| **TruncateBefore** | True / False toggle that determines whether or not a table is to be truncated before reloading:- **True** - Table is truncated and Copy-Into statement is executed to reload the data into target table- **False** - Data is loaded directly into target table and no truncate action takes place |
| **Cluster Key** | Cluster key toggle while Enabled allows us to create subset of columns in a table that are explicitly designated to co-locate the data in the same micro-partitions |
| **Allow Expressions in Cluster Key** | Toggle while Enabled allows you to write expressions |

#### Copy-Into Iceberg Table Source Data

* **Internal or External Stage**
  * **Coalesce Storage Location of Stage**: A storage location in Coalesce where the stage is located.
  * **Stage Name (Required)**: Internal or External stage where the files containing data to be loaded are staged.
  * **File Names (Optional - Ex:'a.csv','b.csv')**: Specifies a list of one or more file names (separated by commas) to be loaded.
  * **File Pattern (Optional - Ex:'.*hea.*[.]csv')**: A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match.
* **External location**
  * **External URI**: Enter the URI of the External location. This URI can point to either a private or public bucket/container.
  * **Storage Integration**: Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity. This field is **not mandatory** for publicly accessible buckets or when another authentication method (such as presigned URLs or public access) is used.

#### Copy-Into Iceberg - File Format

* **File Format Definition**: File Format Name
* **File Format Name**: Specifies an existing named file format to use for loading data into the table.
* **Coalesce Storage Location of File Format**: Location in Coalesce pointing to the database and schema where the file format resides.
* **File Type**:
  * CSV
  * JSON
  * ORC
  * AVRO
  * PARQUET
  * XML

* **File Format Definition**: File Format Values
* **File Format Values** - Provides file format options for the File Type chosen.
* **File Type**: Each file type has different configurations available.
  * **CSV**
  * **Compression**: String (constant) that specifies the current compression algorithm for the data files to be loaded. Choosing **Blank** will automatically skip this option.
  * **Record delimiter**: Characters that separate records in an input file.
  * **Field delimiter**: One or more single-byte or multibyte characters that separate fields in an input file.
  * **Field optionally enclosed by**: Character used to enclose strings.
  * **Number of header lines to skip**: Number of lines at the start of the file to skip.
  * **Skip blank lines**: Boolean that specifies to skip any blank lines encountered in the data files.
  * **Trim Space**: Boolean that specifies whether to remove white space from fields.
  * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
  * **Date format**: String that defines the format of date values in the data files to be loaded.
  * **Time format**: String that defines the format of time values in the data files to be loaded.
  * **Timestamp format**: String that defines the format of timestamp values in the data files to be loaded.
  * **JSON**
    * **Compression**: String (constant) that specifies the current compression algorithm for the data files to be loaded. Choosing **Blank** will automatically skip this option.
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
    * **Trim Space**: Boolean that specifies whether to remove white space from fields.
    * **Strip Outer Array**: Boolean that instructs the JSON parser to remove outer brackets [ ].
    * **Date format**: String that defines the format of date values in the data files to be loaded.
    * **Time format**: String that defines the format of time values in the data files to be loaded.
    * **Timestamp format**: String that defines the format of timestamp values in the data files to be loaded.
  * **ORC**
    * **Trim Space**: Specifies whether to remove white space from fields.
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
  * **AVRO**
    * **Trim Space**: Boolean that specifies whether to remove white space from fields.
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
  * **PARQUET**
    * **Trim Space**: Boolean that specifies whether to remove white space from fields.
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
  * **XML**
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.

#### Copy-Into Iceberg Copy Options

| **Option** | **Description** |
|------------|-----------------|
| **On Error Behavior** | String (constant) that specifies the error handling for the load operation: - CONTINUE- SKIP_FILE- SKIP_FILE_num- SKIP_FILE_num%- ABORT_STATEMENTChoosing **Blank** will automatically skip this option.|
| **Specify the number of errors that can be skipped** | Required when On Error Behavior is either `SKIP_FILE_num` or `SKIP_FILE_num%`. Specify the number of errors that can be skipped. |
| **Size Limit** | Number (> 0) that specifies the maximum size (in bytes) of data to be loaded for a given COPY statement. |
| **Purge Behavior** | Boolean that specifies whether to remove the data files from the stage automatically after the data is loaded successfully.Choosing **Blank** will automatically skip this option. |
| **Return Failed Only** | Boolean that specifies whether to return only files that have failed to load in the statement result. |
| **Force** | Boolean that specifies to load all files, regardless of whether they've been loaded previously and have not changed since they were loaded.Choosing **Blank** will automatically skip this option. |
| **Load Uncertain Files** | Boolean that specifies to load files for which the load status is unknown. The COPY command skips these files by default.Choosing **Blank** will automatically skip this option. |
| **Enforce Length** | Boolean that specifies whether to truncate text strings that exceed the target column length.Choosing **Blank** will automatically skip this option. |
| **Truncate Columns** | Boolean that specifies whether to truncate text strings that exceed the target column length.Choosing **Blank** will automatically skip this option. |

#### Copy-Into Iceberg System Columns

The set of columns which has source data and file metadata information.

* **SRC** - The data from the file is loaded into this variant column.
* **LOAD_TIMESTAMP** - Current timestamp when the file gets loaded.
* **FILENAME** - Name of the staged data file the current row belongs to. Includes the full path to the data file.
* **FILE_ROW_NUMBER** - Row number for each record in the staged data file.
* **FILE_LAST_MODIFIED** - Last modified timestamp of the staged data file the current row belongs to
* **SCAN_TIME** - Start timestamp of operation for each record in the staged data file. Returned as TIMESTAMP_LTZ.

### Copy-Into Iceberg Deployment

#### Copy-Into Iceberg Deployment Parameters

The Copy-Into Iceberg includes an environment parameter that allows you to specify if you want to perform a full load or a reload based on the load type when you are performing a copy-into-iceberg operation.
Alternatively, the data can be reloaded into the table by truncating the data in the table before load using the TruncateBefore option in node config.

The parameter name is `loadType` and the default value is ``.

MDXSAFE_PLACEHOLDER_4_ENDPLACEHOLDER

When the parameter value is set to `Reload`, the data is reloaded into the table regardless of whether they’ve been loaded previously and have not changed since they were loaded.

#### Copy-Into Iceberg Initial Deployment

When deployed for the first time into an environment the copy-into Iceberg node of materialization type table will execute the below stage:

|**Stage** | **Description**|
|---------|---------------|
|**Create table/transient table** | This will execute a CREATE OR REPLACE statement and create a table or transient table in the target environment.|

### Copy-Into Iceberg Redeployment

#### Altering the Copy-Into Iceberg Tables

These changes can be made either in isolation or all together, and will result in a Create statement to modify the Work Table in the target environment.

• Change in table name
• Dropping existing column
• Alter Column data type
• Adding a new column

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg table** | Create table statement is executed to perform the alter operation. |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Copy-Into Iceberg Undeployment

If the Copy-Into Iceberg node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

| **Stage** | **Description** |
|----------|---------------|
| **Drop table/transient table** | Target table in Snowflake is dropped |

---

## Snowpipe Iceberg Table

The Coalesce Snowpipe Iceberg node is a node that creates an Iceberg table and performs two operations. It can be used to load historical data using [CopyInto](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table). Also, the Snowpipe node can be used to create a pipe to automatically ingest files from AWS, GCP, or Azure.

[Snowpipe](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro) enables loading data from files as soon as they're available in a stage.

This means you can load data from files in micro-batches, making it available to users within minutes, rather than manually executing COPY statements on a schedule to load larger batches.

### Snowpipe Iceberg Prerequisites

* The Role in the Workspace and Environment properties of Coalesce should be `ACCOUNTADMIN` inorder to successfully create an  Iceberg table. You can also grant `SYSADMIN` roles to `EXTERNAL VOLUME`, `CATALOG INTEGRATION`created.
* An `EXTERNAL VOLUME`, `CATALOG INTEGRATION` is expected to be created in Snowflake at the Storage Location chosen in the Node properties.
* In case of creating a Snowflake Iceberg table with structured column type like `OBJECT`, `MAP` or `ARRAY`. Ensure the data type is updated with the appropriate structure. For example,the source Snowflake table has OBJECT data type,then the data type of the same column in the Iceberg table node added on top is expected to structured type OBJECT (age string,id number) based on the data it has.
* Snowpipe node can be created by just clicking on Create node from browser if we want the data from the file to be loaded into single string column in target table.Ensure to change the data type of the column to structured object to derive the columns in further steps.
* Snowpipe node can be added on top of an inferred table(table created by running the inferschema node) if you want to load data into specific columns as defined in the files.Refer to [Inferschema](https://github.com/coalesceio/External-Data-Package/blob/main/README.md#inferschema) to know more on how to use the node and add Copy-Into on top of it.

### Snowpipe Iceberg Node Configuration

* [Snowpipe Iceberg Node Properties](#snowpipe-iceberg-node-properties)
* [Snowpipe Iceberg Options](#snowpipe-iceberg-options)
* [Snowpipe Iceberg Table Iceberg Options](#snowpipe-iceberg-table-iceberg-options)
* [Snowpipe Iceberg File Location](#snowpipe-iceberg-file-location)
* [Snowpipe Iceberg File Format](#snowpipe-iceberg-file-format)
* [Snowpipe Iceberg Copy Options](#snowpipe-iceberg-copy-options)
* [Snowpipe Iceberg System Columns](#snowpipe-iceberg-system-columns)

#### Snowpipe Iceberg Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Snowpipe Iceberg Options

| **Stage** | **Description** |
|----------|---------------|
| **Enable Snowpipe** | Drop down that helps us to create a pipe to auto ingest files from external stage or validate the Copy-Into statement.When Run is executed, with Enable Snowpipe selection, a message appears suggesting the user it is a snowpipe and no test run possible. |
| **Cloud Provider Options:** | **AWS** - AWS SNS Topic. Specifies the Amazon Resource Name (ARN) for the SNS topic for your S3 bucket**Azure** - Integration. Specifies the existing notification integration used to access the storage queue**GCP** - Integration. Specifies the existing notification integration used to access the storage queue |
| **Test Copy Statement** | To validate the Copy-into statement before we use it to create PIPE |
| **Load historical data** | Loads the historic data into the target table by executing a COPY_INTO statement |

#### Snowpipe Iceberg Table Iceberg Options

| **Options** | **Description** |
|------------|-----------------|
| **Type of catalog** | Specify the type of catalog: - **Snowflake**- **Polaris** |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in Snowflake as a prerequisite. |
| **Catalog integration** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. This option is enabled if the type of catalog is Polaris. |
| **Base location name** | Specifies the identifier (name) of the [catalog integration](https://docs.snowflake.com/user-guide/tables-iceberg#label-tables-iceberg-catalog-integration-def) for this table. |
| **Cluster Key** | Cluster key Toggle while Enabled allows us to create subset of columns in a table that are explicitly designated to co-locate the data in the table in the same micro-partitions. |
| **Allow Expressions in Cluster Key** | Toggle while Enabled allows you to write expressions. |

#### Snowpipe Iceberg File Location

| **Options** | **Description** |
|------------|-----------------|
| **Coalesce Stage Storage Location of stage (Required)** | A storage location in Coalesce where the stage is located. |
| **Stage Name (Required)** | Internal or External stage where the files containing data to be loaded are staged. |
| **File Names** | Enabled when 'Enable Snowpipe' under Snowpipe Options is toggled off. Specifies a list of one or more files names (separated by commas) to be loaded. For example, `'a.csv','b.csv'`. |
| **File Pattern** | A regular expression pattern string, enclosed in single quotes, specifying the file names or paths to match. For example, `*hea.*[.]csv'`. |

#### Snowpipe Iceberg File Format

* **File Format Definition**: File Format Name
  * **File Format Name**: Specifies an existing named file format to use for loading data into the table.
  * **Coalesce Storage Location of File Format**: File format location in snowflake that is mapped as storage location in Coalesce
  * **File Type**:
    * CSV
    * JSON
    * ORC
    * AVRO
    * PARQUET
    * XML

* **File Format Definition**: File Format Values
  * **File Format Values** -Provides file format options for the File Type chosen.
  * **File Type**: Each file type has different configurations available.
    * **CSV**
    * **Compression**: String (constant) that specifies the current compression algorithm for the data files to be loaded.Choosing **Blank** will automatically skip this option.
    * **Record delimiter**:Characters that separate records in an input file
    * **Field delimiter**:One or more singlebyte or multibyte characters that separate fields in an input file
    * **Field optionally enclosed by**:Character used to enclose strings
    * **Number of header lines to skip**:Number of lines at the start of the file to skip.
    * **Skip blank lines**:Boolean that specifies to skip any blank lines encountered in the data files.
    * **Trim Space**: Boolean that specifies whether to remove white space from fields.
    * **Replace invalid characters**: Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
    * **Date format**:String that defines the format of date values in the data files to be loaded.
    * **Time format**: String that defines the format of time values in the data files to be loaded
    * **Timestamp format**:String that defines the format of timestamp values in the data files to be loaded.
    * **JSON**
      * **Compression**: String (constant) that specifies the current compression algorithm for the data files to be loaded.Choosing **Blank** will automatically skip this option.
      * **Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
      * **Trim Space** - Boolean that specifies whether to remove white space from fields.
      * **Strip Outer Array**:Boolean that instructs the JSON parser to remove outer brackets [ ].
      * **Date format**:String that defines the format of date values in the data files to be loaded.
      * **Time format**:String that defines the format of time values in the data files to be loaded
      * **Timestamp format**: String that defines the format of timestamp values in the data files to be loaded.
    * **ORC**
      * **Trim Space** - Specifies whether to remove white space from fields
      * **Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
    * **AVRO**
      * **Trim Space** - Boolean that specifies whether to remove white space from fields.
      * **Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
    * **PARQUET**
      * **Trim Space** - Boolean that specifies whether to remove white space from fields.
      * **Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.
    * **XML**
      * **Replace invalid characters** - Boolean that specifies whether to replace invalid UTF-8 characters with the Unicode replacement character.

#### Snowpipe Iceberg Copy Options

| **Options** | **Description** |
|------------|-----------------|
| **On Error Behavior** | String (constant) that specifies the error handling for the load operation: - CONTINUE- SKIP_FILE- SKIP_FILE_num&nbsp;&nbsp;&nbsp;&nbsp;- Specify the number of errors that can be skipped.- SKIP_FILE_num%&nbsp;&nbsp;&nbsp;&nbsp;- Specify the number of errors that can be skipped.Choosing **Blank** will automatically skip this option. |
| **Enforce Length** | Boolean that specifies whether to truncate text strings that exceed the target column length.Choosing **Blank** will automatically skip this option. |
| **Truncate Columns** | Boolean that specifies whether to truncate text strings that exceed the target column length.Choosing **Blank** will automatically skip this option. |

#### Snowpipe Iceberg System Columns

The set of columns which has source data and file metadata information.

| **Options** | **Description** |
|------------|-----------------|
| **SRC** | The data from the file is loaded into this string column. |
| **LOAD_TIMESTAMP** | Current timestamp when the file gets loaded. |
| **FILENAME** | Name of the staged data file the current row belongs to. Includes the full path to the data file. |
| **FILE_ROW_NUMBER** | Row number for each record in the staged data file. |
| **FILE_LAST_MODIFIED** | Last modified timestamp of the staged data file the current row belongs to. |
| **SCAN_TIME** | Start timestamp of operation for each record in the staged data file. Returned as TIMESTAMP_LTZ. |

### Snowpipe Iceberg Deployment

#### Snowpipe Iceberg Deployment Parameters

The Snowpipe includes two environment parameters.

The parameter `loadType` allows you to specify if you want to perform a full load or a reload based on the load type when you are performing a Copy-Into operation.

The other parameter `targetIntegrationOrAwsSNSTopic` allows you to specify the AWS SNS Topic or Integration name to be used for auto ingesting files from Cloud providers.

The parameter name is `loadType` and the default value is ``.

MDXSAFE_PLACEHOLDER_5_ENDPLACEHOLDER

When the parameter value is set to `Reload`, the data is reloaded into the table regardless of whether they’ve been loaded previously
and have not changed since they were loaded.

The parameter name is `targetIntegrationOrAwsSNSTopic` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT` the value entered in the Snowpipe Options config AWS SNS Topic or Integration is used for ingesting files from Cloud providers.

MDXSAFE_PLACEHOLDER_6_ENDPLACEHOLDER

When set to any value other than `DEV ENVIRONMENT` the node will use the specified value for AWS SNS Topic/Integration.

For example, the Snowpipe node will use the specific value for auto ingestion.

MDXSAFE_PLACEHOLDER_7_ENDPLACEHOLDER

#### Snowpipe Iceberg Initial Deployment

When deployed for the first time into an environment the Snowpipe node will execute the below stages depending on if Enable Snowpipe is enabled,`Load Historical Data` is enabled and the `loadType` parameter.

| **Deployment Behavior** | **Enable Snowpipe** | **Historical Load** | **Load Type** | **Stages Executed** |
|------------------------|---------------------|-------------------|--------------|-------------------|
| Initial Deployment | Enable Snowpipe | true | `` | - Create Iceberg Table- Historical full load using CopyInto- Create Pipe- Alter Pipe |
| Initial Deployment | Enable Snowpipe | true | Reload | - Create Iceberg Table- Truncate Target Table- Historical full load using CopyInto- Create Pipe- Alter Pipe |
| Initial Deployment | Enable Snowpipe | false | Reload or Empty | - Create Iceberg Table- Truncate Target Table- Create Pipe |
| Initial Deployment | Test Copy Statement | false | Reload or Empty | - Create Iceberg Table- Test Copy Statement - No pipe creation |

### Snowpipe Iceberg Redeployment

#### Altering the Snowpipe Iceberg Node

Changes that result in a CREATE statement to modify the target table, whether made in isolation or altogether, in the target environment include:

* Change in table name
* Dropping existing column
* Altering column data type
* Adding a new column

Any table-level changes or node configuration changes result in recreation of the pipe.

The following stages are executed:

| **Stage** | **Description** |
|----------|---------------|
| **Create Iceberg Table** | Create table statement is executed to perform the alter operation. |
| **Truncate Target Table** | The target table is truncated in case the Load Type parameter is set to 'Reload'. |
| **Historical Full Load Using CopyInto** | Historical data are loaded if 'Load Historical' toggle is on. |
| **Create Pipe** | Pipe is recreated if Enable Snowpipe option is true. |
| **Alter Pipe** | Pipe is refreshed if 'Load Historical' toggle is on. |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Snowpipe Iceberg Undeployment

If the Snowpipe node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the target table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Target table is dropped |
| **Drop Pipe** | Pipe is dropped |

## Dynamic Iceberg Work Table

[**Dynamic Iceberg tables**](https://docs.snowflake.com/en/user-guide/dynamic-tables-create-iceberg) combine the benefits of dynamic tables and Snowflake-managed Iceberg tables, offering features like external cloud storage management, automated data transformation, and performance optimization.

Dynamic Iceberg tables integrate with data lakes, which let you store data in external cloud storage such as AWS S3 or Azure Blob Storage while being managed by Snowflake. These tables support ACID transactions, schema evolution, hidden partitioning, and table snapshots. Tasks. 

### Dynamic Iceberg Work Node Prerequisites

* The Role mentioned in the Workspace and Environment properties of Coalesce should be ACCOUNTADMIN in order to successfully create an Iceberg table.
* An EXTERNAL VOLUME is expected to be created in Snowflake at the Storage Location chosen in the Node properties.
* In case of creating a Snowflake Iceberg table with structured column types like OBJECT, MAP, or ARRAY, ensure the data type is updated with the appropriate structure. For example, if the source Snowflake table has OBJECT data type, then the data type of the same column in the Iceberg table node added on top is expected to be structured type OBJECT (age string, id number) based on the data it has.

### Dynamic Iceberg Work Node Configuration

The Dynamic Iceberg table has three configuration groups:

* [Dynamic Iceberg Node Properties](#Dynamic-iceberg-work-node-properties)
* [Dynamic Iceberg Options](#Dynamic-iceberg-options)
* [General Options](#general-options)

#### Dynamic Iceberg Work Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Dynamic Iceberg Options

| **Option** | **Description** |
|--------|-------------|
| **Warehouse** | (Required) Name of warehouse used to refresh the Dynamic Table |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in snowflake as a prerequisite |
| **Base location name** | The path to a directory where Snowflake can write data and metadata files for the table. Specify a relative path from the table's EXTERNAL_VOLUME location |
| **[Target Lag Specification](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh)** | - **Time Based** - **Downstream**Set Time Based refresh schedule with:- **Time Value**: Frequency of refresh for a given Time Period.- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh_Mode** | - **Blank**: If chosen, **INCREMENTAL** is added- **AUTO**: If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | - **Blank**: **ON_CREATE** is considered at creation- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |
| **Copy Grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created. Useful during [replication](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Cluster key** | True/False to determine whether the Iceberg table is to be clustered or not***True** - Allows you to specify the column based on which clustering is to be done.* **Allow Expressions Cluster Key** - True allows to add an expression to the specified cluster key* **False** - No clustering done |

#### General Options

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True/False toggle to return DISTINCT rows |
| **Group By All** | True/False toggle to add non-aggregated columns to GROUP BY |
| **Multi Source** | True/False toggle for combining multiple sources via UNION or UNION ALL |

### Dynamic Iceberg Work Deployment

#### Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

MDXSAFE_PLACEHOLDER_8_ENDPLACEHOLDER

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

MDXSAFE_PLACEHOLDER_9_ENDPLACEHOLDER
> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters

#### Dynamic Iceberg Work Initial Deployment

When deployed for the first time into an environment the node will execute the following stage:

| **Stage** | **Description** |
|-------|-------------|
| **Create Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Dynamic Iceberg Table in the target environment |

#### Dynamic Iceberg Work Redeployment

Any changes in config options will result in recreating the table during redeployment.

* `EXTERNAL VOLUME`
* `Base location name`
* `Node Properties`
* `Column Name`
* `Join`
* `Transform`

| **Stage** | **Description** |
|-------|-------------|
| **Create or Replace Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment |

#### Altering the Dynamic Iceberg Table Work

The following config changes trigger ALTER statements:

1. Warehouse name
2. Target Lag
3. Time based Lag specification
4. Cluster Key
5. Table description
6. Column description
7. Node rename
8. Storage location change

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

## Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dynamic Iceberg Work Undeployment

If a Dynamic Iceberg Work table is dropped from the workspace and commited to Git results in table dropped from target environment.

| **Stage** | **Description** |
|-------|-------------|
| **Drop Dynamic Table** | Removes the Iceberg table from the target environment |

## Dynamic Iceberg Dimension Table

[**Dynamic Iceberg tables**](https://docs.snowflake.com/en/user-guide/dynamic-tables-create-iceberg) combine the benefits of dynamic tables and Snowflake-managed Iceberg tables, offering features like external cloud storage management, automated data transformation, and performance optimization.

Dynamic Iceberg tables integrate with data lakes, which let you store data in external cloud storage such as AWS S3 or Azure Blob Storage while being managed by Snowflake. These tables support ACID transactions, schema evolution, hidden partitioning, and table snapshots. Tasks. 

### Dynamic Iceberg Dimension Node Prerequisites

* The Role mentioned in the Workspace and Environment properties of Coalesce should be ACCOUNTADMIN in order to successfully create an Iceberg table.
* An EXTERNAL VOLUME is expected to be created in Snowflake at the Storage Location chosen in the Node properties.

### Dynamic Iceberg Dimension Node Configuration

The Dynamic Iceberg table has three configuration groups:

* [Dynamic Iceberg Node Properties](#Dynamic-iceberg-dimension-node-properties)
* [Dynamic Iceberg Options](#Dynamic-iceberg-options-1)
* [Dimension Options](#dimension-options)

#### Dynamic Iceberg Dimension Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Dynamic Iceberg Options

| **Option** | **Description** |
|--------|-------------|
| **Warehouse** | (Required) Name of warehouse used to refresh the Dynamic Table |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in snowflake as a prerequisite |
| **Base location name** | The path to a directory where Snowflake can write data and metadata files for the table. Specify a relative path from the table's EXTERNAL_VOLUME location |
| **[Target Lag Specification](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh)** | - **Time Based** - **Downstream**Set Time Based refresh schedule with:- **Time Value**: Frequency of refresh for a given Time Period.- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh_Mode** | - **Blank**: If chosen, **INCREMENTAL** is added- **AUTO**: If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | - **Blank**: **ON_CREATE** is considered at creation- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |
| **Copy Grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created. Useful during [replication](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Cluster key** | True/False to determine whether the Iceberg table is to be clustered or not***True** - Allows you to specify the column based on which clustering is to be done.* **Allow Expressions Cluster Key** - True allows to add an expression to the specified cluster key* **False** - No clustering done |

#### Dimension Options

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | (Required) Business key columns for Dimension key formation |
| **Record versioning** | (Required) Type of column for history maintenance:- **Datetime column**- **Date and Time column**- **Integer column** |
| **Timestamp** | Required if Datetime column chosen for Record versioning.**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |
| **Sequence** | Required if Integer column chosen for Record versioning|
| **Timetamp-track data load**| Required if Integer column chosen for Record versioning**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |
| **Date/Timestamp Columns** | Required if Date and Time columns chosen for Record versioning**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |

### Dynamic Iceberg Dimension Deployment

#### Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

MDXSAFE_PLACEHOLDER_10_ENDPLACEHOLDER

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

MDXSAFE_PLACEHOLDER_11_ENDPLACEHOLDER
> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters

#### Dynamic Iceberg Dimension Initial Deployment

When deployed for the first time into an environment the node will execute the following stage:

| **Stage** | **Description** |
|-------|-------------|
| **Create Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Dynamic Iceberg Table in the target environment |

#### Dynamic Iceberg Dimension Redeployment

Any changes in config options will result in recreating the table during redeployment.

* `EXTERNAL VOLUME`
* `Base location name`
* `Node Properties`
* `Column Name`
* `Join`
* `Transform`

| **Stage** | **Description** |
|-------|-------------|
| **Create or Replace Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment |

#### Altering the Dynamic Iceberg Table Dimension

The following config changes trigger ALTER statements:

1. Warehouse name
2. Target Lag
3. Time based Lag specification
4. Cluster Key
5. Table description
6. Column description
7. Node rename
8. Storage location change

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

#### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

## Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dynamic Iceberg Dimension Undeployment

If a Dynamic Iceberg Dimension table is dropped from the workspace and commited to Git results in table dropped from target environment.

| **Stage** | **Description** |
|-------|-------------|
| **Drop Dynamic Table** | Removes the Iceberg table from the target environment |

## Dynamic Iceberg Latest Record Version Table

[**Dynamic Iceberg tables**](https://docs.snowflake.com/en/user-guide/dynamic-tables-create-iceberg) combine the benefits of dynamic tables and Snowflake-managed Iceberg tables, offering features like external cloud storage management, automated data transformation, and performance optimization.

Dynamic Iceberg tables integrate with data lakes, which let you store data in external cloud storage such as AWS S3 or Azure Blob Storage while being managed by Snowflake. These tables support ACID transactions, schema evolution, hidden partitioning, and table snapshots. Tasks. 

### Dynamic Iceberg Latest Record Version Node Prerequisites

* The Role mentioned in the Workspace and Environment properties of Coalesce should be ACCOUNTADMIN in order to successfully create an Iceberg table.
* An EXTERNAL VOLUME is expected to be created in Snowflake at the Storage Location chosen in the Node properties.

### Dynamic Iceberg Latest Record Version Node Configuration

The Dynamic Iceberg table has three configuration groups:

* [Dynamic Iceberg Node Properties](#Dynamic-iceberg-node-properties)
* [Dynamic Iceberg Options](#Dynamic-iceberg-options-2)
* [Latest Record Version Options](#Latest-Record-Version-options)

#### Dynamic Iceberg Node Properties

| **Property** | **Description** |
|----------|-------------|
| **Storage Location (required)** | Storage Location where the Iceberg Table will be created |
| **Node Type (required)** | Name of template used to create node objects |
| **Deploy Enabled (required)** | If TRUE the node will be deployed or redeployed when changes are detected.If FALSE the node will not be deployed or the node will be dropped during redeployment |

#### Dynamic Iceberg Options

| **Option** | **Description** |
|--------|-------------|
| **Warehouse** | (Required) Name of warehouse used to refresh the Dynamic Table |
| **Snowflake EXTERNAL VOLUME name** | Specifies the identifier (name) for the external volume where the Iceberg table stores its metadata files and data in Parquet format. [External volume](https://docs.snowflake.com/sql-reference/sql/create-external-volume) needs to be created in snowflake as a prerequisite |
| **Base location name** | The path to a directory where Snowflake can write data and metadata files for the table. Specify a relative path from the table's EXTERNAL_VOLUME location |
| **[Target Lag Specification](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh)** | - **Time Based** - **Downstream**Set Time Based refresh schedule with:- **Time Value**: Frequency of refresh for a given Time Period.- **Time Period**: Seconds/Minutes/Hours/Days |
| **Refresh_Mode** | - **Blank**: If chosen, **INCREMENTAL** is added- **AUTO**: If the CREATE DYNAMIC TABLE statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode.- **INCREMENTAL**: Force incremental refresh- **FULL**: Force full refresh |
| **Initialize** | - **Blank**: **ON_CREATE** is considered at creation- **ON_CREATE**: Refresh synchronously at creation- **ON_SCHEDULE**: Refresh at next scheduled time |
| **Copy Grants** | Specifies to retain the access privileges from the original table when a new dynamic table is created. Useful during [replication](https://docs.snowflake.com/en/user-guide/account-replication-considerations#replication-and-dynamic-tables) |
| **Cluster key** | True/False to determine whether the Iceberg table is to be clustered or not***True** - Allows you to specify the column based on which clustering is to be done.* **Allow Expressions Cluster Key** - True allows to add an expression to the specified cluster key* **False** - No clustering done |

#### Latest Record Version Options

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | (Required) Business key columns for Latest Record Version key formation |
| **Record versioning** | (Required) Type of column for history maintenance:- **Datetime column**- **Date and Time column**- **Integer column** |
| **Timestamp** | Required if Datetime column chosen for Record versioning.**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |
| **Sequence** | Required if Integer column chosen for Record versioning|
| **Timetamp-track data load**| Required if Integer column chosen for Record versioning**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |
| **Date/Timestamp Columns** | Required if Date and Time columns chosen for Record versioning**Note**: If multiple columns are chosen.The first timestamp column chosen is considered for versioning order |

### Dynamic Iceberg Latest Record Version Deployment

#### Initial Deployment Parameters

The Dynamic Table Work includes an environment parameter that allows you to specify a different warehouse to refresh a Dynamic Table in different environments.

The parameter name is `targetDynamicTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Dynamic Table Options config "Warehouse on which to execute Dynamic Table" will be used when creating the Dynamic Table.

MDXSAFE_PLACEHOLDER_12_ENDPLACEHOLDER

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Dynamic Table using a Snowflake warehouse with the specified value.

For example, the Dynamic Table will refresh using a warehouse named `compute_wh`.

MDXSAFE_PLACEHOLDER_13_ENDPLACEHOLDER
> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters

#### Dynamic Iceberg Latest Record Version Initial Deployment

When deployed for the first time into an environment the node will execute the following stage:

| **Stage** | **Description** |
|-------|-------------|
| **Create Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Dynamic Iceberg Table in the target environment |

#### Dynamic Iceberg Latest Record Version Redeployment

Any changes in config options will result in recreating the table during redeployment.

* `EXTERNAL VOLUME`
* `Base location name`
* `Node Properties`
* `Column Name`
* `Join`
* `Transform`

| **Stage** | **Description** |
|-------|-------------|
| **Create or Replace Dynamic Iceberg Table** | This stage will execute a `CREATE` OR `REPLACE` statement and create a Iceberg Table in the target environment |

#### Altering the Dynamic Iceberg Table Latest Record Version

The following config changes trigger ALTER statements:

1. Warehouse name
2. Target Lag
3. Time based Lag specification
4. Cluster Key
5. Table description
6. Column description
7. Node rename
8. Storage location change

These execute the two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Dynamic Table** | Executes ALTER to modify parameters |
| **Refresh Dynamic Table** | Refreshes table to make data available |

Other column or table level changes like data type change, column name change, column addition/deletion result in a `CREATE` statement.

#### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

## Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dynamic Iceberg Latest Record Version Undeployment

If a Dynamic Iceberg Latest Record Version table is dropped from the workspace and commited to Git results in table dropped from target environment.

| **Stage** | **Description** |
|-------|-------------|
| **Drop Dynamic Table** | Removes the Iceberg table from the target environment |

-----------------

## Node Type Switching Logic

| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Iceberg Table | Iceberg Table | Follows existing redeployment stages |
| Any Other | Iceberg Table | 1. Warning (if applicable)2. Drop  3. Create |
| Dynamic Iceberg Table | Dynamic Iceberg Table | Follows existing redeployment stages |
| Any Other | Dynamic Iceberg Table | 1. Warning (if applicable)2. Drop  3. Create |

Please review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)(Pending Release) |

--------------

### Code

### Snowflake Iceberg table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/SnowflakeIcebergtable-305/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/SnowflakeIcebergtable-305/create.sql.j2)

### External Iceberg table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/ExternalIcebergtable-307/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/ExternalIcebergtable-307/create.sql.j2)

### External Iceberg REST Table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/ExternalIcebergRESTTable-655/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/ExternalIcebergRESTTable-655/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/ExternalIcebergRESTTable-655/run.sql.j2)

### Copy-Into Iceberg table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/tree/main/nodeTypes/CopyIntoIcebergtable-320)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/tree/main/nodeTypes/CopyIntoIcebergtable-320)

### Snowpipe Iceberg table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/tree/main/nodeTypes/SnowpipeIcebergtable-321)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/tree/main/nodeTypes/SnowpipeIcebergtable-321)

### Dynamic Iceberg Work table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableWork-565/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableWork-565/create.sql.j2)

### Dynamic Iceberg Dimension table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableDimension-566/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableDimension-566/create.sql.j2)

### Dynamic Iceberg Latest Record Version table

* [Node definition](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableLatestRecordVersion-571/definition.yml)
* [Create Template](https://github.com/coalesceio/Iceberg-tables/blob/main/nodeTypes/DynamicIcebergTableLatestRecordVersion-571/create.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 3.3.0 | May 27, 2026 | New node type - External Iceberg REST Table - for create/write to AWS Glue-managed Iceberg tables |
| 3.2.1 | April 14, 2026 | Task Node Enhancements - Notification Integrations, Execute As User, and Advanced Task Parameters |
| 3.1.0 | February 26, 2026 | Node Type Switching supported in Advanced Deploy Templates |
| 3.0.3 | November 18, 2025 | Empty SQL Changes and Macro Dictionary Changes |
| 3.0.2 | October 28, 2025 | Add blank option to all optional dropdownSelector configs |
| 3.0.1 | October 23, 2025 | Conditional STORAGE_INTEGRATION Parameter Generation Changes |
| 3.0.0 | October 06, 2025 | Added 3 new Node types for Dynamic Iceberg Tables |
| 2.0.0 | July 31, 2025 | Deploy phases removed in stage functions |
| 1.1.4 | March 17, 2025 | Fix to force users to enter Parameters When Required |
| 1.1.3 | March 04, 2025 | Redeployment fixes  |
| 1.1.2 | November 28, 2024 | Re-Sync Columns config enabled |
| 1.1.1 | November 08, 2024 | Truncate Option added to Copy Into Iceberg Node type |
| 1.1.0 | September 09, 2024 | Support for Polaris Catalog added.Two new node types Copy-Into Iceberg and Snowpipe Iceberg added |

---

## Incremental Loading(Package)

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/incremental-loading</code> <CopyButton textToCopy='@coalesce/snowflake/incremental-loading' />
Supported Platform: 
Latest Version: 2.1.1 (March 10, 2026)

batch processingchange detectiondelta processingincremental loadingperformance optimizationwatermark

## Overview
Nodes designed for immediate use in incremental data loading.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/incremental-loading</code> <CopyButton textToCopy='@coalesce/snowflake/incremental-loading' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
## Snowflake - Incremental Package - Brief Summary

- **Incremental Load:** Serves as the core engine for efficient data updates by processing only new or modified records rather than entire datasets. By leveraging high-water mark logic and `MERGE` statements, these nodes significantly reduce Snowflake compute costs and processing time.

- **Grouped Incremental Load:** Optimized for handling data that requires aggregation or grouping before being merged into a persistent target. This node ensures that incremental deltas are correctly summarized at the business grain, maintaining data accuracy while processing only the most recent changes.

- **Looped Load:** Designed for handling massive datasets by breaking them into smaller, manageable batches (loops). This prevents memory exhaustion and optimizes performance by processing data in "chunks" based on a specific dimension, such as a date range or ID range.

- **Run View:** Acts as the orchestration metadata layer. It tracks the "state" of the pipeline by identifying the most recent records successfully loaded, providing the necessary reference points to ensure no data is missed or duplicated during subsequent runs.

- **Test Passed Records:** Functions as a data quality gatekeeper. This node filters and surfaces only the records that meet predefined business rules and validation criteria, ensuring that downstream persistent tables remain clean and reliable.

- **Test Failed Records:** Provides a native error-handling and auditing framework. By capturing records that fail validation, these nodes allow for seamless data quality monitoring and remediation without interrupting the flow of the primary data pipeline.

**Summary:** 
Together, these node types provide a robust, automated framework for managing delta-processing within Snowflake. By combining high-performance incremental loading with integrated data quality testing and batch orchestration, this package ensures that your data platform remains scalable, cost-effective, and accurate.

----
## Nodetypes Config Matrix

| Category | Feature | Incremental Load | Test Passed Records | Test Failed Records | Looped Load | Run View | Grouped Incr. Load |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Create** | Create As (Table/View/Transient) | ✅ | ✅ | ✅ | ⬜ | ✅ | ✅ |
| **Create** | Persistent Table Location/Name | ✅ | ⬜ | ⬜ | ⬜ | ✅ | ✅ |
| **Create** | Storage Mapping ID | ⬜ | ⬜ | ⬜ | ⬜ | ⬜  | ✅ |
| **Logic** | Filter based on Persistent Table | ✅ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ |
| **Logic** | Incremental Load Column (Date) | ✅ | ⬜ | ⬜ | ✅ | ✅ | ✅ |
| **Logic** | Batch / Group / Loop Loading | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ✅ |
| **Logic** | Multi-Source / Source Filtering | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ |
| **Tests** | Enable Tests / Handle DQ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ |
| **Tests** | Nullability / Duplicate Checks | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ |
| **Tests** | First/Last Occurrence Logic | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ |
| **SQL** | Pre-SQL / Post-SQL | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ |
| **SQL** | Override Create SQL | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ |
| **Metadata**| Dedicated Load History Table | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ |
| **Metadata**| Joining Table Column Name | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ✅ |

---

The Coalesce Incremental Package includes:

* [Incremental load](#incremental-load)
* [Test Passed Records](#test-passed-records)
* [Test Failed Records](#test-failed-records)
* [Looped Load](#loop-load)
* [Run view](#run-view)
* [Grouped Incremental load](#grouped-incremental-load)
* [Code](#code)

## Incremental Load

The Coalesce Incremental load node is a versatile node that allows you to develop and deploy a Stage table/view in Snowflake where we can perform incremental load in comparison with a persistent table added on top of it.

### Incremental Load Node Configuration

* [Node Properties](#incremental-load-node-properties)
* [Options](#incremental-load-options)

#### Incremental Load Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Incremental node will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Incremental Load Options

| **Options** | **Description** |
|-----------|-----------------|
| **Create As** | Provides option to choose materialization type as `table` or `view`. |
| **Filter data based on Persistent table** | - True - provides option to perform incremental load.- False - a normal initial load of data from source is done. |
| **Persistent table location(required)** | The Coalesce storage location. |
| **Persistent table name(required)** | The table name of the persistent table. |
| **Incremental load column(date)** | A date column based on which incremental data is loaded. |
| **Handle Data Quality**|When enabled you can add new columns to handle nullability and duplicate test.|
| **Nullability test**| When enabled,you can choose columns based on which nullability checks needs to be done|
| **Duplicate test**|When enabled,you can choose columns based on which duplicate checks can be done|
| **Pre-SQL**|SQL to execute before data insert operation|
| **Post-SQL**|SQL to execute after data insert operation|

## Incremental Load Example Workflow

1. Add a source node.
2. Add the Incremental UDN.
3. Leave the 'Filter data based on Persistent Table' option set to False.
4. Create the node.
5. Add the Persistent table to the view.
6. Create and Run the node.
7. Go to the Incremental UDN and change the 'Filter data based on Persistent Table' option to true.
8. Use the pattern based option to match the persistent table (alter the definition of the UDN, if necessary), or add the table name manually in the last config item.
9. Remove the existing (basic) join and use the 'Copy To Editor' to add the new join, including sub-select.
10. Re-run the Incremental UDN.

## Incremental Load-With data quality Example Workflow

1. Add a source node.
2. Add the Incremental UDN.
3. Enable 'Handle Data Quality' toggle
4. When data quality is handled ,filter based on target(incremental processing)is diabled
5. Two types of Data Quality checks possible,Nullability and Duplicate records.
6. Lets assume the column name chosen for handling DQ checks is "ACCOUNT_ID".Then,you can find ACCOUNT_ID_nulll,ACCOUNT_ID_dup,TOTAL_FAILED_TESTS and QUALITY_FLAG added to target table
7. Re-sync the columns after table/view is created
8.The target table has data quality info with the flag set to 'G' or 'B' denoting good or bad records
9. Further,you can drill down good or bad records using [Test Passed records](#test-passed-records) or [Test failed records](#test-failed-records)

### Incremental Load Deployment

#### Incremental Load Initial Deployment

When deployed for the first time into an environment the Incremental load node of materialization type table will execute the Create State Table.

| **Stage** | **Description** |
|----------|-----------------|
| **Create Stage Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment. When deployed for the first time into an environment the Work node of materialization type view will execute the Create Stage View. |
| **Create Stage View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |

#### Incremental Load Redeployment

After the Incremental Load  has been deployed for the first time into a target environment, subsequent deployments with column level changes or table level changes may result in altering the target Table.

#### Incremental Load Altering the Stage Tables

There are few column or table changes if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

* Changing the table name
* Dropping an existing column
* Altering Column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Clone Table** | Creates an internal table. |
| **Rename Table \| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation. |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost. |
| **Delete Table** | Drops the internal table. |

#### Incremental Load Recreating the Stage Views

The subsequent deployment of Incremental load node of materialization type view with changes in view definition,adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete View** | Delete the view |
| **Create View**| Create a new view |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Incremental Load Undeployment

If a Incremental load node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the stage table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete Table** | Coalesce Internal table is dropped. |
| **Delete Table** | Target table in Snowflake is dropped. |

If a Incremental load node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the StageView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete View** | Drops the existing stage view from target environment. |

## Test Passed records

The Coalesce "Test Passed Records" (GR) node is a specialized filtering node type designed to extract "clean" data from a source that has already undergone Data Quality (DQ) processing in Incremental node. It serves as a gatekeeper, ensuring only records that meet your business standards move further downstream.

## Key-functions

* **Quality Filtering:** By default, it acts as a simple filter that only selects records where the QUALITY_FLAG is set to 'G' (Good/Passed).
* **Duplicate Salvaging:** Unlike a standard filter, it provides logic to "salvage" duplicated records. If a record is flagged as a duplicate, you can configure the node to keep either the First Occurrence or the Last Occurrence based on a chosen timestamp column (timcol).
* **Always use on an incremental node with data quality handled**

## Test passed records Example Workflow

1. Add a source node.
2. Add the Incremental UDN.
3. Add Test Passed records UDN on top of Incremental node.
4. By default,where clause to filter 'G' quality is added in JOIN tab
5. If you want the first or last occurrence of record as valid enable corresponding toggles in config
6. Go to join tab and override join clause by clicking on 'Copy to editor'
7. The target will have good quality records

### Test Passed records Load Node Configuration

* [Node Properties](#test-passed-records-properties)
* [Options](#test-passed-records-options)

#### Test Passed records Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the test passed records node will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Test Passed records Options

| **Options** | **Description** |
|-----------|-----------------|
| **Create As** | Provides option to choose materialization type as `table` or `view`. |
| **First Occurence of Duplicate considered as valid**|When enabled,timestamp column should be provided to add the first occurence of duplicate as valid record based on timestamp column|
| **Last Occurence of Duplicate considered as valid**|When enabled,timestamp column should be provided to add the last occurence of duplicate as valid record based on timestamp column|

## Test Failed records

The Coalesce "Test Failed Records" node acts as a Quarantine and Error-Capture component. This node isolates all records that failed your Data Quality (DQ) tests for auditing and remediation.

## Key-functions

* **Error Isolation:** It filters the source to extract only records where the QUALITY_FLAG is 'B', ensuring that any record failing your Data Quality (DQ) standards is captured for auditing. This ensures that bad quality data is kept separate from your production analytics.
* **2.Duplicate Capture:** If deduplication logic is enabled (First or Last Occurrence), this node specifically captures the rejected duplicates, and remediation.

* **Always use on an incremental node with data quality handled**
  
## Test failed records Example Workflow

1. Add a source node.
2. Add the Incremental UDN.
3. Add Test Failed records UDN on top of Incremental node.
4. By default,where clause to filter 'B' quality is added in JOIN tab
5. If you want the first or last occurrence of record as valid enable corresponding toggles in config
6. Go to join tab and override join clause by clicking on 'Copy to editor'
7. The target will have good quality records

### Test Failed records Load Node Configuration

* [Node Properties](#test-failed-records-properties)
* [Options](#test-failed-records-options)

#### Test Failed records Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the test passed records node will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Test Failed records Options

| **Options** | **Description** |
|-----------|-----------------|
| **Create As** | Provides option to choose materialization type as `table` or `view`. |
| **First Occurence of Duplicate considered as valid**|When enabled,timestamp column should be provided to capture the records other than first occurence of duplicate considered as  valid record based on timestamp column as bad quality records|
| **Last Occurence of Duplicate considered as valid**|When enabled,timestamp column should be provided to capture the records other than last occurence of duplicate considered as  valid record based on timestamp column as bad quality record|

## Loop Load

The Coalesce node Looped Load dynamically groups incoming source data (incrementally if configured) and loads it into target data by looping through those groups.

It creates "load buckets" dynamically based on a selection of table keys and then uses those buckets to loop through source data and load into a target.

### Looped Load Node Configuration

* [Looped Load Node Properties](#looped-load-node-properties)
* [Looped Load Options](#looped-load-options)
* [Looped Load Incremental Load Options](#looped-load-incremental-load-options)
* [Looped Load Group Table Options](#looped-load-group-table-options)

#### Looped Load Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the WORK will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Looped Load Options

| **Option** | **Description** |
|----------|-----------------|
| **Enable tests** | This toggle can be enabled to perform data quality tests. |
| **Pre-SQL** | Any SQL to be executed as a predecessor to the data insert operation can be specified here. |
| **Post-SQL** | Any SQL to be executed after the data insert operation can be specified here. |
| **Load Type** | This toggle helps you choose different accepted Load Types from Config. If no load type parameter is specified in the workspace, then it falls back to the config level load type. |

#### Looped Load Incremental Load Options

| **Option** | **Description** |
|----------|-----------------|
| **Set Incremental Load Options** | - **True**: Prompts for load column based on which incremental data is loaded- **False**: Incremental processing is not done |
| **Incremental Load Column(date)** | A date column based on which incremental data is loaded |
| **Group Incremental** | - **True**: The data is grouped based on the number of buckets input- **False**: The data is grouped into a single group |

#### Looped Load Group Table Options

| **Option** | **Description** |
|----------|-----------------|
| **Dedicated Load History Table** | - **True**: A history table specific to this target table is created by prefixing target table name to `TABLE_GROUP_LOAD`- **False**: A history table with common name `TABLE_GROUP_LOAD` is created |
| **Load History Table Location** | - Same as Target: The history table is created in the same location as Target table- Utility Schema: The history table is created in the location specified in `UtilitySchema` parameter |

### Looped Load Parameters

The Looped Load node specifies a `loadType` parameter to perform:

* A full load
* A full reload
* Incremental load
* Reprocess load

Also a parameter `utilitySchema` can be used to set a target table in a location other than target table location.

```json
{
     "loadType_accepted_values": [
             "Incremental Load",
                    "Full Load",
               "Reprocess Load",
                  "Full Reload"
 ],
       "loadType": "Full Load",
   "utilitySchema": "TARGET_DB"
}
```

### Key Points on Looped Load Node

* No data gets loaded when `Load_type` is set to `Reprocess load` if the previous run was all successful.
  Only when the previous runs has any entries with process status '-1' denoting failure, data gets loaded during `Reprocess Load`.
* The choice of bucket column is crucial for Keybased grouping as choosing a column with distinct unique values affects the performance of data load.

### Looped Load Deployment

#### Looped Load Initial Deployment

When deployed for the first time into an environment the Looped load node will execute the following stages:

| **Stage** | **Description** |
|----------|-----------------|
| **Create Load Group Table** | This will execute a CREATE OR REPLACE statement and create a load group table in the target environment. |
| **Create Stage Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment. |

#### Looped Load Redeployment

After the Looped Load  has been deployed for the first time into a target environment, subsequent deployments with column level changes or table level changes may result in altering the target Table.

#### Looped Load Altering the Target Tables

Column or table changes that will result in an ALTER statement to modify the Work Table in the target environment. These changes can be made either in isolation or all together.

* Change in table name
* Dropping existing column
* Alter Column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Clone Table** | Creates an internal table |
| **Rename Table \| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation accordingly |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost |
| **Delete Table** | Drops the internal table |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Looped Load Undeployment

If a Incremental load node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the stage table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete Table** | Coalesce Internal table is dropped |
| **Delete Table** | Target table in Snowflake is dropped |

If a Incremental load node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the StageView in the target environment will be dropped.

The stage executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete View** | Drops the existing stage view from target environment |

## Run View

The Coalesce Run View is a variant of the basic View node type. In contrast to the standard View node, the custom Run View node actually includes the view creation logic in the Run tab. When a job using these views executes, the view will be recreated as part of the job and will utilize whatever parameter values are passed in. The parameters are used for selecting which sources to load into the target table.

### Run View Node Configuration

* [Node Properties](#run-view-node-properties)
* [Run View Options](#run-view-options)
* [Run View Increment Options](#run-view-increment-options)

#### Run View Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the WORK will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Run View Options

| **Options** | **Description** |
|---|---|
| **Multi Source** | True / False toggle that is Coalesce implementation of SQL UNIONs.  - **True** - Multiple sources can be combined in a single node. The sources are combined using the option specified in the Multi Source Strategy.  - **False** - Single source node or multiple sources combined using a join. |
| **Override Create SQL** | True/False that determines whether a customized Create SQL is required to be executed.  - **True** - Customized Create SQL specified in the Create SQL space is executed. All other options are invisible.  - **False** - Create view SQL based on the options chosen are framed and executed. |
| **Filter data based on source** | True/False that determines whether data is to be filtered based on parameter  - **True** - Length of [source suffix](#tips-on-choosing-the-source-suffix) is required to be added to frame the filter in join clause. A point to remember here is clear the join tab and use the 'Copy To Editor' to add the new join after enabling 'Filter data based on source' option  - **False** - No filter is added to the join tab. |

#### Run View Increment Options

| **Options** | **Description** |
|---|---|
| **Incremental load based on Persistent table** | True / False toggle that helps us to load incremental load.  - **True** - provides option to perform incremental load.  - **False** - a normal initial load of data from source is done. |
| **Persistent table location (required)** | The Coalesce storage location is required to be added here. |
| **Persistent table name (required)** | The table name of the persistent table is required to be added. |
| **Incremental load column(date)** | A date column based on which incremental data is loaded. |

### Run View Example Workflow

![Workflow](https://github.com/coalesceio/Incremental-Nodes/assets/7216836/84205415-3eb2-4d3a-a4ba-a3f21c4edb9f)

1. Add a source node.
2. Add the Run View nodes on different sources.
3. Leave the `Filter data based on Source` option set to false.
4. Create the base views from source.
5. Add another view node with multi source enabled to combine the created sources.
6. Enable the multi-source toggle and `Filter data based on Source`.
7. Enter the length of the source suffix to identify the source. For instance, if the source names follow the pattern `RV_CUSTOMERA`, `RV_CUSTOMERB`, then the length of suffix can be set to 1, or if the source names follow the pattern `RV_COURSES_IT`, `RV_COURSES_SC`, then the length of suffix can be set to 2 to uniquely identify the source to be loaded.
8. Ensure the source suffix for sources follows a unique pattern like an alphabetical sequence (A,B,C,D), numerical sequence (1,2,3,4) or any meaningful pattern ('SC','IT').
9. Set the [parameter](#run-view-parameters) source suffix to `ALL` if you want to process all sources, else set it in accordance with the preferred sources to be processed.
10. Remove the existing (basic) join and use the 'Copy To Editor' to add the new join, including the filter clause for filtering out data from specific sources.
11. Create and Run the Run View.
12. You can dynamically create a view with data from preferred sources in case of multiple sources.

### Run View Incremental Processing Example

1. Add the Persistent table to the view.
2. Create and Run the node.
3. Now go back to the Run View node and set the Incremental options in Config.
4. Remove the existing (basic) join and use the 'Copy To Editor' to add the new join, including sub-select.
5. Re-run the Run View node.

### Tips on Choosing the Source Suffix

* Ensure that the multiple sources have source name suffixes with a uniform pattern and length.
* For example, the multiple sources can be named PRODUCTA, PRODUCTB, PRODUCTC, and so on. In this case, we can set the parameter **sourcesuffix** to ('A') to process only source PRODUCTA, ('A','B') to process sources PRODUCTA, PRODUCTB, and ('ALL') to load all sources. In a similar way, we can have numerical suffixes as well. In this case, the source suffix length in Config is to be set to **1**.
* Another example, source suffixes can have a meaningful pattern with uniform length like *`PRODUCT_ELEC`*, *`PRODUCT_HOME`*, *`PRODUCT_SPTS`* wherein the parameter **sourcesuffix** can be ('ELEC'), ('HOME'), ('ALL'). In this case, the source suffix length in Config is to be set to **4**.

### Run View Parameters

The Run View node specifies a sourcesuffix parameter for selecting which sources to load into the target table.The sourcesuffix refers to the suffix of the sources you want to load to the target view.

For instance,if we have multiple sources like `RV_CUSTOMERA`, `RV_CUSTOMERB`, and `RV_CUSTOMERC`.

* You can set the sourcesuffix to 'A' if you want to add data from only source `RV_CUSTOMERA` to target data.

```json
{
  "sourcesuffix": "('A')"
  }
```

* You can set the sourcesuffix to 'A','B' if you want to add data from only source `RV_CUSTOMERA` and `RV_CUSTOMERB` to target data.

```json
  {
  "sourcesuffix": "('A','B')"
  }
```

* You can set  the sourcesuffix to 'ALL' if you want to add data from all sources to target data.
  
```json
{
  "sourcesuffix": "('ALL')"
  }
```

* You can add numeric prefixes.
  
```json
{
"sourcesuffix": "('1','2')"
}
```

### Run View Deployment

When deploying the DAG using Run View nodes the environment parameter should be set to `DEPLOY`.

**Deployment Value** - When deploying this pipeline, use this parameter value.

```json
{
"sourcesuffix": "('DEPLOY')"
}
```

When running a job with this pipeline, fill in the parameter with the sources you want to execute.

#### Run View Initial Deployment

When deployed for the first time into an environment the Run View will execute the following stages:

| **Stage** | **Description** |
|---|---|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |

#### Run View Redeployment

The subsequent deployment of a View node with changes in view definition, adding table description, adding secure option, or renaming view results in recreating the view.

**Recreating the View**

| **Stage** | **Description** |
|---|---|
| **Create View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Run View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

* **Delete View**

## Grouped Incremental Load

The Coalesce Grouped Incremental load node is a versatile node that allows you to develop and deploy a Stage table/Transient Table in Snowflake where we can perform Grouped Incremental load from different storage locations with a persistent table added on top of it.

### Grouped Incremental Load Node Configuration

* [Node Properties](#grouped-incremental-load-node-properties)
* [Options](#grouped-incremental-load-options)

#### Grouped Incremental Load Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the WORK will be created. |
| **Node Type** | Name of template used to create node objects. |
| **Description** | A description of the node's purpose. |
| **Deploy Enabled** | - If TRUE the node will be deployed / redeployed when changes are detected.- If FALSE the node will not be deployed or will be dropped during redeployment. |

#### Grouped Incremental Load Options

| **Options** | **Description** |
|-----------|-----------------|
| **Create As** | Provides option to choose materialization type as `table` or `transient table`. |
| **Storage Mapping ID** | Provides option enter storage locations defined in Coalesce separated by comma(E.g EDW_RAW,EDW_PA). |
| **Batch Load** | - True - provides option to perform data loading in batches.- False - a normal load of data from each storage location is done. |
| **Persistent table location(required)** | The Coalesce storage location of Persistent table. |
| **Persistent table name(required)** | The table name of the persistent table. |
| **Persistent table Column Name(required)** | The column name of the persistent table which represents the storage location id. |
| **Joining Table Column Name(optional)** | The column name of the table which will be joined with main table. |
| **Grouped Incremental load column(date)** | A date column based on which grouped incremental data is loaded. |

## Grouped Incremental Load Example Workflow

1. Add a source node.
2. Add the Grouped Incremental UDN.
3. Add Storage Mapping ID seprated by comma.
3. Leave the 'Batch Load' option set to False.
4. Enter Persistent Table location
5. Enter Persistent Table name 
6. Enter Persistent Table Column Name 
7. Enter Joining Table Column Name if join is present
8. Select Incremental Load Column
9. Generate join, use the 'Copy To Editor' to add the new join, including sub-select..
10.Create and Run the node.

### Grouped Incremental Load Deployment

#### Grouped Incremental Load Initial Deployment

When deployed for the first time into an environment the Grouped Incremental load node of materialization type table will execute the Create State Table.

| **Stage** | **Description** |
|----------|-----------------|
| **Create Stage Table** | This will execute a CREATE OR REPLACE statement and create a table in the target environment. When deployed for the first time into an environment the Work node of materialization type view will execute the Create Stage View. |
| **Create Stage View** | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |

#### Grouped Incremental Load Redeployment

After the Grouped Incremental Load  has been deployed for the first time into a target environment, subsequent deployments with column level changes or table level changes may result in altering the target Table.

#### Grouped Incremental Load Altering the Stage Tables

There are few column or table changes if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.

* Changing the table name
* Dropping an existing column
* Altering Column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Clone Table** | Creates an internal table. |
| **Rename Table \| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation. |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost. |
| **Delete Table** | Drops the internal table. |

#### Grouped Incremental Load Altering the Transient Tables

There are few column or table changes if made in isolation or all-together will result in an ALTER statement to modify the table in the target environment.

* Changing the table name
* Dropping an existing column
* Altering Column data type
* Adding a new column

The following stages are executed:

| **Stage** | **Description** |
|----------|-----------------|
| **Clone Table** | Creates an internal table. |
| **Rename Table \| Alter Column \| Delete Column \| Add Column \| Edit table description** | Alter table statement is executed to perform the alter operation. |
| **Swap cloned Table** | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost. |
| **Delete Table** | Drops the internal table. |

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Grouped Incremental Load Undeployment

If a Grouped Incremental load node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the stage table in the target environment will be dropped.

This is executed in two stages:

| **Stage** | **Description** |
|----------|-----------------|
| **Delete Table** | Coalesce Internal table is dropped. |
| **Delete Table** | Target table in Snowflake is dropped. |

If a Grouped Incremental load node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the StageView in the target environment will be dropped.

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Table | Table | Follows existing redeployment stages |
| Transient Table | Transient Table | Follows existing redeployment stages |
| Run View |Run View | Follows existing redeployment stages |
| Any Other | Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Transient Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Run View | 1. Warning (if applicable)2. Drop  3. Create |

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added when there's a node type switch |

--------------

### Code

#### Incremental Load Code

* [Node definition](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/IncrementalLoad-230/definition.yml)
* [Create Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/IncrementalLoad-230/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/IncrementalLoad-230/run.sql.j2)

#### Looped Load Code

* [Node definition](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/LoopedLoad-278/definition.yml)
* [Create Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/LoopedLoad-278/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/LoopedLoad-278/run.sql.j2)

#### Run View Code

* [Node definition](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/RunView-281/definition.yml)
* [Create Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/RunView-281/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Incremental-Nodes/blob/main/nodeTypes/RunView-281/run.sql.j2)

#### Run View Macros

* [Macros](https://github.com/coalesceio/Incremental-Nodes/blob/main/macros/macro-1.yml)

#### Grouped Incremental Load Code

* [Node definition](https://github.com/coalesceio/Incremental-Nodes/tree/main/nodeTypes/GroupedIncrementalLoad-394/definition.yml)
* [Create Template](https://github.com/coalesceio/Incremental-Nodes/tree/main/nodeTypes/GroupedIncrementalLoad-394/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Incremental-Nodes/tree/main/nodeTypes/GroupedIncrementalLoad-394/run.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.1.1 | March 10, 2026 | NM-220 Added Node Type Switch support in advance deploy templates |
| 2.1.0 | February 17, 2026 | Data Quality Handling added to Incremental node, Test Passed/Failed records |
| 2.0.1 | November 18, 2025 | Added an else block to the RUN template to handle cases where the materializationType is View |
| 2.0.0 | July 18, 2025 | Adding and removing deploy phases at appropriate stages |
| 1.1.3 | March 18, 2025 | Fix to force users to add required parameters |
| 1.1.2 | February 28, 2025 | Deployment fixes for redeployment |
| 1.1.1 | December 20, 2024 | NM-92 - Grouped Incremental Load Added |
| 1.1.0 | June 27, 2024 | Comments added in Insert clause of Incremental load removed |
| 1.0.0 | May 15, 2024 | Initial Versions |

---

## Interactive Table

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/interactive-table</code> <CopyButton textToCopy='@coalesce/snowflake/interactive-table' />
Supported Platform: 
Latest Version: 1.0.0 (December 08, 2025)

analytics-readyinteractive-tablesreal-time-data

## Overview
Interactive table deliver low-latency query performance for high-concurrency, interactive workloads.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/interactive-table</code> <CopyButton textToCopy='@coalesce/snowflake/interactive-table' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
### Interactive Table Package

### Package Includes
* [Interactive Table](#interactive-table)
  
---
## Interactive Table

The Coalesce Interactive Table UDN allows you to build and deploy Snowflake Interactive Table, which are optimized for low-latency, high-performance analytical queries.

Interactive Table refresh automatically based on a SQL definition and maintain optimized storage for low-latency, high-concurrency queries. This makes them ideal for powering live dashboards, APIs, and sub-second analytical loads—while reducing operational complexity and maintaining strong price-performance.
 
**NOTE**: Interactive Table can be only created at the end of a data pipeline. It cannot be used as a source for other table.

### Interactive Table Node Configuration
* [Node properties](#interactive-table-node-properties)
* [General Options](#interactive-table-general-options)
* [Dynamic Interactive Table Options](#interactive-table-dynamic-interactive-table-options)

### Node properties
| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Interactive Table will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed/redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |
					
					
### General Options 
 

| **Property** | **Description** |
|---------|-------------|
| **Multi Source** | Toggle: True/FalseImplementation of SQL UNIONs**True**: Combine multiple sources in a single nodeTrue Options:- **UNION**: Combines with duplicate elimination- **UNION ALL**: Combines without duplicate elimination |
| **Enable tests** | Toggle: True/FalseDetermines if tests are enabled |
| **Distinct** | Toggle: True/False**True**: Group by All is invisible. DISTINCT data is chosen for processing**False**: Group by All is visible |
| **Group by All** | Toggle: True/False**True**: DISTINCT is invisible. Data is grouped by all columns for processing**False**: DISTINCT is visible |
| **Cluster key** | True/False toggle for clustering:- **True**: Specify clustering column and optional expressions- **False**: No clustering |
| **Allow Expressions Cluster Key**| When cluster key is set to true. Allows to add an expression to the specified cluster key| 
| **Auto Refresh** | Toggle: True/False **True** Dynamic Interactive Table options visible.-**False**: Creates Static Interactive table.|

### Dynamic Interactive Table Options

| **Option** | **Description** |
|------------|----------------|
| **Lag Specification** |  **Time Value**: Frequency of the refresh- **Time Period**: Seconds/Minutes/Hours/Days |
| **Warehouse** | (Required) Name of warehouse used to refresh the Interactive Table |	

### Interactive Table  Deployment

#### Interactive Table  Initial Deployment Parameters

The Interactive Table  includes an environment parameter that allows you to specify a different warehouse to refresh a Interactive Table in different environments.

The parameter name is `targetInteractiveTableWarehouse` and the default value is `DEV ENVIRONMENT`.

When set to `DEV ENVIRONMENT`, the value entered in the Interactive Table Options config "Warehouse on which to execute Interactive Table" will be used when creating the Interactive Table.

```json
{
    "targetInteractiveTableWarehouse": "DEV ENVIRONMENT"
}
```

When set to any value other than `DEV ENVIRONMENT` the node will attempt to create the Interactive Table using a Snowflake warehouse with the specified value.

For example, the Interactive Table will refresh using a warehouse named `compute_wh`.

```json
{
    "targetInteractiveTableWarehouse": "compute_wh"
}
```
> 📘 **Deployment of nodes without adding parameters**
>
> This results in a WARNING stage getting executed insisting to execute the node after adding parameters 

### Interactive Table  Initial Deployment

When deployed for the first time into an environment the Interactive Table  node will execute the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create  Interactive Table | This stage will execute a `CREATE OR REPLACE` statement and create a Interactive Table in the target environment. |

#### Interactive Table  Redeployment

After initial deployment, subsequent deployments may recreate the Interactive Table.

#### Recreating the Interactive Table 

The following config changes trigger REPLACE statements:

1. Warehouse name
2. Auto refresh setting  
3. Lag specification

This executes a stage:

| **Stage** | **Description** |
|-----------|----------------|
| **CREATE or REPLACE Interactive Table** | Executes CREATE or REPLACE to modify parameters |

Also if the location of the node, node name, column level description, and table level description results in an `CREATE or REPLACE` statement, whereas table name change result in a `ALTER` statement.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

### Interactive Table  Undeployment

A table will be dropped if all of these are true:

* The Interactive  Node is deleted from a space.
* The space is committed to Git.
* The space committed to Git is deployed to a higher level environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Interactive Table** | Removes table from target environment |

---

## Code

### Interactive Table Code
* [Node definition](https://github.com/coalesceio/interactive-tables/blob/main/nodeTypes/InteractiveTable-602/definition.yml)
* [Create Template](https://github.com/coalesceio/interactive-tables/blob/main/nodeTypes/InteractiveTable-602/create.sql.j2)
* [Run Template](https://github.com/coalesceio/interactive-tables/blob/main/nodeTypes/InteractiveTable-602/run.sql.j2) 

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.0 | December 08, 2025 | Interactive table first release. |

---

## Materialized View(Package)

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/materialized-view</code> <CopyButton textToCopy='@coalesce/snowflake/materialized-view' />
Supported Platform: 
Latest Version: 2.0.1 (February 18, 2026)

cachingmaterialized viewperformacepre-aggregationquery optimization

## Overview
A materialized view is a pre-computed dataset derived from a query, stored for faster querying than against the base table.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/materialized-view</code> <CopyButton textToCopy='@coalesce/snowflake/materialized-view' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Materialized View Stage

The Coalesce Materialized View Stage UDN is a versatile node that allows you to develop and deploy a Materialized View in Snowflake.

A [materialized view](https://docs.snowflake.com/en/user-guide/views-materialized) is a pre-computed data set derived from a query specification (the SELECT in the view definition) and stored for later use. materialized views can speed up expensive aggregation, projection, and selection operations, especially those that run frequently and that run on large data sets.

## Node Configuration

The Materialized View Stage has three configuration groups:

* [Node Properties](#node-properties)
* [Materialized View Options](#materialized-view-options)
* [General Options](#general-options)

### Node Properties
There are four configs within the **Node Properties** group.

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Materialized View will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed / redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

### Materialized View Options

There are two configs within the **Materialized View Options** group.

| **Option** | **Description** |
|------------|----------------|
| **Cluster key** | True/False to determine whether Materialized view is to be clustered or not- **True**: Allows you to specify the column based on which clustering is to be done  -**Allow Expressions Cluster Key**: True allows to add an expression to the specified cluster key- **False**: No clustering done |
| **Secure** | True / False Toggle to determine whether Materialized view to be created in a secured mode- **True**: Materialized view created in a secured mode- **False**: No additional secure option added during Materialized view creation |

### General Options

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True / False toggle that specifies whether or not to return DISTINCT rows |

## Limitations of Materialized view

Materialized view has a set of limitations:

* A materialized view cannot query a materialized or non-materialized view
* A materialized view does not support group by all but group by is supported
* A materialized view can query only a single table.

Review [Snowflake's Limitations on Creating Materialized Views](https://docs.snowflake.com/en/user-guide/views-materialized#limitations-on-creating-materialized-views).

## Deployment

### Initial Deployment
When deployed for the first time into an environment Materialized View will execute three stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Materialized View** | This stage will execute a CREATE OR REPLACE statement and create a Materialized View in the target environment |
| **Applying Materialized View Clustering** | This stage will apply clustering to the created Materialized View if Clustering is set to true |
| **Resume recluster Materialized View** | This stage will resume the Materialized View based on clustering |

### Redeployment

After the Materialized View has deployed for the first time into a target environment, subsequent deployments may result in either altering the Materialized View or recreating the Materialized View.

If a Materialized View is to be altered this will run the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Materialized View** | This stage will execute an ALTER statement and alter the Materialized View in the target environment setting the new parameters |
  
#### Altering the Materialized View
  
There are two config changes that if made in isolation or all-together will result in an ALTER statement to modify the Materialized View in the target environment.

* Cluster key
* Secure

#### Recreating the Materialized View

If anything changes other than the configuration options specified above then the Materialized View will be recreated by running a CREATE OR REPLACE statement.

### Redeployment with no changes 

If the nodes are redeployed with no changes compared to previous deployment,then no stages are executed

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.29+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more info click here - [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Undeployment

If a Materialized View is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the Materialized View in the target environment will be dropped.

This is executed as a single stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Materialized View** | Removes the materialized view |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Table | Table | 1. Warning (if applicable)2. Metadata Update(if applicable)3. Alter |
| Transient Table | TransientTable | 1. Warning (if applicable)2. Metadata Update(if applicable)3. Alter |
| View | View | 1. Warning (if applicable)2. Create |
| Any Other | Table | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | View | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Transient Table | 1. Warning (if applicable)2. Drop  3. Create |

Please review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older Version Create or Alter View | Any | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |

--------------

## Code

* [Node definition](https://github.com/coalesceio/Materialized-View-Node/blob/main/nodeTypes/MaterializedViewStage-178/definition.yml)
* [Create Template](https://github.com/coalesceio/Materialized-View-Node/blob/main/nodeTypes/MaterializedViewStage-178/create.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.0.1 | February 18, 2026 | NM-220 Node Type Switching supported in Materialized view Template |
| 2.0.0 | July 18, 2025 | Addition of deploy phase for Create/cluster stages and remove for alter steps |
| 1.1.2 | November 14, 2024 | Node description/comment changes |
| 1.1.1 | November 01, 2024 | Sngle initialization block for namespace variables |
| 1.1.0 | October 11, 2024 | Cluster key formatting and default option changes |

---

## Semantic View

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/semantic-view</code> <CopyButton textToCopy='@coalesce/snowflake/semantic-view' />
Supported Platform: 
Latest Version: 1.0.1 (June 10, 2026)

analytics-layerbusiness-layerdata-modelingfact-dimensionmetrics-layersemantic-layersemantic-modelsemantic-querysemantic-viewsnowflakesnowflake-features

## Overview
A package that provides nodes to create Semantic Views and Semantic Queries, enabling a business-friendly semantic layer on top of Snowflake data for easier analytics and reporting.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/semantic-view</code> <CopyButton textToCopy='@coalesce/snowflake/semantic-view' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
# Semantic View Package

The Coalesce Semantic View Package helps you build Semantic Views and Semantic Queries in Snowflake so analysts can query data using business terms instead of complex SQL.

---

## About Semantic Views

A Semantic View is a logical layer built on top of multiple database tables to make data easier to understand and query. Instead of writing complex SQL with many joins and aggregations, you can query the Semantic View using simple business terms like dimensions and metrics.

It defines how tables are related, which columns represent descriptive attributes (dimensions), and which represent measurable values (metrics or facts). The semantic view hides the underlying complexity of joins, keys, and calculations, and presents a clean analytical model.

In simple terms, a semantic view acts like a business-friendly data model that allows analysts or tools to query data without needing to know the full database structure. It is especially useful for reporting, dashboards, and analytics because it ensures consistent calculations and relationships across queries.

Within a semantic view, you define logical tables that typically correspond to business entities, such as customers, orders, or suppliers. You can define relationships between logical tables through joins on shared keys, enabling you to analyze data across entities (as you would when joining database tables).

Using logical tables, you can define Facts, Metrics, and Dimensions:

### Facts

Facts are row-level attributes in your data model that represent specific business events or transactions. While facts can be defined using aggregates from more detailed levels of data, such as `SUM(t.x)` where `t` represents data at a more detailed level, they are always presented as attributes at the individual row level of the logical table. Facts capture "how much" or "how many" at the most granular level, such as individual sales amounts, quantities purchased, or costs. Facts typically function as helper concepts within the Semantic View to help construct dimensions and metrics.

### Metrics

Metrics are quantifiable measures of business performance calculated by aggregating facts or other columns from the same table using functions like SUM, AVG, and COUNT across multiple rows. They transform raw data into meaningful business indicators, often combining multiple calculations in complex formulas. Examples include Total Revenue or Profit Margin Percentage. Metrics represent the KPIs in reports and dashboards that drive business decision-making.

### Dimensions

Dimensions represent categorical attributes. They provide the contextual framework that gives meaning to metrics by grouping data into meaningful categories. They answer "who," "what," "where," and "when" questions, such as purchase date, customer details, product category, or location. Typically text-based or hierarchical, dimensions enable you to filter, group, and analyze data from multiple perspectives.

In a Semantic View, these 3 elements have distinct roles, but metrics and dimensions are the primary elements that you interact with when analyzing data through the Semantic View. Facts provide the underlying row-level numerical data, metrics transform data into actionable insights through aggregation and calculation, and dimensions determine viewing perspectives.

## Package Includes
* [Semantic View](#semantic-view)
* [Semantic Query](#semantic-query)

## Semantic View 

### Semantic View Node Configuration

* [Node Properties](#node-properties)
* [Semantic View Options](#semantic-view-options)

### Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Semantic View will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detectedIf FALSE the Node will not be deployed or will be dropped during redeployment |

### Semantic View Options

| **Property** | **Description** |
|---------|-------------|
| Create Schema Table | Toggle: True/False`True`: Creates a schema table for the Semantic View. Required when using a Semantic Query node so downstream nodes can fetch and use columns.`False`: Semantic View is created |
| Materialization | Defines how the semantic model is created.Options:- Semantic View |
| Consider Table Only (No Primary Key) | Toggle: True/False`True`: Builds only table names`False`: Builds table names along with Primary keys |
| TABLES (Primary Key) | Tabular configuration for defining source tables and their Primary Key columns.- `Source Column`: Select the primary key column from the source table.- `Alias`: Optional alias name for the column.- `Synonym`: Optional alternative names for the column.- `Comment`: Optional description for the column. |
| Enable Relationships | Toggle: True/False`True`: Enables the RELATIONSHIPS (Foreign Key) configuration section.`False`: Relationship configuration remains hidden. |
| RELATIONSHIPS (Foreign Key) | Tabular configuration for defining foreign key relationships between tables.- `Source Column`: Column acting as the foreign key.- `Reference Table`: Table being referenced.- `Alias`: Optional alias for the reference column. |
| Enable Facts | Toggle: True/False`True`: Enables the FACTS configuration section.`False`: Facts configuration remains hidden. |
| FACTS | Tabular configuration for defining fact columns or measures.- `Derived Stage`: Processing stage for derived columns (1, 2, or 3). Visible only when schema table creation is enabled.- `Source Column`: Column selected from the source table.- `Transform`: Expression applied on the column.- `Alias`: Name used in the semantic model.- `Synonym`: Optional alternative names.- Comment: Optional description. |
| Enable Dimensions | Toggle: True/False`True`: Enables the DIMENSIONS configuration section.`False`: Dimensions configuration remains hidden. |
| DIMENSIONS | Tabular configuration for defining dimension attributes.- `Derived Stage`: Processing stage for derived columns (1, 2, or 3). Visible only when schema table creation is enabled.- `Source Column`: Column selected from the source table.- `Transform`: Expression applied to the column.- `Alias`: Name used in the semantic model.- `Synonym`: Optional alternative names.- `Comment`: Optional description. |
| Enable Metrics | Toggle: True/False`True`: Enables the METRICS configuration section.`False`: Metrics configuration remains hidden. |
| METRICS | Tabular configuration for defining aggregated metrics.- `Derived Stage`: Processing stage for derived columns (1, 2, or 3). Visible only when schema table creation is enabled.- `Metric Applied`: Aggregation function applied to the column (COUNT, SUM, AVG, MIN, MAX).- `Source Column`: Column used for aggregation.- `Derived Column`: Optional custom expression for metric calculation.- `Alias`: Name used for the metric in the semantic model.- `Synonym`: Optional alternative names.- `Comment`: Optional description. |
| Instructions For SQL Generation | Embeds custom instructions for Cortex Analyst, guiding how it generates SQL from natural-language questions |
| Instructions For Question Categorization | Governs how Cortex Analyst classifies incoming questions before SQL generation - used to reject out-of-scope or sensitive question categories |

### Semantic View Browser Creation Steps

1. **Enable "Create Schema Table"**
   - Turn **Create Schema Table** toggle **ON**.
   - Set the Derived Stage dropdown to the appropriate level based on the columns used in the expression.
   - This creates an intermediate schema table that helps fetch and manage columns during configuration.

2. **Configure Required Components**
   Enter the necessary configurations depending on your use case:
   - **Primary Keys** under *TABLES (Primary Key)*
   - **Relationships** under *RELATIONSHIPS (Foreign Key)* (if required)
   - **Facts**
   - **Dimensions**
   - **Metrics**

3. **Create Table in Browser**
   - Use **Create Table in Browser** to generate the schema table.
   - This allows the columns to be available for further configuration and downstream usage.

4. **Resync Columns**
   - After successful table creation, click **Resync Columns**.
   - This fetches the generated columns and updates the node metadata.

5. **Disable "Create Schema Table"**
   - Turn **Create Schema Table** toggle **OFF**.
   - This switches the node to generate a **Semantic View** instead of a schema table.

6. **Create Semantic View**
   - Deploy or create the **Semantic View** using the same configuration already defined.

7. **Important Note for Deployment**
   - Before deploying to the **target Environment**, ensure **Create Schema Table** is **OFF**.
   - If left **ON**, the process may recreate the schema table instead of the semantic view.
   - The same key can serve dual roles, such as a primary key and a dimension/fact attribute, so it appears as a single field by default             unless explicitly separated using an alias.

### Limitations

- This feature currently creates only **basic-level semantic nodes**.

- Complex semantic models that require **advanced transformations or multi-step logic** may not be supported at this time. Support for these scenarios is planned in future updates.

- **Validation must be completed before deployment** to ensure the semantic view definition is valid.

- For the full set of validation rules and constraints, refer to the [Snowflake Semantic View documentation](https://docs.snowflake.com/en/user-guide/views-semantic/validation-rules).

## Semantic View Deployment

### Semantic View Initial Deployment

When deployed for the first time into an Environment, the Semantic View Node executes the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Semantic View** | This stage executes a `CREATE OR REPLACE` statement and creates a Semantic View in the target Environment. |

#### Semantic View Redeployment

After initial deployment, subsequent deployments recreate the Semantic View.

### Redeployment with no changes 

If the Nodes are redeployed with no changes compared to previous deployment, then no stages are executed

### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a Node's materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Semantic View Undeployment

A table is dropped if all of these are true:

* The Semantic View Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher-level Environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Semantic View** | Removes object from target Environment |

------

### Semantic Query Node Configuration

* [Node Properties](#semantic-query-node-properties)
* [Semantic Query Options](#semantic-query-options)

### Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | (Required) Storage Location where the Semantic Query will be created |
| **Node Type** | (Required) Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the Node will be deployed or redeployed when changes are detectedIf FALSE the Node will not be deployed or will be dropped during redeployment |

### Semantic Query Options 

| **Property** | **Description** |
|---------|-------------|
| Materialization | Defines how the semantic query output is created.Options:- `view` |
| Order By | Toggle: True/False`True`: Enables sorting configuration for the semantic query.`False`: No ordering is applied. |
| Order By Columns | Tabular configuration for defining sorting columns. Visible only when Order By is enabled.- `Sort Column`: Column used for sorting.- `Sort Order`: Sorting direction.Options:- DESC (default)- ASC |
| FACTS | Column selector used to choose fact columns from the semantic view for the query output. |
| DIMENSIONS | Column selector used to choose dimension columns from the semantic view for the query output. |
| METRICS | Column selector used to choose metric columns from the semantic view for the query output. |

## Semantic Query Deployment

### Semantic Query Initial Deployment

When deployed for the first time into an Environment, the Semantic Query Node executes the following stage:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Semantic Query View** | This stage executes a `CREATE OR REPLACE` statement and creates a view in the target Environment. |

#### Semantic Query Redeployment

After initial deployment, subsequent deployments recreate the Semantic Query View with the default deployment strategy.

### Redeployment with no changes 

If the Nodes are redeployed with no changes compared to the previous deployment, then no stages are executed.

### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a Node's materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Semantic Query Undeployment

A table is dropped if all of these are true:

* The Semantic Query Node is deleted from a Workspace.
* The Workspace is committed to Git.
* The Workspace committed to Git is deployed to a higher-level Environment.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Semantic Query** | Removes object from target Environment |

---

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Semantic View | Semantic View | Follows existing redeployment stages |
| Any Other | Semantic View | 1. Warning (if applicable)2. Drop  3. Create |
| Any  | View(Semantic Query) | May recreate with the default deployment strategy, but it might not work as expected |

Review the documented limitations before performing a Node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless the current Node uses the latest package that supports Node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First Node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first Nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the Node's materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Stream for Directory Table (and vice versa) | Metadata columns are not automatically synchronized. Specific directory columns (e.g., `relative_path`, `size`, `md5`) are not added when switching to Directory Table, nor are they removed when switching back to a standard Stream. |
| 11 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added during a Node type switch |
| 12 | Any | View(Semantic Query) | May recreate it with the default deployment strategy, but it might not work as expected since semantic queries are only valid when the source is a Semantic Query. |

--------------

## Code

### Semantic View Code
* [Node definition](https://github.com/coalesceio/semantic-node-types/blob/main/nodeTypes/SemanticView-632/definition.yml)
* [Create Template](https://github.com/coalesceio/semantic-node-types/blob/main/nodeTypes/SemanticView-632/create.sql.j2)

### Semantic Query Code

* [Node definition](https://github.com/coalesceio/semantic-node-types/blob/main/nodeTypes/SemanticQuery-631/definition.yml)
* [Create Template](https://github.com/coalesceio/semantic-node-types/blob/main/nodeTypes/SemanticQuery-631/create.sql.j2)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.0.1 | June 10, 2026 | Added Cortex Analyst configs for SQL generation instructions and question categorization control. |
| 1.0.0 | March 13, 2026 | Initial release version of basic semantic view and query node types |

---

## Streams and Tasks

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/streams-and-tasks</code> <CopyButton textToCopy='@coalesce/snowflake/streams-and-tasks' />
Supported Platform: 
Latest Version: 2.7.2 (July 01, 2026)

automationcdcchange data captureevent-drivenincremental processingreal-timeschedulingstreamingtask orchestration

## Overview
Snowflake streams and tasks out of the box.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/streams-and-tasks</code> <CopyButton textToCopy='@coalesce/snowflake/streams-and-tasks' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
<!-- markdownlint-disable MD033 -->
## Snowflake - Streams and Tasks Package - Brief Summary

-   **Work with Task:** Used for intermediate "Work" tables. It automates the cleaning and transformation of data in the middle of a pipeline using a scheduled Task.
-   **Dimension with Task:** Automates the loading of descriptive business data (e.g., Customers, Products). It uses a Task to ensure that attributes are kept up-to-date incrementally.
-   **Fact with Task:** Handles large-scale transactional data. This node uses a Task to process new metrics or events at regular intervals, ensuring large tables stay current without full reloads.
-   **Task DAG Create Root:** Defines the "parent" or starting point of a **Directed Acyclic Graph (DAG)**. This is the first task in a sequence that triggers all other dependent child tasks.
-   **Task DAG Custom-SQL:** Executes a developer-defined custom SQL statement as a child task within an existing Task DAG. The task is triggered after one or more predecessor tasks complete successfully, allowing arbitrary SQL (via Override SQL) to be chained into an existing workflow.
-   **Task DAG Resume Root:** A management node used to "start" or "enable" the entire chain of tasks. In Snowflake, tasks are created in a 'Suspended' state; this node ensures the pipeline is actually running.
-   **Stream:** Directly implements Snowflake's **Change Data Capture (CDC)**. It acts as a bookmark on a source table to track new Inserts, Updates, and Deletes without moving data itself.
-   **Stream and Insert or Merge:** A combined logic node. It reads the "delta" changes from a Stream and applies them to a target table using either a simple INSERT or a MERGE statement.
-   **Delta Stream Merge:** An advanced CDC node designed for high-precision synchronization. It uses the Stream’s metadata (like `METADATA$ACTION`) to ensure the target table perfectly mirrors the source, even when complex updates or deletes occur.
-   **Stream for Directory Table:** Specifically designed for **Unstructured Data**. It tracks changes to files (like PDFs or images) stored in a Snowflake Stage, allowing you to trigger processing as soon as a new file is uploaded.
-   **Insert or Merge with Task:** A general-purpose automation node. It encapsulates the SQL logic to update a target table and schedules it to run via a Task, either on a set interval or when data arrives in a stream.

**Summary:**
These nodes work together to create **Continuous Data Pipelines**. The **Stream** nodes capture the "Delta" (CDC), and the **Task** nodes provide the "Schedule" (Orchestration). By using these together, you achieve **Transactional Consistency**: data is only processed once, and the stream's offset only advances when the task successfully commits.

----

## Nodetypes Config Matrix

### Matrix 1: Task-Based Node Types - 1

| Category | Feature | Work with Task | Dimension with Task | Fact with Task | Insert or Merge with Task |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Development** | Development Mode | ✅ | ✅ | ✅ | ✅ |
| | Multi Source | ✅ | ✅ | ✅ | ✅ |
| | Create As | ⬜ | ⬜ | ⬜ | ✅ |
| **Logic** | Distinct / Group By All | ✅ | ✅ | ✅ | ✅ |
| | Order By | ✅ | ✅ | ✅ | ✅ |
| | Truncate Before | ✅ | ⬜ | ✅ | ✅ |
| **Keys** | Business / Table Keys | ⬜ | ✅ | ⬜ | ✅ |
| | Change Tracking (Type 2) | ⬜ | ✅ | ⬜ | ⬜ |
| | Record Date / Timestamp | ⬜ | ⬜ | ⬜ | ✅ |
| | Cluster Key | ✅ | ✅ | ✅ | ✅ |
| **Scheduling** | Warehouse / Serverless | ✅ | ✅ | ✅ | ✅ |
| | Stream Has Data Flag | ✅ | ✅ | ✅ | ✅ |
| | Multi-Stream Logic | ✅ | ✅ | ✅ | ⬜ |
| | Schedule (Min / Cron) | ✅ | ✅ | ✅ | ✅ |
| | Predecessor / Root Task | ✅ | ✅ | ✅ | ✅ |
| **Advanced Scheduling** | * Serverless Size* Size Bounds* User Task Timeout* Execute As user* Overlapping Execution* Task Graph Config* Auto-Suspend* Auto-Retry* Error Notifications | ✅ | ✅ | ✅ | ✅ |

### Matrix 1: Task-Based Node Types - 2

| Category | Feature | Task DAG Create Root | Task DAG custom-SQL | Task DAG Resume Root |
| :--- | :--- | :---: | :---: | :---: |
| **Development** | Development Mode | ⬜ | ✅ | ⬜ | 
| **Scheduling** | Warehouse / Serverless | ✅ | ✅ | ⬜ |
| | Stream Has Data Flag | ✅ | ✅ |⬜ |
| | Multi-Stream Logic | ✅ | ✅ | ⬜ |
| | Schedule (Min / Cron) | ✅ | ⬜ | ⬜ |
| | Predecessor / Root Task | ⬜ | ✅ | ✅ |
| **Advanced Scheduling** | * Serverless Size* Size Bounds* User Task Timeout* Execute As user* Overlapping Execution* Task Graph Config* Auto-Suspend* Auto-Retry* Error Notifications | ✅ | * User Task Timeout* Execute As user | ⬜ |
| **Logic** | Override/Custom-SQL | ✅ | ✅ | ⬜ |

### Matrix 2: Stream-based Node Types

| Category | Feature | Stream and Insert or Merge | Delta Stream Merge | Stream | Stream for Directory Table |
| :--- | :--- | :---: | :---: | :---: | :---: |
| **Source Support** | **Source: Table** | ✅ | ✅ | ✅ | ⬜ |
| | **Source: View** | ✅ | ✅ | ✅ | ⬜ |
| | **Source: Dynamic Table** | ✅ | ✅ | ✅ | ⬜ |
| | **Source: External Table** | ✅ | ✅ | ✅ | ⬜ |
| | **Source: Directory Table** | ⬜ | ⬜ | ⬜ | ✅ |
| **Development** | Development Mode | ✅ | ✅ | ⬜ | ⬜ |
| | Create As  | ✅ | ✅ | ⬜ | ⬜ |
| **Logic** | Distinct / Group By All | ✅ | ✅ | ⬜ | ⬜ |
| | Record Date / Timestamp | ✅ | ✅ | ⬜ | ⬜ |
| | Qualify Latest Record | ⬜ | ✅ | ⬜ | ⬜ |
| **Keys** | Business / Table Keys | ✅ | ✅ | ⬜ | ⬜ |
| | Cluster Key | ✅ | ✅ | ⬜ | ⬜ |
| **Stream Config** | Append Only Option | ✅ | ✅ | ✅ | ⬜ |
| | Show Initial Rows | ✅ | ✅ | ✅ | ⬜ |
| | Propagate Deletes | ✅ | ✅ | ⬜ | ⬜ |
| | Redeployment Behavior | ✅ | ✅ | ✅ | ✅ |
| **Scheduling** | Warehouse / Serverless | ✅ | ✅ | ⬜ | ⬜ |
| | Stream Has Data Flag | ✅ | ✅ | ⬜ | ⬜ |
| | Schedule (Min / Cron) | ✅ | ✅ | ⬜ | ⬜ |
| | Predecessor Task Logic | ✅ | ✅ | ⬜ | ⬜ |

---

The Coalesce Stream and Task Node Types Package includes:

* [Work with Task](#work-with-task)
* [Dimension with Task](#dimension-with-task)
* [Fact with Task](#fact-with-task)
* [Task DAG Create Root](#task-dag-create-root)
* [Task DAG Custom-SQL](#task-dag-custom-sql)
* [Task DAG Resume Root](#task-dag-resume-root)
* [Stream](#stream)
* [Stream and Insert or Merge](#stream-and-insert-or-merge)
* [Stream for Directory Table](#stream-for-directory-table)
* [Delta Stream Merge](#delta-stream-merge)
* [Insert Or Merge with Task](#Insert-Or-Merge-with-Task)
* [Code](#code)
---

## Work With Task

The Coalesce Work with Task UDN is a work node that wraps the standard Coalesce Work node with a Task.

Tasks can be combined with Coalesce Stream node (table streams) for continuous ELT workflows to process recently changed table rows. Streams ensure exactly once semantics for new or changed data in a table.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or performing other periodic work.

More information about Tasks can be found in [Snowflake Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Work with Task Node Configuration

The Work with Task node has two or three configuration groups depending on config options selected:

* [Node Properties](#work-with-task-properties)
* [Options](#work-with-task-options)
* [Scheduling Options](#work-with-task-scheduling-options) - Visible when Development Mode is false

#### Work With Task Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Work With Task Options

![Worktask-opt](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/2582e574-e368-4c00-8172-b259a86a4548)

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action.**True** - A table will be created and SQL will execute as a Run action**False** - After testing the SQL as a Run action, setting to false will wrap SQL in a task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **Multi Source** | True / False toggle that is Coalesce implementation of SQL UNIONs**True** - Multiple sources can be combined using:- UNION - Combines with duplicate elimination- UNION ALL - Combines without duplicate elimination**False** - Single source node or multiple sources combined using a join |
| **Cluster key** | True/False toggle that determines if clustering is enabled**True** - Specify clustering column and optionally allow expressions**False** - No clustering |
| **Truncate Before** | True / False toggle that determines if table should be truncated before insert**True** - Uses INSERT OVERWRITE**False** - Uses INSERT to append data |

#### Work with Task General Options

![Work with Task-GO](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/cb8cb62c-c4f1-4c22-87c2-3781a5328dfd)

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True/False toggle that determines whether to add DISTINCT to SQL Query**True** - Group by All is invisible. DISTINCT data is chosen**False** - Group by All is visible |
| **Group by All** | True/False toggle that determines whether to add GROUP BY ALL to SQL Query**True** - DISTINCT is invisible. Data grouped by all columns**False** - DISTINCT is visible |
| **Order By** | True/False toggle that determines whether to add ORDER BY to SQL Query**True** - Sort column and order options visible**False** - Sort options invisible |

#### Work with Task Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format*|
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Work with Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Work with Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

#### Example of Serverless Task with Multiple Predecessors and Root Task

![Example_of_Serverless_Task](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/0af0c04a-5e88-42d1-85de-fd71a792436d)

#### Example of Warehouse Task With 60 Minute Task Schedule

![Example_of_Warehouse_Task](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/04afc6b2-aebc-4346-b1fe-33340e7df0bf)

#### Example of Warehouse Task With Cron Schedule Not Using a Stream

![Example_of_Warehouse_Task_with_Cron](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/89c85551-59af-44e6-b383-a9bdaacf7cce)

### Work With Task Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Work With Task Initial Deployment

When deployed for the first time into an environment the Work with Task node will execute the following stages:

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Create Task** | Creates task that will load the target table on schedule |
| **Resume Task** | Resumes the task so it runs on schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called "Task DAG Resume Root." This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change.

#### Work With Task Redeployment

After initial deployment, changes in task schedule, warehouse, or scheduling options will result in a CREATE or RESUME

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Recreates task with new schedule |
| **Resume Task** | Resumes task with new schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Recreates task with new schedule |

#### Work With Task Altering Tables

Subsequent deployments with changes in table like add or drop column and change in data type will result in an ALTER table statement followed by CREATE TASK AND RESUME TASK statements being issued.

| **Stage** | **Description** |
|-----------|----------------|
| **Change Column Attributes/Delete Column/Add Column/Change table description** | Alter table statement is executed to perform the alter operation. |
| **Create Task** | Recreates task with new schedule |
| **Resume Task** | Resumes task with new schedule |

### Redeployment with only metadata changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates when the **Development Mode** is **ON**. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Work With Task Undeployment

If a Work with Task node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then all objects created by the node in the target environment will be dropped.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** |  Drop the table originally created to be loaded by the task. |
| **Drop Current Task** | Drop the task |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table |
| **Suspend Root Task** | Drop a task from a DAG of task the root task needs to be put into a suspended state. |
| **Drop Task** | Drops the task |

If a task is part of a DAG of tasks the DAG needs to include a node type called **Task Dag Resume Root**. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

---

## Dimension With Task

The Coalesce Dimension with Task UDN is a node that wraps the standard Coalesce Dimension node with a Task.

Tasks can be combined with Coalesce Stream node (table streams) for continuous ELT workflows to process recently changed table rows. Streams ensure exactly once semantics for new or changed data in a table.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or performing other periodic work.

More information about Tasks can be found in [Snowflake Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Dimension With Task Node Configuration

The Dimension with Task node has two or three configuration groups depending on config options selected:

* [Node Properties](#dimension-with-task-node-properties)
* [Options](#dimension-with-task-options)
* [General Options](#dimension-with-task-general-options)
* [Scheduling Options](#dimension-with-task-scheduling-options) - Visible when Development Mode is false

#### Dimension With Task Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Dimension With Task Options

![Dimension_task_opt](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/3dfa88ac-b46b-4e92-9639-d45e5a6c242e)

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action**True** - A table will be created and SQL will execute as a Run action**False** - After testing the SQL as a Run action, setting to false will wrap SQL in a task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **Multi Source** | True / False toggle that is Coalesce implementation of SQL UNIONs**True** - Multiple sources can be combined using:- UNION - Combines with duplicate elimination- UNION ALL - Combines without duplicate elimination**False** - Single source node or multiple sources combined using a join |
| **Business key** | Required column for both Type 1 and Type 2 Dimensions |
| **Change tracking** | Required column for Type 2 Dimension |
| **Cluster key** | True/False toggle that determines if clustering is enabled**True** - Specify clustering column and optionally allow expressions**False** - No clustering |

#### Dimension With Task General Options

![Work with Task-GO](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/cb8cb62c-c4f1-4c22-87c2-3781a5328dfd)

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True/False toggle that determines whether to add DISTINCT to SQL Query**True** - Group by All is invisible. DISTINCT data is chosen**False** - Group by All is visible |
| **Group by All** | True/False toggle that determines whether to add GROUP BY ALL to SQL Query**True** - DISTINCT is invisible. Data grouped by all columns**False** - DISTINCT is visible |
| **Order By** | True/False toggle that determines whether to add ORDER BY to SQL Query**True** - Sort column and order options visible**False** - Sort options invisible |

#### Dimension With Task Scheduling Options

If Development Mode is set to false, use Scheduling Options to configure how and when the task will run.

![Task Scheduling Options](https://github.com/user-attachments/assets/2f8bc6c3-117c-4130-9619-45ee50feebef)

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Dimension with Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Dimension with Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Dimension With Task Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Dimension With Task Initial Deployment

When deployed for the first time into an environment the Dimension with Task node will execute three stages dependent on whether or not the task schedule relies on a predecessor task.

For tasks without predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Create Task** | Creates task that will load the target table on schedule |
| **Resume Task** | Resumes the task so it runs on schedule |

For tasks with predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change.

### Dimension With Task Redeployment

After initial deployment, changes in task schedule, warehouse, or scheduling options will result in a CREATE or RESUME TASK.

For tasks without predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Recreates task with a new schedule |
| **Resume Task** | Resumes task with a new schedule |

For tasks with predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Recreates task with new schedule |

#### Dimension With Task Altering Tables

Subsequent deployments with changes in table such as add or drop column and change in data type will result in an ALTER table statement followed by CREATE TASK AND RESUME TASK statements being issued.

| **Stage** | **Description** |
|-----------|----------------|
| **Change Column Attributes/Delete Column/Add Column/Change table description** | Alter table statement is executed to perform the alter operation. |
| **Create Task** | Recreates task with new schedule |
| **Resume Task** | Resumes task with new schedule |

### Redeployment With Only Metadata Changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates when the **Development Mode** is **ON**. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Dimension With Task Undeployment

If a Work with Task node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then all objects created by the node in the target environment will be dropped.

For tasks without predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table originally created to be loaded by the task. |
| **Drop Current Task** | Drops the task |

For tasks with predecessor:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table originally created to be loaded by the task. |
| **Suspend Root Task** | Drop a task from a DAG of task the root task needs to be put into a suspended state. |
| **Drop Task** | Drop the task |

If a task is part of a DAG of tasks the DAG needs to include a node type called **Task Dag Resume Root**. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

---

## Fact With Task

The Coalesce Fact with Task UDN is a node that wraps the standard Coalesce Fact node with a Task.

Tasks can be combined with Coalesce Stream node (table streams) for continuous ELT workflows to process recently changed table rows. Streams ensure exactly once semantics for new or changed data in a table.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or performing other periodic work.

More information about Tasks can be found in [Snowflake Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Fact With Task Node Configuration

The Fact with Task node has two or three configuration groups depending on config options selected:

* [Node Properties](#fact-with-task-node-properties)
* [Options](#fact-with-task-options)
* [Scheduling Options](#fact-with-task-scheduling-options) - Visible when Development Mode is false

#### Fact With Task Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Fact With Task Options

![Dimension_task_opt](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/3dfa88ac-b46b-4e92-9639-d45e5a6c242e)

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action**True** - A table will be created and SQL will execute as a Run action**False** - After testing the SQL as a Run action, setting to false will wrap SQL in a task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **Multi Source** | True / False toggle that is Coalesce implementation of SQL UNIONs**True** - Multiple sources can be combined using:- UNION - Combines with duplicate elimination- UNION ALL - Combines without duplicate elimination**False** - Single source node or multiple sources combined using a join |
| **Cluster key** | True/False toggle that determines if clustering is enabled**True** - Specify clustering column and optionally allow expressions**False** - No clustering |
| **Truncate Before** | Specifies that the target table should be truncated before inserting the values into the table. |

#### Fact With Task General Options

![Work with Task-GO](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/cb8cb62c-c4f1-4c22-87c2-3781a5328dfd)

| **Option** | **Description** |
|------------|----------------|
| **Distinct** | True/False toggle that determines whether to add DISTINCT to SQL Query**True** - Group by All is invisible. DISTINCT data is chosen**False** - Group by All is visible |
| **Group by All** | True/False toggle that determines whether to add GROUP BY ALL to SQL Query**True** - DISTINCT is invisible. Data grouped by all columns**False** - DISTINCT is visible |
| **Order By** | True/False toggle that determines whether to add ORDER BY to SQL Query**True** - Sort column and order options visible**False** - Sort options invisible |

#### Fact With Task Scheduling Options

If Development Mode is set to false, use Scheduling Options to configure how and when the task will run.

![Task Scheduling Options](https://github.com/user-attachments/assets/2f8bc6c3-117c-4130-9619-45ee50feebef)

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **Multiple Stream has Data Flag** | True / False toggle that checks whether source streams have data before executing a task. **True**: Only run task if source stream has capture change data  **False**:  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false.|
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Fact with Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Fact with Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Fact With Task Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Fact With Task Initial Deployment

When deployed for the first time into an environment the Fact with Task node will execute three stages dependent on whether or not the task schedule relies on a predecessor task.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Create Task** | Creates task that will load the target table on schedule |
| **Resume Task** | Resumes the task so it runs on schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Target Table** | Creates table that will be loaded by the task |
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change.

#### Fact With Task Redeployment

After initial deployment, changes to task schedule, warehouse, or scheduling options will result in a CREATE and RESUME TASK.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Recreates task with new schedule |
| **Resume Task** | Resumes task with new schedule |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for DAG modification |
| **Create Task** | Creates task that will load the target table on schedule |

#### Fact With Task Altering the Tables

Changes to add or drop column, or change in data type will result in a ALTER, CREATE, AND RESUME TASK.

| **Stage** | **Description** |
|-----------|----------------|
| **Alter Table** | Modifies table structure |
| **Create Task** | Recreates task |
| **Resume Task** | Resumes updated task |

### Redeployment With Only Metadata Changes

Sometimes, changes to config can result in metadata changes from node edits, DML changes, or storage updates when the **Development Mode** is **ON**. A few cases are listed below:

1. Changes in business keys
2. Changes to change tracking keys
3. Changes in join clauses
4. Transformations made at column level
5. Changing DML options like DISTINCT, ORDER BY, GROUP BY ALL

And many more. Most of the time, specific ‘is’ and ‘was’ values will be displayed to specifically show what changed.

The following stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Metadata Update \| Business Keys \| Change Tracking \| Distinct \| Transformation \| Join** | A dummy statement would execute with specific changes listed in comments|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Fact With Task Undeployment

If a Fact with Task node is deleted from a Workspace, and that Workspace is committed to Git and that commit is deployed to a higher-level environment, then all objects created by the node in the target environment will be dropped.

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table created to be loaded by the task |
| **Drop Current Task** | Removes the task |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table created to be loaded by the task |
| **Suspend Root Task** | Suspends root task |
| **Drop Task** | Removes the task |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task Dag Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

---

## Task DAG Create Root

The Coalesce Task DAG Create Root UDN is a node that helps to create a standalone root task.

The root task should have a defined schedule that initiates a run of the DAG. Each of the other tasks has at least one defined predecessor to link the tasks in the DAG. A child task runs only after all of its predecessor tasks run successfully to completion.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or perform other periodic work.

More information about Tasks can be found in [Snowflake's Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Task DAG Create Root Node Configuration

The Task DAG Create Root node has two configuration groups:

* [Node Properties](#task-dag-create-root-node-properties)
* [Scheduling Options](#task-dag-create-root-scheduling-options)

#### Task DAG Create Root Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Task DAG Create Root Scheduling Options

![Task_dag_create_root_scheduling_options](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/79323e3a-4674-4676-9441-4497e1788680)

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task** - User managed warehouse executes tasks- **Serverless Task** - Uses serverless compute |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task.  Name of warehouse to run task on without quotes |
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task  Initial compute size for serverless tasks. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- Minutes - Specify interval in minutes- Cron - Use cron expression-Triggered task - To create task when the source streams have data |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Multiple source streams(if disabled,considered as single source stream)** |(visible obly for Triggered Task)Toggle- Enabled denotes multiple streams are connected|
| **Multiple Stream has Data Logic**| AND/OR logic when multiple streams (visible obly for Triggered Task and multiple streams is enabled)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data | 
| **Enter root task SQL** | The SQL statement to be run when a standalone root task executes |

#### Root with Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Root with Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Task DAG Create Root Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Task DAG Create Root Initial Deployment

When deployed for the first time into an environment, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for creation |
| **Create Root Task** | Creates task that will execute Root SQL on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task Dag Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

#### Task DAG Create Root Redeployment

After the Task has deployed for the first time into a target environment, subsequent deployments will execute two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task |
| **Create Root Task** | Recreates root task |

### Task DAG Create Root Undeployment

When a Task DAG Create Root node is deleted, two stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root task** | Suspends root task |
| **Drop current task** | Removes the task |

---

## Task DAG Custom-SQL

The Coalesce Task DAG Custom-SQL UDN is a node that creates a dependent task within an existing Task DAG.

The task is configured with one or more predecessor tasks and executes only after all predecessor tasks complete successfully. The SQL executed by the task is provided by the developer using the Override SQL property, allowing arbitrary SQL statements such as stored procedure calls, DML, or other custom operations to be incorporated into a Task DAG.

During deployment, the node follows the same suspend/resume behavior as other predecessor-based task nodes, allowing it to be added safely to an existing Task DAG.

More information about Tasks can be found in [Snowflake's Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Task DAG Custom-SQL Node Configuration

The Task DAG Custom-SQL node has two configuration groups:

* [Node Properties](#task-dag-custom-sql-node-properties)
* [Scheduling Options](#task-dag-custom-sql-scheduling-options)

#### Task DAG Custom-SQL Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Task DAG Custom-SQL Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if the SQL to be used in the task will execute as DML as a Run action**True** - Custom-SQL will execute as a Run action**False** - After testing the SQL as a Run action, setting to false will wrap SQL in a task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **Override Create SQL** | Toggle: True/False**True**: Custom Create SQL**False**: N/A Please ensure **toggle is always enabled** for this nodetype |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Multiple Stream has Data Logic** | AND/OR logic when multiple streams (visible if Stream has Data Flag is true)**AND** - If there are multiple streams task will run if all streams have data**OR** -  If there are multiple streams task will run if one or more streams has data | 
| **Scheduling Mode** | Choose compute type:- **Warehouse Task** - User managed warehouse executes tasks- **Serverless Task** - Uses serverless compute |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task.  Name of warehouse to run task on without quotes |
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task  Initial compute size for serverless tasks. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | - Predecessor - Specify dependent tasks |
| **Enter predecessor tasks separated by a comma**| One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma(,) |
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. |
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |

### Task DAG Custom-SQL Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Task DAG Custom-SQL Initial Deployment

When deployed for the first time into an environment, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task for creation |
| **Create Task** | Creates task that will execute custom-SQL on schedule |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task Dag Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

#### Task DAG Custom-SQL Redeployment

After the Task has deployed for the first time into a target environment, subsequent deployments will execute two stages:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root Task** | Suspends root task |
| **Create Root Task** | Recreates task |

### Task DAG Custom-SQL Undeployment

When a Task DAG Custom-SQL node is deleted, two stages are executed:

| **Stage** | **Description** |
|-----------|----------------|
| **Suspend Root task** | Suspends root task |
| **Drop current task** | Drops the task |

> **Note:** Preview Data is not supported for nodes created with this nodetype because the node exists only within Coalesce. The corresponding object created in Snowflake uses the _TASK suffix, so there is no matching object available for Preview Data.

---

## Task DAG Resume Root

The Coalesce Task DAG Resume Root UDN is a node type that helps to resume the root task and its dependents or child tasks. Recursively resumes all dependent tasks tied to a specified root task in a DAG using the root task name specified.

The root task should have a defined schedule that initiates a run of the DAG. Each of the other tasks has at least one defined predecessor to link the tasks in the DAG. A child task runs only after all of its predecessor tasks run successfully to completion.

Tasks can also be used independently to generate periodic reports by inserting or merging rows into a report table or perform other periodic work.

More information about Tasks can be found in [Snowflake's Introduction to tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).

### Task DAG Resume Root Node Configuration

The Task DAG Resume Root node has two configuration groups:

* [Node Properties](#task-dag-resume-root-node-properties)
* [Scheduling Options](#task-dag-resume-root-scheduling-options)

#### Task DAG Resume Root Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Task DAG Resume Root Scheduling Options

![Task_dag_resume_root_scheduling_options](https://github.com/coalesceio/Streams-and-Task-Nodes/assets/7216836/1dadf5e1-b846-4f2f-b09a-1a3722e60cdf)

| **Option** | **Description** |
|------------|----------------|
| **Enter root task name** | Name of the root task to be resumed - recursively resumes all dependent tasks tied to this specified root task |

### Task DAG Resume Root Deployment

#### Task DAG Resume Root Initial Deployment

When deployed for the first time into an environment the following stage executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Try Enable Root Task** | Resumes root task and all its dependents |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task Dag Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

#### Task DAG Resume Root Undeployment

If a Dimension with Task node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then two stages are executed.

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Removes the table |
| **Drop Current Task** | Removes the task |

---

## Stream

The Coalesce Stream UDN is a node that allows you to develop and deploy a stream on top of a table, view or external table.

A stream logically takes an initial snapshot of every row in the source object (e.g. table, external table, or the underlying tables for a view) by initializing a point in time (called an offset) as the current transactional version of the object. The change tracking system utilized by the stream then records information about the DML changes after this snapshot was taken. Change records provide the state of a row before and after the change. Change information mirrors the column structure of the tracked source object and includes additional metadata columns that describe each change event.

More information about Streams can be found in the official [Snowflake Introduction to Streams](https://docs.snowflake.com/en/user-guide/streams-intro).

### Stream Node Configuration

The Stream has two configuration groups:

* [Node Properties](#stream-node-properties)
* [Stream Options](#stream-options)

#### Stream Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Stream Options

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation:**Table**:**Append Only Stream**: True: Append-only stream False: Standard stream **Show Initial Rows**: Specify the records to return the first time the stream is consumed. **True**: The stream returns only the rows that existed in the source object at the moment when the stream was created. Subsequently, the stream returns any DML changes to the source object since the most recent offset - the normal stream behavior. **False**: The stream returns any DML changes to the source object since the most recent offset.**Redeployment Behavior**: Options for redeployment**Dynamic Table**:**Show Initial Rows**: Specify the records to return the first time the stream is consumed. **True**: The stream returns only the rows that existed in the source object at the moment when the stream was created. Subsequently, the stream returns any DML changes to the source object since the most recent offset - the normal stream behavior. **False**: The stream returns any DML changes to the source object since the most recent offset.**Redeployment Behavior**: Options for redeployment**View**:**Append Only Stream**: True: Append-only stream False: Standard stream **Show Initial Rows**: Specify the records to return the first time the stream is consumed.**True**: The stream returns only the rows that existed in the source object at the moment when the stream was created. Subsequently, the stream returns any DML changes to the source object since the most recent offset - the normal stream behavior. **False**: The stream returns any DML changes to the source object since the most recent offset.**Redeployment Behavior**: Options for redeployment**External table**:Redeployment Behavior: Options for redeployment**External iceberg table**:Redeployment Behavior: Options for redeployment |

### Stream System Columns

A Stream UDN adds three system columns to the output of the node. These columns can be used together to track INSERT, UPDATE and DELETE operations against a source object.

| **Column** | **Description** |
|------------|----------------|
| **METADATA$ACTION** | Indicates the DML operation (INSERT, DELETE) recorded |
| **METADATA$ISUPDATE** | Indicates whether the operation was part of an UPDATE statement. Updates to rows in the source object are represented as a pair of DELETE and INSERT records in the stream with a metadata column METADATA$ISUPDATE values set to TRUE.|
| **METADATA$ROW_ID** | Specifies the unique and immutable ID for the row, used to track changes over time |

### Stream Deployment

#### Stream Deployment Parameters

No deployment parameters are required.

#### Stream Initial Deployment

When deployed for the first time into an environment the Stream node executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Executes a CREATE OR REPLACE statement to create a Stream in the target environment |

#### Stream Redeployment

After initial deployment, subsequent deployments will create a new stream based on the selected redeployment behavior:

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| Create Stream if not exists | Re-Create Stream at existing offset |
| Create or Replace | Create Stream |
| Create at existing stream | Re-Create Stream at existing offset |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Stream Undeployment

When a Stream Node is deleted from a Workspace and that commit is deployed, the following stage executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Stream** | Removes the stream from the target environment |

---

## Stream and Insert or Merge

The Coalesce Streams and Insert or Merge UDN is a node that allows you to develop and deploy a stream on top of a table, view or external table. Also, provides option to create a target table to insert or merge data from source with a task on top of it.

A stream logically takes an initial snapshot of every row in the source object (e.g. table, external table, or the underlying tables for a view) by initializing a point in time (called an offset) as the current transactional version of the object. The change tracking system utilized by the stream then records information about the DML changes after this snapshot was taken. Change records provide the state of a row before and after the change. Change information mirrors the column structure of the tracked source object and includes additional metadata columns that describe each change event.

More information about Streams can be found in the official [Snowflake's Introduction to Streams](https://docs.snowflake.com/en/user-guide/streams-intro).

### Stream and Insert or Merge Node Configuration

The Stream and Insert or Merge node has the following configuration groups:

* [Node Properties](#stream-and-insert-or-merge-node-properties)
* [General Options](#stream-and-insert-or-merge-general-options)
* [Stream Options](#stream-and-insert-or-merge-stream-options)
* [Target Loading Options](#stream-and-insert-or-merge-target-loading-options)
* [Scheduling Options](#stream-and-insert-or-merge-scheduling-options)

#### Stream and Insert or Merge Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Stream and Insert or Merge General Options

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if SQL executes as DML**True** - Table created and SQL executes as Run action**False** - SQL wrapped in task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **CREATE AS** | Choose target object type:- Table - Permanent table with data retention and fail-safe- Transient Table - Temporary table without data retention |
| **DISTINCT** | True/False toggle for DISTINCT in SQL Query**True** - Group by All invisible, DISTINCT used**False** - Group by All visible |
| **GROUP BY ALL** | True/False toggle for GROUP BY ALL in SQL Query**True** - DISTINCT invisible, group by all columns**False** - DISTINCT visible |

#### Stream and Insert or Merge Stream Options

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation:**Table**:- Append Only Stream: True/False toggle for stream type- Show Initial Rows: True/False toggle for initial records- Propagate Deletes : True/False toggle for adding filter for condition METADATA$ACTION != 'DELETE' - Redeployment Behavior: Options for redeployment**Dynamic Table**:**Show Initial Rows**: Specify the records to return the first time the stream is consumed. **True**: The stream returns only the rows that existed in the source object at the moment when the stream was created. Subsequently, the stream returns any DML changes to the source object since the most recent offset - the normal stream behavior. **False**: The stream returns any DML changes to the source object since the most recent offset.**Redeployment Behavior**: Options for redeployment**View**:- Append Only Stream: True/False toggle for stream type- Show Initial Rows: True/False toggle for initial records- Redeployment Behavior: Options for redeployment |

#### Stream and Insert or Merge Target Loading Options

| **Option** | **Description** |
|------------|----------------|
| **Load Type** | Choose data loading method:**Insert** - Data inserted from source**Merge** - Latest record changes merged into target |
| **Table keys** | Business key columns for merging data (enabled for Merge load type) |
| **Record Date/Timestamp** | Date/Timestamp column for latest record merging (enabled for Merge load type) |
| **Cluster key** | True/False toggle for clustering**True** - Specify clustering column and expressions. - **Allow Expressions Cluster Key**: Add an expression to the specified cluster key.**False** - No clustering |

#### Stream and Insert or Merge Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Stream and Insert or Merge Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Stream and Insert or Merge Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Stream and Insert or Merge System Columns

| **Column** | **Description** |
|------------|----------------|
| **METADATA$ACTION** | Indicates the DML operation (INSERT, DELETE) recorded |
| **METADATA$ISUPDATE** | Indicates whether operation was UPDATE (shown as DELETE/INSERT pair with TRUE value) |
| **METADATA$ROW_ID** | Unique and immutable row ID for change tracking |

**NOTE: In the WHERE clause, always use the original column name, not with an alias of table name, because aliases are only recognized in the SELECT clause and cannot be used for filtering.**

### Stream and Insert or Merge Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Stream and Insert or Merge Initial Deployment

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Creates Stream in target environment |
| **Create Work Table/Transient Table** | Creates table loaded by task |
| **Target Table Initial Load** | Loads initial data |
| **Create Task** | Creates scheduled task |
| **Resume Task** | Enables task execution |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Creates Stream in target environment |
| **Create Work Table/Transient Table** | Creates target table |
| **Target Table Initial Load** | Loads initial data |
| **Suspend Root Task** | Suspends root task |
| **Create Task** | Creates scheduled task |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change

#### Stream and Insert or Merge Redeployment

Stream redeployment behavior:

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| **Create Stream if not exists** | Re-Create Stream at existing offset |
| **Create or Replace** | Create Stream |
| **Create at existing stream** | Re-Create Stream at existing offset |

Table changes execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/Alter Column/Delete Column/Add Column/Edit description** | Alters table as needed |
| **Target Initial Load** | If the initial load toggle is enabled and the redeployment behavior of the stream is "Create or Replace," it loads the table with "INSERT OVERWRITE INTO." For all other scenarios, it uses "INSERT INTO." |

If the materialization type is changed from one type to another(transient table/table) the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table/Transient Table** | Drop the target table|
| **Create Work/Transient table**| Create the target table|
| **Target Initial Load** | If the initial load toggle is enabled and the redeployment behavior of the stream is "Create or Replace," it loads the table with "INSERT OVERWRITE INTO." For all other scenarios, it uses "INSERT INTO." |

Task changes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Creates scheduled task |
| **Resume Task**| Resumes the task|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Stream and Insert or Merge Undeployment

When node is deleted, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Stream** | Removes the stream |
| **Drop Table** | Drop the table |
| **Drop Current Task** | Drop the task |

---

## Stream for Directory Table

The Coalesce Stream for Directory Table Node Type is a node that allows you to develop and deploy a stream on top of a directory table

More information about Streams can be found in the official [Snowflake Introduction to Streams](https://docs.snowflake.com/en/user-guide/streams-intro).

### Stream for Directory Table Node Configuration

The Stream has two configuration groups:

* [Node Properties](#stream-node-properties)
* [Stream Options](#stream-options)

#### Stream Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Stream Options

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation:**Directory Table****Storage Location**:  Storage location of the stage for which directory table is enabled **Stage Name**: Stage name of the directory table.**Redeployment Behavior**: Options for redeployment

### Stream for Directory Table System Columns

A Stream for Directory Table adds nine system columns to the output of the node. These columns can be used together to track INSERT, UPDATE and DELETE operations against a source object.

| **Column** | **Description** |
|------------|----------------|
| **RELATIVE_PATH** | The relative path of the file within the stage.
| **SIZE** | The file size in bytes.
| **LAST_MODIFIED** | The timestamp of the last modification of the file.
| **MD5** | The MD5 hash of the file content for integrity verification.
| **ETAG** | The entity tag (ETag) used for cache validation and versioning.
| **FILE_URL** | The full URL of the file in the stage.
| **METADATA$ACTION** | Indicates the DML operation (INSERT, DELETE) recorded |
| **METADATA$ISUPDATE** | Indicates whether the operation was part of an UPDATE statement. Updates to rows in the source object are represented as a pair of DELETE and INSERT records in the stream with a metadata column METADATA$ISUPDATE values set to TRUE.|
| **METADATA$ROW_ID** | Specifies the unique and immutable ID for the row, used to track changes over time |

### Stream for Directory Table Deployment

#### Stream for Directory Table Deployment Parameters

No deployment parameters are required.

#### Stream for Directory Table Initial Deployment

When deployed for the first time into an environment the Stream node executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Executes a CREATE OR REPLACE statement to create a Stream in the target environment |

#### Stream for Directory Table Redeployment

After initial deployment, subsequent deployments will create a new stream based on the selected redeployment behavior:

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| Create Stream if not exists | Re-Create Stream at existing offset |
| Create or Replace | Create Stream |
| Create at existing stream | Re-Create Stream at existing offset |

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Stream for Directory Table Undeployment

When a Stream Node is deleted from a Workspace and that commit is deployed, the following stage executes:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Stream** | Removes the stream from the target environment |

---

## Delta Stream Merge

The Coalesce Delta Stream Merge UDN is a node that allows you to develop and deploy a stream on top of a table, view ,dynamic table,external iceberg and external table. Also, provides option to create a target table to merge data handling inserts,deletes and updates from source with a task on top of it.

A stream logically takes an initial snapshot of every row in the source object (e.g. table, dynamic table, or the underlying tables for a view) by initializing a point in time (called an offset) as the current transactional version of the object. The change tracking system utilized by the stream then records information about the DML changes after this snapshot was taken. Change records provide the state of a row before and after the change. Change information mirrors the column structure of the tracked source object and includes additional metadata columns that describe each change event.

More information about Streams can be found in the official [Snowflake's Introduction to Streams](https://docs.snowflake.com/en/user-guide/streams-intro).

### Delta Stream Merge Node Configuration

The Stream and Insert or Merge node has the following configuration groups:

* [Node Properties](#stream-and-insert-or-merge-node-properties)
* [General Options](#stream-and-insert-or-merge-general-options)
* [Stream Options](#stream-and-insert-or-merge-stream-options)
* [Target Loading Options](#stream-and-insert-or-merge-target-loading-options)
* [Scheduling Options](#stream-and-insert-or-merge-scheduling-options)

#### Delta Stream Merge Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Stream will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Delta Stream Merge General Options

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if SQL executes as DML**True** - Table created and SQL executes as Run action**False** - SQL wrapped in task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **CREATE AS** | Choose target object type:- Table - Permanent table with data retention and fail-safe- Transient Table - Temporary table without data retention |
| **DISTINCT** | True/False toggle for DISTINCT in SQL Query**True** - Group by All invisible, DISTINCT used**False** - Group by All visible |
| **GROUP BY ALL** | True/False toggle for GROUP BY ALL in SQL Query**True** - DISTINCT invisible, group by all columns**False** - DISTINCT visible |

#### Delta Stream Merge Stream Options

| **Option** | **Description** |
|------------|----------------|
| **Source Object** | Type of object for stream creation:**Table**:- Append Only Stream: True/False toggle for stream type- Show Initial Rows: True/False toggle for initial records- Propagate Deletes : True/False toggle for adding filter for condition METADATA$ACTION != 'DELETE' - Redeployment Behavior: Options for redeployment**Dynamic Table**:**Show Initial Rows**: Specify the records to return the first time the stream is consumed. **True**: The stream returns only the rows that existed in the source object at the moment when the stream was created. Subsequently, the stream returns any DML changes to the source object since the most recent offset - the normal stream behavior. **False**: The stream returns any DML changes to the source object since the most recent offset.**Redeployment Behavior**: Options for redeployment**View**:- Append Only Stream: True/False toggle for stream type- Show Initial Rows: True/False toggle for initial records- Redeployment Behavior: Options for redeployment**External Table**:- Show Initial Rows: True/False toggle for initial records- Redeployment Behavior: Options for redeployment**External Table**:- Show Initial Rows: True/False toggle for initial records- Redeployment Behavior: Options for redeployment |

#### Delta Stream Merge Target Loading Options

| **Option** | **Description** |
|------------|----------------|
| **Table keys** | Business key columns for merging data (enabled for Merge load type) |
| **Record Date/Timestamp** | Date/Timestamp column for latest record merging (enabled for Merge load type) |
| **Qualifies selection based on Table Key and Record Date / Timestamp** | True/False toggle for clustering**True** - Adds qualify keyword to add only latest records from source |
| **Cluster key** | True/False toggle for clustering**True** - Specify clustering column and expressions. - **Allow Expressions Cluster Key**: Add an expression to the specified cluster key.**False** - No clustering |

#### Delta Stream Merge Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

**NOTE: In the WHERE clause, always use the original column name, not with an alias of table name, because aliases are only recognized in the SELECT clause and cannot be used for filtering.**

#### Delta Stream Merge Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Delta Stream Merge Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Delta Stream Merge Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Delta Stream Merge Initial Deployment

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Creates Stream in target environment |
| **Create Work Table/Transient Table** | Creates table loaded by task |
| **Target Table Initial Load** | Loads initial data |
| **Create Task** | Creates scheduled task |
| **Resume Task** | Enables task execution |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Stream** | Creates Stream in target environment |
| **Create Work Table/Transient Table** | Creates target table |
| **Target Table Initial Load** | Loads initial data |
| **Suspend Root Task** | Suspends root task |
| **Create Task** | Creates scheduled task |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change

#### Delta Stream Merge Redeployment

Stream redeployment behavior:

| **Redeployment Behavior** | **Stage Executed** |
|--------------------------|-------------------|
| **Create Stream if not exists** | Re-Create Stream at existing offset |
| **Create or Replace** | Create Stream |
| **Create at existing stream** | Re-Create Stream at existing offset |

**NOTE:Column Schema Changes results in recreating stream at existing offset irrespective of redeployment beahviour**

Table changes execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/Alter Column/Delete Column/Add Column/Edit description** | Alters table as needed |
| **Target Initial Load** | If the initial load toggle is enabled and the redeployment behavior of the stream is "Create or Replace," it loads the table with "INSERT OVERWRITE INTO." For all other scenarios, it uses "Dual Merge statement" |

**NOTE:Column name modification/addition/deletion should be made to mapping grid of DSM node only if the same are done in Upstream source node as here stream is the source for target table.Hence the changes made in mapping grid are not considered**

If the materialization type is changed from one type to another(transient table/table) the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table/Transient Table** | Drop the target table|
| **Create Work/Transient table**| Create the target table|
| **Target Initial Load** | If the initial load toggle is enabled and the redeployment behavior of the stream is "Create or Replace," it loads the table with "INSERT OVERWRITE INTO." For all other scenarios, it uses "Dual Merge statement" |

Task changes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Creates scheduled task |
| **Resume Task**| Resumes the task|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Delta Stream Merge Undeployment

When node is deleted, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Stream** | Removes the stream |
| **Drop Table** | Drop the table |
| **Drop Current Task** | Drop the task |

---

## Insert or Merge with Task

The Coalesce Insert or Merge with Task UDN is a node that allows you to create a target table to insert or merge data from source with a task on top of it.

### Insert or Merge with Task Node Configuration

The Insert or Merge with Task node has the following configuration groups:

* [Node Properties](#insert-or-merge-with-task-node-properties)
* [General Options](#insert-or-merge-with-task-general-options)
* [Target Loading Options](#insert-or-merge-with-task-target-loading-options)
* [Scheduling Options](#insert-or-merge-with-task-scheduling-options)

#### Insert or Merge with Task Node Properties

| **Property** | **Description** |
|-------------|-----------------|
| **Storage Location** | Storage Location where the Task will be created |
| **Node Type** | Name of template used to create node objects |
| **Description** | A description of the Node's purpose |
| **Deploy Enabled** | If TRUE the node will be deployed or redeployed when changes are detectedIf FALSE the node will not be deployed or will be dropped during redeployment |

#### Insert or Merge with Task General Options

| **Option** | **Description** |
|------------|----------------|
| **Development Mode** | True / False toggle that determines whether a task will be created or if SQL executes as DML**True** - Table created and SQL executes as Run action**False** - SQL wrapped in task with specified Scheduling Options. When Run is executed, a message appears prompting the user to wait or suggesting a manual run. |
| **CREATE AS** | Choose target object type:- Table - Permanent table with data retention and fail-safe- Transient Table - Temporary table without data retention |
| **Cluster key** | True/False toggle for clustering**True** - Specify clustering column and expressions. - **Allow Expressions Cluster Key**: Add an expression to the specified cluster key.**False** - No clustering |

#### Insert or Merge with Task Target Loading Options

| **Option** | **Description** |
|------------|----------------|
| **Load Type** | Choose data loading method:**Insert** - Data inserted from source**Merge** - Latest record changes merged into target |
| **Table keys** | Business key columns for merging data (enabled for Merge load type) |
| **Record Date/Timestamp** | Date/Timestamp column for latest record merging (enabled for Merge load type) |
| **Multi Source** | True / False toggle that is Coalesce implementation of SQL UNIONs**True** - Multiple sources can be combined using:- UNION - Combines with duplicate elimination- UNION ALL - Combines without duplicate elimination**False** - Single source node or multiple sources combined using a join |
| **DISTINCT** | True/False toggle for DISTINCT in SQL Query**True** - Group by All invisible, DISTINCT used**False** - Group by All visible |
| **GROUP BY ALL** | True/False toggle for GROUP BY ALL in SQL Query**True** - DISTINCT invisible, group by all columns**False** - DISTINCT visible |
| **Order By** | True/False toggle that determines whether to add ORDER BY to SQL Query**True** - Sort column and order options visible**False** - Sort options invisible |
| **Truncate Before** | True / False toggle that determines if table should be truncated before insert (enabled for Insert load type)**True** - Uses INSERT OVERWRITE**False** - Uses INSERT to append data |

#### Insert or Merge with Task Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **Scheduling Mode** | Choose compute type:- **Warehouse Task**: User managed warehouse executes tasks- **Serverless Task**: Uses serverless compute |
| **When Source Stream has Data Flag** | True/False toggle to check for stream data**True** - Only run task if source stream has capture change data**False** -  Run task on schedule regardless of whether the source stream has data. If the source is not a stream should set this to false. |
| **Select Warehouse** | Visible if Scheduling Mode is set to Warehouse Task. Enter the name of the warehouse you want the task to run on without quotes.|
| **Select initial serverless size** | Visible when Scheduling Mode is set to Serverless Task. Select the initial compute size on which to run the task. Snowflake will adjust size from there based on target schedule and task run times. |
| **Enable Size Bounds** | Toggle to set explicit limits on serverless scaling. (Visible if **Serverless Task** is selected).**Validation Rules:**- Min size must be ≤ Initial size- Max size must be ≥ Initial size- Min size must be ≤ Max size |
| **Minimum Warehouse Size** | The smallest compute size allowed for the task (e.g., 1. XSMALL). |
| **Maximum Warehouse Size** | The largest compute size allowed for the task (e.g., 6. XXLARGE). |
| **Task Schedule** | Choose schedule type:- **Minutes** - Specify interval in minutes. Enter a whole number from 1 to 11520 which represents the number of minutes between task runs.- **Cron** - Uses [Cron expressions](https://docs.coalesce.io/docs/reference/cron-reference/). Specifies a cron expression and time zone for periodically running the task. Supports a subset of standard cron utility syntax.- **Predecessor** - Specify dependent tasks |
| **Execution Time** | The specific duration for the task run limit. Supported ranges:- **SECONDS**: 10 - 691200- **MINUTES**: 1 - 11520- **HOURS**: 1 - 192 *Note: For upgrades from version 2.4.3 or earlier, ensure the scheduling configuration is manually updated to align with the new tabular input format* |
| **Enter predecessor tasks separated by a comma**| Visible when Task Schedule is set to Predecessor. One or more task names that precede the task being created in the current node. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|
| **Root task name** | Visible when Task Schedule is set to Predecessor. Name of the root task that controls scheduling for the DAG of tasks. Task names are case sensitive, should not be quoted and must exist in the same schema in which the current task is being created. If there are multiple predecessor task separate the task names using a comma and no spaces.|

#### Insert or Merge with Task Advanced Scheduling Options

| **Option** | **Description** |
|------------|----------------|
| **User Task Timeout** | Specifies the time limit on a single run of the task before it times out (in milliseconds)Values: 0 - 604800000 (7 days). A value of 0 specifies that the maximum timeout value is enforced |
| **Execute As Specific User** | Toggle to run on behalf of another user. Requires `GRANT IMPERSONATE` privileges. |
| **User Name** | The specific user account name used when **Execute As Specific User** is enabled. |
| **Allow Overlapping Execution** | Allows a new instance of the task to start if the previous one is still running. |
| **Enable Task Graph Config** | Enables a text box to provide **Configuration JSON** for the task graph. |
| **Auto-Suspend After Failures** | Automatically suspends the task after a set number of consecutive failures. |
| **Number of Consecutive Failures** | Set the threshold (0 - No Limit) before the task is automatically suspended. - When toggle is OFF: Parameter is not included (uses Snowflake default of 10).- When toggle is ON with value 0: **Disables** auto-suspension.- When toggle is ON with value > 0: **Suspends** after that many consecutive failures. |
| **Enable Auto-Retry** | Toggle to automatically retry the task if it fails. |
| **Retry Attempts** | Specify the number of retry attempts allowed (Range: 0 - 30). |

#### Insert or Merge with Task Notification Options

| **Option** | **Description** |
|------------|----------------|
| **Enable Error Notifications** | Toggle to send alerts on failure. Requires an **Error Integration Name**. |
| **Enable Success Notifications** | Toggle to send alerts on success. Requires a **Success Integration Name**. |

> **Note:** Options under **Advanced Scheduling Options** and **Notification Options** (Execution Time, Overlapping Execution, Auto-Suspend, Auto-Retry, etc.) are only applicable to **Root** and **Independent** tasks. The only exception is **Execute As Specific User**, which can be configured for any task in the graph.

### Insert or Merge with Task Deployment

refer to [this section](#Prerequisites-to-Use-Task-Scheduling-Options) for more details on the prerequisites required to set up tasks.

#### Insert or Merge with Task Initial Deployment

For tasks without predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table/Transient Table** | Creates table loaded by task |
| **Create Task** | Creates scheduled task |
| **Resume Task** | Enables task execution |

For tasks with predecessors:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Work Table/Transient Table** | Creates target table |
| **Suspend Root Task** | Suspends root task |
| **Create Task** | Creates scheduled task |

If a task is part of a DAG of tasks, the DAG needs to include a node type called `Task DAG Resume Root`. This node will resume the root node once all the dependent tasks have been created as part of a deployment.

The task node has no ALTER capabilities. All task-enabled nodes are CREATE OR REPLACE only, though this is subject to change

#### Insert or Merge with Task Redeployment

Table changes execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Rename Table/Alter Column/Delete Column/Add Column/Edit description** | Alters table as needed |

If the materialization type is changed from one type to another(transient table/table) the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table/Transient Table** | Drop the target table|
| **Create Work/Transient table**| Create the target table|

Task changes:

| **Stage** | **Description** |
|-----------|----------------|
| **Create Task** | Creates scheduled task |
| **Resume Task**| Resumes the task|

#### Node Type Switching

Node Type switching is supported starting from Coalesce version **7.28+**.

From this version onward, a node’s materialization type can be switched from one supported type to another, subject to certain limitations.

For more information, see [Node Type Switching Logic and Limitations](#node-type-switching-logic)

### Insert or Merge with Task Undeployment

When node is deleted, the following stages execute:

| **Stage** | **Description** |
|-----------|----------------|
| **Drop Table** | Drop the table |
| **Drop Current Task** | Drop the task |

-----------------

#### Node Type Switching Logic
| Current MaterializationType | Desired MaterializationType | Stage |
|------------|--------|-------|
| Task | Task | Follows existing redeployment stages |
| Stream | Stream | Follows existing redeployment stages |
| Table | Task | 1. Warning (if applicable)2. Alter |
| Any Other | Task | 1. Warning (if applicable)2. Drop  3. Create |
| Any Other | Stream | 1. Warning (if applicable)2. Drop  3. Create |

Review the documented limitations before performing a node type switch to ensure compatibility and avoid unintended deployment issues.

#### ⚠ Limitations of Node Type Switching (Current)

| # | Current Materialization | Desired Materialization | Limitation |
|---|--------------------------|--------------------------|------------|
| 1 | Older Version Iceberg Table | Table | Results in `ALTER` failure. Iceberg tables require `ALTER ICEBERG TABLE`. Works only if latest package (with switching support) is already used. |
| 2 | Older VersionCreate or Alter-ViewData Quality-DMF | Any(except View) | Switch fails unless current node uses latest package supporting node type switching. |
| 3 | First Node in Pipeline | Any | Not supported. First node is foundational and switching may disrupt the pipeline. |
| 4 | External Packages | Any | Not supported as they typically act as first nodes in the pipeline. |
| 5 | Functional Packages | Any | Not supported due to column re-sync behavior which may cause schema inconsistencies. |
| 6 | Dynamic Dimension / LRV | Any | System columns must be manually dropped before redeployment. |
| 7 | Any | Any Other | After performing node switching, the `Create/Run` in Workspace browser may not work as expected due to changes in the node’s materialization type. |
| 8 | Table(Data Profiling) | Table | This may result in ALTER failure unless latest package is used(with system column removal support)**(Pending Release)** |
| 9 | Any | Any Stream-based Node (Stream, Stream & I/M, Delta Merge, or Directory Stream) | When switching to a Stream-based node, do not select **'Create At Existing Stream'** from the Redeployment Behavior; this causes deployment errors. Use **'Create or Replace'** or **'Create If Not Exists'**. |
| 10 | Stream | Stream for Directory Table (and vice versa) | Metadata columns are not automatically synchronized. Specific directory columns (e.g., `relative_path`, `size`, `md5`) are not added when switching to Directory Table, nor are they removed when switching back to a standard Stream. |
| 11 | Stream | Any Other (and vice versa) | Snowflake CDC metadata columns (`METADATA$ACTION`, `METADATA$ISUPDATE`, `METADATA$ROW_ID`) are not automatically managed. They are neither removed nor added when there's a node type switch |

--------------
## Prerequisites to Use Task Scheduling Options

### 1. Task Deployment Parameters

The task includes an environment parameters that allows you to specify a different warehouse used to run a task in different environments.

| Parameter Name                 | Config Value       | Refers To / Used From                                      | Behavior                                                                 |
|--------------------------------|--------------------|-------------------------------------------------------------|--------------------------------------------------------------------------|
| targetTaskWarehouse            | DEV ENVIRONMENT    | Scheduling Options → Select Warehouse                       | Uses warehouse selected in UI                                            |
| targetTaskWarehouse            | COMPUTE_WH         | Explicit warehouse name                                     | Uses COMPUTE_WH directly                                                 |
| targetTaskErrorIntegration     | DEV ENVIRONMENT    | Notification Options → Error Integration                    | Uses error integration from UI                                           |
| targetTaskErrorIntegration     | &lt;err_int>          | Explicit Error Integration                                  | Uses &lt;err_int> directly                                                  |
| targetTaskSuccessIntegration   | DEV ENVIRONMENT    | Notification Options → Success Integration                  | Uses success integration from UI                                         |
| targetTaskSuccessIntegration   | &lt;suc_int>          | Explicit Success Integration                                | Uses &lt;suc_int> directly                                                  |
| targetTaskExecuteAsUser        | DEV ENVIRONMENT    | Advanced Scheduling Options → Execute As                    | Uses user defined in UI                                                  |
| targetTaskExecuteAsUser        | &lt;execute_as>       | Explicit Execute As                                         | Uses &lt;execute_as> directly                                               |

### 2. Using EXECUTE AS USER

You must have:

- `ACCOUNTADMIN` role (recommended)
 
```sql
GRANT IMPERSONATE ON USER <user_name> TO ROLE SYSADMIN;
```

### 3. Create the Notification Integration
```sql
CREATE OR REPLACE NOTIFICATION INTEGRATION my_sns_integration
  TYPE = QUEUE
  NOTIFICATION_PROVIDER = AWS_SNS
  ENABLED = TRUE
  AWS_SNS_TOPIC_ARN = '<your_sns_topic_arn>'
  AWS_SNS_ROLE_ARN = '<your_iam_role_arn>';
```

Please refer [this documentation](https://docs.snowflake.com/en/user-guide/notifications/about-notifications) for more info about notifications.

-------------

## Code

### Work With Task Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/run.sql.j2)

### Dimension With Task

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DimensionwithTask-149/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DimensionwithTask-149/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DimensionwithTask-149/run.sql.j2)

### Fact With Task

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/FactwithTask-150/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/FactwithTask-150/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/FactwithTask-150/run.sql.j2)

### Task DAG Create Root Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGCreateRoot-154/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGCreateRoot-154/create.sql.j2)

### Task DAG Custom-SQL Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGCustom-SQL-677/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGCustom-SQL-677/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGCustom-SQL-677/run.sql.j2)

### Task DAG Resume Root Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGResumeRoot-155/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/TaskDAGResumeRoot-155/create.sql.j2)

### Stream Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/Stream-153/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/Stream-153/create.sql.j2)

### Stream and Insert or Merge Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/StreamandInsertorMerge-152/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/StreamandInsertorMerge-152/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/StreamandInsertorMerge-152/run.sql.j2)

### Stream for Directory Table

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/WorkwithTask-151/run.sql.j2)

### Delta Stream Merge

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DeltaStreamMerge-483/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DeltaStreamMerge-483/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/DeltaStreamMerge-483/run.sql.j2)

### Insert OR Merge with Task Code

* [Node definition](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/InsertorMergewithTask-567/definition.yml)
* [Create Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/InsertorMergewithTask-567/create.sql.j2)
* [Run Template](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/nodeTypes/InsertorMergewithTask-567/run.sql.j2)

[Macros](https://github.com/coalesceio/Streams-and-Task-Nodes/blob/main/macros/macro-1.yml)

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 2.7.2 | July 01, 2026 | Added Task Time Out support for all the tasks in the package |
| 2.7.1 | June 26, 2026 | Task DAG Custom-SQL: Comment fix and Task Time out addition |
| 2.7.0 | June 23, 2026 | Introduced a new node type for predecessor-based custom-SQL execution and minor bug fixes. |
| 2.6.1 | April 15, 2026 | Task Node Enhancements - Minor bug fixes |
| 2.6.0 | March 17, 2026 | Task Node Enhancements - Notification Integrations, Execute As User, and Advanced Task Parameters |
| 2.4.3 | March 05, 2026 | Added Node Type Switch support for Streams and Tasks package |
| 2.4.2 | November 11, 2025 | Empty SQL Statement issue fixed |
| 2.4.1 | October 07, 2025 | Triggered Task option to Task DAG Create Root, Bug fix for SIM and DSM |
| 2.4.0 | September 23, 2025 | Added new Node Type - Insert Or Merge with Task (TIM) |
| 2.3.0 | September 04, 2025 | Metadata Refresh during deployment - Added |
| 2.2.0 | August 22, 2025 | Delta Stream Merge-Dual Merge Logic |
| 2.1.0 | August 07, 2025 | Adding deploy phases in appropriate stage functions |
| 2.0.3 | July 29, 2025 | Delta Stream Merge,Stream Insert or Merge bug fix |
| 2.0.2 | July 11, 2025 | Skipping create date in update for Delta Stream Merge |
| 2.0.1 | July 02, 2025 | System Create date and Update date added to SIM and DSM node types |
| 2.0.0 | June 23, 2025 | Removal of deploy phases in stage function across all node types |
| 1.4.2 | June 12, 2025 | ref function update for Stream and task package |
| 1.4.1 | May 12, 2025 | New node type -Delta Stream Merge added |
| 1.4.0 | May 12, 2025 | New node type Delta Stream Merge |
| 1.3.2 | March 17, 2025 | Added warning while deployment if parameter is missing for warehouse name |
| 1.3.1 | February 28, 2025 | Fix for Column Description and Removed Nothing To Do stges |
| 1.3.0 | February 06, 2025 | Added a Node Type Stream for Directory Node.
Added a toggle 'Propagate Delete' in Stream Node Type. |
| 1.2.0 | January 14, 2025 | Added support of Dynamic Table as a Source to Node Type 1. Stream 2. Insert or Merge Stream |
| 1.1.5 | November 08, 2024 | Fix for typo error and single namespace variable initialization |
| 1.1.4 | July 31, 2024 | New node type Iceberg table with Task added |
| 1.1.3 | July 29, 2024 | Fix for empty current storage location |
| 1.1.2 | July 08, 2024 | Support for external iceberg tables in Stream node type |
| 1.1.1 | June 28, 2024 | Stream Insert or Merge materialization type added |
| 1.1.0 | June 13, 2024 | Default options modified |

---

## Test Utility

<!-- DO NOT EDIT MARKETPLACE LISTINGS DIRECTLY -->
<!-- THIS LISTING IS RENDERED BY A TEMPLATE -->
<!-- markdownlint-disable -->
<!-- vale off -->

‹ View all packages

Package ID: <code class="package-identifier">@coalesce/snowflake/test-utility</code> <CopyButton textToCopy='@coalesce/snowflake/test-utility' />
Supported Platform: 
Latest Version: 1.1.1 (July 28, 2025)

assertionscolumn testdata qualitydata testinggreat expectationsnode testnull checkrow counttestinguniquenessvalidation

## Overview
This is an extension package for Coalesce, inspired by the Great Expectations package for Python. The intent is to allow Coalesce users to deploy GE-like tests in their data warehouse.

## Installation

1. Copy the Package ID: <code class="package-identifier">@coalesce/snowflake/test-utility</code> <CopyButton textToCopy='@coalesce/snowflake/test-utility' />
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

## Description
test-utility

        

## About

`test-utility` is an extension package for [**Coalesce**], inspired by the [Great Expectations package for Python](https://greatexpectations.io/). The intent is to allow Coalesce users to deploy GE-like tests in their data warehouse directly from Coalesce, vs having to add another integration with their data warehouse.

## How to use?

Step 1: Once you install this package from Coalesce marketplace then you will need to import this package to workspace macro.

Assuming the alias you gave the package is TestUtility you would add the below Jinja.  `testUtils` could be anything.  You will use this alias to reference test macros in the imported package.

```yaml
{% import "TestUtility" as testUtils with context %}
```

![image](https://github.com/user-attachments/assets/62d6075d-f1cb-4b90-818e-b2a1c79285dc)

Step 2: Open a Node for which you want to create a test case.

Step 3: Goto Testing Configuration.

Step 4: Click on 'New Test' button.

![image](https://github.com/user-attachments/assets/dcc56a4b-5f89-49e8-be24-ee0deaa6a099)

Step 5: You will see new test case added for the Node.

![image](https://github.com/user-attachments/assets/aab2130d-9bbe-43f0-91b5-b1ae592bf842)

Step 6: In the text field, call the macro using the package import alias followed by a dot and the test case name.
        
_For Example, I am trying to Run Test case, 'expect_table_row_count_to_be_between' from the avilable test below._
        
![image](https://github.com/user-attachments/assets/29febbe9-ffe4-446a-8c1c-00cd0c438d1e)

Step 7: Replace the input parameters in macro call as per requirment.

_In this test case 'group_by' and 'filterCondition' inputs are optional, so i am ignoring here._

![image](https://github.com/user-attachments/assets/febcfc74-68dd-4447-a28a-94fb6b350935)

Note - You can refer object name with all the avilable pattern in Coalesce.  For example, [ref_no_link()](https://docs.coalesce.io/docs/reference/ref-functions/#ref_no_link) may be used instead of [this](https://docs.coalesce.io/docs/reference/ref-functions/#this).

Step 8: You can execute this test case using 'Run' button.

## Example

Lets consider, i have one table with named, TEST_TABLE. 

1. I want to check if each column value to be in a given set.
   _I will refer test case expect_column_values_to_be_in_set for this scenario. And my test case systax will be_
```yaml
   {{ testUtils.expect_column_values_to_be_in_set( 'CONTACT_VIA', ['EMAIL','CALL','TEXT']) }}
```

2. I want to check if each column entries to be strings that match a given SQL like pattern.
   _I will refer test case expect_column_values_to_match_like_pattern for this scenario. And my test case systax will be_
```yaml
   {{ testUtils.expect_column_values_to_match_like_pattern('{{ 'EMAIL_ID', '%@%') }}
```

## Available Tests

### Table shape

- [expect_column_to_exist](#expect_column_to_exist)
- [expect_row_values_to_have_recent_data](#expect_row_values_to_have_recent_data)
- [expect_grouped_row_values_to_have_recent_data](#expect_grouped_row_values_to_have_recent_data)
- [expect_table_aggregation_to_equal_other_table](#expect_table_aggregation_to_equal_other_table)
- [expect_table_column_count_to_be_between](#expect_table_column_count_to_be_between)
- [expect_table_column_count_to_equal_other_table](#expect_table_column_count_to_equal_other_table)
- [expect_table_column_count_to_equal](#expect_table_column_count_to_equal)
- [expect_table_columns_to_not_contain_set](#expect_table_columns_to_not_contain_set)
- [expect_table_columns_to_contain_set](#expect_table_columns_to_contain_set)
- [expect_table_columns_to_match_ordered_list](#expect_table_columns_to_match_ordered_list)
- [expect_table_columns_to_match_set](#expect_table_columns_to_match_set)
- [expect_table_row_count_to_be_between](#expect_table_row_count_to_be_between)
- [expect_table_row_count_to_equal_other_table](#expect_table_row_count_to_equal_other_table)
- [expect_table_row_count_to_equal_other_table_times_factor](#expect_table_row_count_to_equal_other_table_times_factor)
- [expect_table_row_count_to_equal](#expect_table_row_count_to_equal)

### Missing values, unique values, and types

- [expect_column_values_to_be_of_type](#expect_column_values_to_be_of_type)
- [expect_column_values_to_be_in_type_list](#expect_column_values_to_be_in_type_list)
- [expect_column_values_to_have_consistent_casing](#expect_column_values_to_have_consistent_casing)

### Sets and ranges

- [expect_column_values_to_be_in_set](#expect_column_values_to_be_in_set)
- [expect_column_values_to_not_be_in_set](#expect_column_values_to_not_be_in_set)
- [expect_column_values_to_be_between](#expect_column_values_to_be_between)
- [expect_column_values_to_be_decreasing](#expect_column_values_to_be_decreasing)
- [expect_column_values_to_be_increasing](#expect_column_values_to_be_increasing)

### String matching

- [expect_column_value_lengths_to_be_between](#expect_column_value_lengths_to_be_between)
- [expect_column_value_lengths_to_equal](#expect_column_value_lengths_to_equal)
- [expect_column_values_to_match_like_pattern](#expect_column_values_to_match_like_pattern)
- [expect_column_values_to_match_like_pattern_list](#expect_column_values_to_match_like_pattern_list)
- [expect_column_values_to_match_regex](#expect_column_values_to_match_regex)
- [expect_column_values_to_match_regex_list](#expect_column_values_to_match_regex_list)
- [expect_column_values_to_not_match_like_pattern](#expect_column_values_to_not_match_like_pattern)
- [expect_column_values_to_not_match_like_pattern_list](#expect_column_values_to_not_match_like_pattern_list)
- [expect_column_values_to_not_match_regex](#expect_column_values_to_not_match_regex)
- [expect_column_values_to_not_match_regex_list](#expect_column_values_to_not_match_regex_list)

### Aggregate functions

- [expect_column_distinct_count_to_be_greater_than](#expect_column_distinct_count_to_be_greater_than)
- [expect_column_distinct_count_to_be_less_than](#expect_column_distinct_count_to_be_less_than)
- [expect_column_distinct_count_to_equal_other_table](#expect_column_distinct_count_to_equal_other_table)
- [expect_column_distinct_count_to_equal](#expect_column_distinct_count_to_equal)
- [expect_column_distinct_values_to_be_in_set](#expect_column_distinct_values_to_be_in_set)
- [expect_column_distinct_values_to_contain_set](#expect_column_distinct_values_to_contain_set)
- [expect_column_distinct_values_to_equal_set](#expect_column_distinct_values_to_equal_set)
- [expect_column_max_to_be_between](#expect_column_max_to_be_between)
- [expect_column_mean_to_be_between](#expect_column_mean_to_be_between)
- [expect_column_median_to_be_between](#expect_column_median_to_be_between)
- [expect_column_min_to_be_between](#expect_column_min_to_be_between)
- [expect_column_most_common_value_to_be_in_set](#expect_column_most_common_value_to_be_in_set)
- [expect_column_proportion_of_unique_values_to_be_between](#expect_column_proportion_of_unique_values_to_be_between)
- [expect_column_quantile_values_to_be_between](#expect_column_quantile_values_to_be_between)
- [expect_column_stdev_to_be_between](#expect_column_stdev_to_be_between)
- [expect_column_sum_to_be_between](#expect_column_sum_to_be_between)
- [expect_column_unique_value_count_to_be_between](#expect_column_unique_value_count_to_be_between)

### Multi-column

- [expect_column_pair_values_A_to_be_greater_than_B](#expect_column_pair_values_a_to_be_greater_than_b)
- [expect_column_pair_values_to_be_equal](#expect_column_pair_values_to_be_equal)
- [expect_column_pair_values_to_be_in_set](#expect_column_pair_values_to_be_in_set)
- [expect_compound_columns_to_be_unique](#expect_compound_columns_to_be_unique)
- [expect_multicolumn_sum_to_equal](#expect_multicolumn_sum_to_equal)
- [expect_select_column_values_to_be_unique_within_record](#expect_select_column_values_to_be_unique_within_record)

### Distributional functions

- [expect_column_values_to_be_within_n_moving_stdevs](#expect_column_values_to_be_within_n_moving_stdevs)
- [expect_column_values_to_be_within_n_stdevs](#expect_column_values_to_be_within_n_stdevs)

## Documentation

### [expect_column_to_exist]

Expect the specified column to exist.

*Applies to:* Column

```yaml
tests: {{ expect_column_to_exist( 'columnName') }}
```

### [expect_row_values_to_have_recent_data]

Expect the model to have rows that are at least as recent as the defined interval prior to the current timestamp. Optionally gives the possibility to apply filters on the results.

*Applies to:* Column

```yaml
tests: {{ expect_row_values_to_have_recent_data( 'columnName', 'datePart_', 'interval_', row_condition = None) }}

Inputs: datepart_ = 'day'
        interval_ = '1'
        row_condition = 'id is not null' #optional
```

### [expect_grouped_row_values_to_have_recent_data] 

Expect the model to have **grouped** rows that are at least as recent as the defined interval prior to the current timestamp.
Use this to test whether there is recent data for each grouped row defined by `group_by` (which is a list of columns) and a `timestamp_column`. Optionally gives the possibility to apply filters on the results.

*Applies to:* Object

```yaml
tests : {{ expect_grouped_row_values_to_have_recent_data(group_by,timestamp_column,datepart,interval,filterCondition=None ) }}

Inputs: group_by = ['group_id', 'other_group_id']
        timestamp_column = 'date_day'
        datepart = day
        interval = 1
        filterCondition = "id is not null" #optional
```

### [expect_table_aggregation_to_equal_other_table]

Expect an (optionally grouped) expression to match the same (or optionally other) expression in a different table.

*Applies to:* Object

Simple Test:

```yaml
tests: {{ expect_table_aggregation_to_equal_other_table(expression,compare_model,group_by=None) }}

Inputs: expression = sum(col_numeric_a)
        compare_model = ref("other_model")
        group_by = [idx] #optional
```

More complex Test:

```yaml
tests: {{ expect_table_aggregation_to_equal_other_table(
          expression,
          compare_model,compare_expression=None,
          group_by=None,compare_group_by=None,
          row_condition=None,compare_row_condition=None,
          tolerance=0.0,tolerance_percent=None ) }}

Inputs: expression = 'max("column_a")'
        compare_model = 'ref("other_model")'
        compare_expression = 'max("column_b")'
        group_by = ['date_column']
        compare_group_by = ['some_other_date_column']
        row_condition = 'some_flag=true'
        compare_row_condition = 'some_flag=false'

```

**Note**: You can also express a **tolerance** factor, either as an absolute tolerable difference, `tolerance`, or as a tolerable % difference `tolerance_percent` expressed as a decimal (i.e 0.05 for 5%).

### [expect_table_column_count_to_be_between]

Expect the number of columns in a model to be between two values.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_be_between( minValue , maxValue) }}
Inputs: minValue = 1
        maxValue = 4 
```

### [expect_table_column_count_to_equal_other_table]

Expect the number of columns in a model to match another model.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_equal_other_table( 'comparing_tableName') }}

```

### [expect_table_columns_to_not_contain_set]

Expect the columns in a model not to contain a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_not_contain_set( notExpectedColumnList) }}
Inputs: notExpectedColumnList = ["col_a", "col_b"]
```

### [expect_table_columns_to_contain_set]

Expect the columns in a model to contain a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_contain_set( ExpectedColumnList) }}
Inputs: ExpectedColumnList = ["col_a", "col_b"]

```

### [expect_table_column_count_to_equal]

Expect the number of columns in a model to be equal to `expected_number_of_columns`.

*Applies to:* Object

```yaml
tests: {{ expect_table_column_count_to_equal( ExpectedCount ) }}
Inputs: ExpectedCount = '7'
```

### [expect_table_columns_to_match_ordered_list]

Expect the columns to exactly match a specified list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_match_ordered_list( column_list, transform="UPPER") }}

Inputs:   column_list = ["col_a", "col_b"]
          transform = upper # (Optional)
```

### [expect_table_columns_to_match_set]

Expect the columns in a model to match a given list.

*Applies to:* Object

```yaml
tests: {{ expect_table_columns_to_match_set( ExpectedColumnList) }}

Inputs: column_list = ["col_a", "col_b"]
```

### [expect_table_row_count_to_be_between]

Expect the number of rows in a model to be between two values.
*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_be_between( minValue , maxValue ,group_by = None, filterCondition = None) }}

Inputs:   minValue = 1 
          maxValue = 4 
          group_by = ['group_id', 'other_group_id', ...] # (Optional)
          filterCondition: 'id is not null' # (Optional)
```

### [expect_table_row_count_to_equal_other_table]

Expect the number of rows in a model match another model.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal_other_table( comparing_tableName, group_by_t1 = None, group_by_t2 = None,  filterCondition_t1 = None, filterCondition_t2 = None) }}

Inputs:   comparing_tableName = {{ ref('target','other_model') }}
          group_by_t1 = ['col1', 'col2'] # (Optional)
          group_by_t2 = ['col1', 'col2'] # (Optional)
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_table_row_count_to_equal_other_table_times_factor]

Expect the number of rows in a model to match another model times a preconfigured factor.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal_other_table_times_factor( comparing_tableName,  group_by_t1 = None, group_by_t2 = None, filterCondition_t1 = None, filterCondition_t2 = None, factor = None) }}

Inputs:   comparing_tableName = {{ ref('target','other_model') }}
          factor = '13'
          group_by_t1 = ['col1', 'col2'] # (Optional)
          group_by_t2 = ['col1', 'col2'] # (Optional)
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_table_row_count_to_equal]

Expect the number of rows in a model to be equal to `expected_number_of_rows`.

*Applies to:* Object

```yaml
tests: {{ expect_table_row_count_to_equal( numberOfRecordExpected, group_by = None, filterCondition = None) }}

Inputs:   numberOfRecordExpected = '4'
          group_by = ['group_id', 'other_group_id', ...] # (Optional)
          filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_of_type]

Expect a column to be of a specified data type.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_of_type( columnName, dataType) }}

Input: dataType = 'date'
```

### [expect_column_values_to_be_in_type_list]

Expect a column to be one of a specified type list.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_in_type_list( columnName, dataType) }}

Inputs: dataType = ['date', 'datetime']
```

### [expect_column_values_to_have_consistent_casing]

Expect a column to have consistent casing. By setting `display_inconsistent_columns` to true, the number of inconsistent values in the column will be displayed in the terminal whereas the inconsistent values themselves will be returned if the SQL compiled test is run.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_have_consistent_casing( column_name, display_inconsistent_columns=False) }}

Input: display_inconsistent_columns = false # (Optional)
```

### [expect_column_values_to_be_in_set]

Expect each column value to be in a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: expectedValueList = ['a','b','c']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_between]

Expect each column value to be between two values.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_between( columnName, minValue , maxValue, filterCondition = None ) }}

Inputs: minValue = '0'  
        maxValue = '10' 
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_be_in_set]

Expect each column value not to be in a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: expectedValueList = ['e','f','g']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_increasing]

Expect column values to be increasing.

If `strictly: True`, then this expectation is only satisfied if each consecutive value is strictly increasing – equal values are treated as failures.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_increasing(  column_name, sort_column=None, strictly=True, filterCondition=None, group_by=None, step=None) }}

Inputs: sort_column = date_day
        filterCondition = "id is not null" # (Optional)
        strictly = true # (Optional for comparison operator. Default is 'true', and it uses '>'. If set to 'false' it uses '>='.)
        group_by = [group_id, other_group_id, ...] # (Optional)
        step = 1 # (Optional. If set, it requires the difference between values to be exactly this step. Requires numeric columns.)
```

### [expect_column_values_to_be_decreasing]

Expect column values to be decreasing.

If `strictly=True`, then this expectation is only satisfied if each consecutive value is strictly decreasing – equal values are treated as failures.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_decreasing( column_name, sort_column=None, strictly=True, filterCondition=None, group_by=None, step=None) }}

Inputs: sort_column = col_numeric_a
        filterCondition = "id is not null" # (Optional)
        strictly = true # (Optional for comparison operator. Default is 'true' and it uses '<'. If set to 'false', it uses '<='.)
        group_by = [group_id, other_group_id, ...] # (Optional)
        step = 1 # (Optional. If set, it requires the difference between values to be exactly this step. Requires numeric columns.)
```

### [expect_column_value_lengths_to_be_between]

Expect column entries to be strings with length between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_value_lengths_to_be_between( columnName, minLength, maxLength, filterCondition = None) }}

Inputs: min_value = '1' 
        max_value = '4' 
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_value_lengths_to_equal]

Expect column entries to be strings with length equal to the provided value.

*Applies to:* Column

```yaml
tests: {{ expect_column_value_lengths_to_equal( columnName, rowsValueLength, filterCondition = None) }}

Inputs: rowsValueLength = '10'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_match_regex]

Expect column entries to be strings that match a given regular expression. Valid matches can be found anywhere in the string, for example "[at]+" will identify the following strings as expected: "cat", "hat", "aa", "a", and "t", and the following strings as unexpected: "fish", "dog".

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_regex( columnName, regex, isRaw = False, flags='', filterCondition=None ) }}

Inputs: regex = '[at]+'
        filterCondition = 'id is not null' # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_not_match_regex]

Expect column entries to be strings that do NOT match a given regular expression. The regex must not match any portion of the provided string. For example, "[at]+" would identify the following strings as expected: "fish”, "dog”, and the following as unexpected: "cat”, "hat”.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_regex( columnName, regex, isRaw = False, flags='', filterCondition=None ) }} 

Inputs: regex = "[at]+"
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_match_regex_list]

Expect the column entries to be strings that can be matched to either any of or all of a list of regular expressions. Matches can be anywhere in the string.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_regex_list( columnName, regexList, matchType='all', isRaw=False, flags='', filterCondition=None ) }}

Inputs: regex_list = ["@[^.]*", "&[^.]*"]
        matchType = 'any' # (Optional. Default is 'all', which applies an 'AND' for each regex. If 'any', it applies an 'OR' for each regex.)
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_not_match_regex_list]

Expect the column entries to be strings that do not match any of a list of regular expressions. Matches can be anywhere in the string.

Optional (keyword) arguments:

- `isRaw` indicates the `regex` pattern is a "raw" string and should be escaped. The default is `False`.
- `flags` is a string of one or more characters that are passed to the regex engine as flags (or parameters). Allowed flags are adapter-specific. A common flag is `i`, for case-insensitive matching. The default is no flags.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_regex_list( columnName, regexList, matchType='all', isRaw=False, flags='', filterCondition=None ) }} 

Inputs: regex_list = ["@[^.]*", "&[^.]*"]
        matchType = 'any' # (Optional. Default is 'all', which applies an 'AND' for each regex. If 'any', it applies an 'OR' for each regex.)
        filterCondition = "id is not null" # (Optional)
        isRaw = 'True' # (Optional)
        flags = 'i' # (Optional)
```

### [expect_column_values_to_match_like_pattern]

Expect column entries to be strings that match a given SQL `like` pattern.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_like_pattern( columnName, pattern, filterCondition = None) }}

Inputs: pattern = '%@%'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_match_like_pattern]

Expect column entries to be strings that do not match a given SQL `like` pattern.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_like_pattern( columnName, pattern, filterCondition = None) }}

Inputs: pattern = '%&%'
        row_condition = "id is not null" # (Optional)
```

### [expect_column_values_to_match_like_pattern_list]

Expect the column entries to be strings that match any of a list of SQL `like` patterns.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_match_like_pattern_list( columnName, patternList, matchType='any', filterCondition = None) }}

Inputs: patternList = ["%@%", "%&%"]
        matchType = 'any' # (Optional. Default is 'any', which applies an 'OR' for each pattern. If 'all', it applies an 'AND' for each regex.)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_not_match_like_pattern_list]

Expect the column entries to be strings that do not match any of a list of SQL `like` patterns.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_not_match_like_pattern_list( columnName, patternList, matchType='any', filterCondition = None) }}

Inputs: patternList = ["%@%", "%&%"]
        matchType = 'any' # (Optional. Default is 'any', which applies an 'OR' for each pattern. If 'all', it applies an 'AND' for each regex.)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_equal]

Expect the number of distinct column values to be equal to a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_equal( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_be_greater_than]

Expect the number of distinct column values to be greater than a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_be_greater_than( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_be_less_than]

Expect the number of distinct column values to be less than a given value.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_be_less_than( columnName, expextedCount, group_by = None, filterCondition = None) }}

Inputs: value = '10'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_be_in_set]

Expect the set of distinct column values to be contained by a given set.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_be_in_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b','c','d']
        row_condition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_contain_set]

Expect the set of distinct column values to contain a given set.

In contrast to `expect_column_values_to_be_in_set` this ensures not that all column values are members of the given set but that values from the set must be present in the column.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_contain_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b']
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_distinct_values_to_equal_set]

Expect the set of distinct column values to equal a given set.

In contrast to `expect_column_distinct_values_to_contain_set` this ensures not only that a certain set of values are present in the column but that these and only these values are present.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_values_to_equal_set( columnName, expectedValueList, filterCondition = None) }}

Inputs: value_set = ['a','b','c']
        row_condition = "id is not null" # (Optional)
```

### [expect_column_distinct_count_to_equal_other_table]

Expect the number of distinct column values to be equal to number of distinct values in another model.

*Applies to:* Column

```yaml
tests: {{ expect_column_distinct_count_to_equal_other_table( columnName, otherTableName, otherColumnName, filterCondition_t1 = None, filterCondition_t2 = None) }}

Inputs:   columnName = 'col_1'
          otherTableName = ref("my_model_2")
          otherColumnName = col_2
          filterCondition_t1 = "id is not null" # (Optional)
          filterCondition_t2 = "id is not null" # (Optional)
```

### [expect_column_mean_to_be_between]

Expect the column mean to be between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_mean_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0' 
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_median_to_be_between]

Expect the column median to be between a min_value value and a max_value value (inclusive).

*Applies to:* Column

```yaml
tests: {{ expect_column_median_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0'
        maxValue = '2'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_quantile_values_to_be_between]

Expect specific provided column quantiles to be between provided min_value and max_value values.

*Applies to:* Column

```yaml
tests: {{ expect_column_quantile_values_to_be_between( columnName, quantile, minValue , maxValue, filterCondition = None, group_by = None ) }}

Inputs: quantile = '.95'
        minValue = '0' 
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition: "id is not null" # (Optional)
```

### [expect_column_stdev_to_be_between]

Expect the column standard deviation to be between a min_value value and a max_value value. Uses sample standard deviation (normalized by N-1).

*Applies to:* Column

```yaml
tests: {{ expect_column_stdev_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '0' 
        maxValue = '2'
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_unique_value_count_to_be_between]

Expect the number of unique values to be between a min_value value and a max_value value.

*Applies to:* Column

```yaml
tests: {{ expect_column_unique_value_count_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = '2'
        maxValue = '2' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_proportion_of_unique_values_to_be_between]

Expect the proportion of unique values to be between a min_value value and a max_value value.

For example, in a column containing [1, 2, 2, 3, 3, 3, 4, 4, 4, 4], there are 4 unique values and 10 total values for a proportion of 0.4.

*Applies to:* Column

```yaml
tests: {{ expect_column_proportion_of_unique_values_to_be_between( columnName, minValue, maxValue,  filterCondition = None, group_by = None) }}

Inputs: minValue = '0'  
        maxValue = '.4' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_most_common_value_to_be_in_set]

Expect the most common value to be within the designated value set

*Applies to:* Column

```yaml
tests: {{ expect_column_most_common_value_to_be_in_set( column_name, value_set, top_n, quote_values=True, data_type="STRING", filterCondition=None) }}

Inputs: value_set: ['0.5']
        top_n: 1
        quote_values: true # (Optional. Default is 'true'.)
        data_type: "decimal" # (Optional. Default is 'decimal')
```

### [expect_column_max_to_be_between]

Expect the column max to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_max_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None) }}

Inputs: minValue = '1' 
        maxValue = '1' 
        group_by = ['group_id', 'other_group_id', ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_min_to_be_between]

Expect the column min to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_min_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None) }}

Inputs: minValue = 0 
        maxValue = 1 
        group_by = [group_id, other_group_id, ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_sum_to_be_between]

Expect the column to sum to be between a min and max value

*Applies to:* Column

```yaml
tests: {{ expect_column_sum_to_be_between( columnName, minValue, maxValue, filterCondition = None, group_by = None ) }}

Inputs: minValue = 1 
        maxValue = 2 
        group_by = [group_id, other_group_id, ...] # (Optional)
        row_condition = "id is not null" # (Optional)
```

### [expect_column_pair_values_A_to_be_greater_than_B]

Expect values in column A to be greater than column B.

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_A_to_be_greater_than_B( columnNameA, columnNameB, orEqualTo = FALSE, filterCondition = None) }}

Inputs: columnNameA = 'col_numeric_a'
        columnNameB = 'col_numeric_a'
        orEqualTo = True
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_pair_values_to_be_equal]

Expect the values in column A to be the same as column B.

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_to_be_equal( columnNameA, columnNameB, filterCondition = None) }}

Inputs: columnNameA = 'col_numeric_a'
        columnNameB = 'col_numeric_a'
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_pair_values_to_be_in_set]

Expect paired values from columns A and B to belong to a set of valid pairs.

Note: value pairs are expressed as lists within lists

*Applies to:* Object

```yaml
tests: {{ expect_column_pair_values_to_be_in_set( columnNameA, columnNameB, validPairs, filterCondition = None) }}

Inputs: column_A = 'col_numeric_a'
        column_B = 'col_numeric_b'
        value_pairs_set = [[0, 1], [1, 0], [0.5, 0.5], [0.5, 0.5]]
        filterCondition = "id is not null" # (Optional)
```

### [expect_select_column_values_to_be_unique_within_record]

Expect the values for each record to be unique across the columns listed. Note that records can be duplicated.

*Applies to:* Object

```yaml
tests: {{ expect_select_column_values_to_be_unique_within_record( column_list, filterCondition=None) }}

Inputs: column_list = ["col_string_a", "col_string_b"]
        filterCondition = "id is not null" # (Optional)
```

### [expect_multicolumn_sum_to_equal]

Expects that sum of all rows for a set of columns is equal to a specific value

*Applies to:* Object

```yaml
tests: {{ expect_multicolumn_sum_to_equal( column_list, sum_total, group_by=None, filterCondition=None) }}

Inputs: column_list = ["col_numeric_a", "col_numeric_b"]
        sum_total = 4
        group_by = [group_id, other_group_id, ...] # (Optional)
        filterCondition = "id is not null" # (Optional)
```

### [expect_compound_columns_to_be_unique]

Expect that the columns are unique together, e.g. a multi-column primary key.

*Applies to:* Object

```yaml
tests: {% macro expect_compound_columns_to_be_unique( columnNames, filterCondition = None) %}

Inputs: column_list = ["date_col", "col_string_b"]
        filterCondition = "id is not null" # (Optional)
```

### [expect_column_values_to_be_within_n_moving_stdevs]

A simple anomaly test based on the assumption that differences between periods in a given time series follow a log-normal distribution.
Thus, we would expect the logged differences (vs N periods ago) in metric values to be within Z sigma away from a moving average.
By applying a list of columns in the `group_by` parameter, you can also test for deviations within a group.

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_within_n_moving_stdevs(
     column_name, date_column_name, group_by=None, 
    period='day', lookback_periods=1, trend_periods=7, test_periods=14, 
    sigma_threshold=3, sigma_threshold_upper=None, sigma_threshold_lower=None, 
    take_diffs=true, take_logs=true
) }}

Inputs: date_column_name = date
        period = day # (Optional. Default is 'day')
        lookback_periods = 1 # (Optional. Default is 1)
        trend_periods = 7 # (Optional. Default is 7)
        test_periods = 14 # (Optional. Default is 14)
        sigma_threshold = 3 # (Optional. Default is 3)
        take_logs = true # (Optional. Default is 'true')
        sigma_threshold_upper = x # (Optional. Replace 'x' with a value. Default is 'None')
        sigma_threshold_lower = y # (Optional. Replace 'y' with a value. Default is 'None')
        take_diffs = true # (Optional. Default is 'true')
        group_by = [group_id] # (Optional. Default is 'None')
```

### [expect_column_values_to_be_within_n_stdevs]

Expects (optionally grouped & summed) metric values to be within Z sigma away from the column average

*Applies to:* Column

```yaml
tests: {{ expect_column_values_to_be_within_n_stdevs( column_name, group_by=None, sigma_threshold=3) }}

Inputs: group_by = ['group_id'] # (Optional. Default is 'None')
        sigma_threshold = '3' # (Optional. Default is 3)
```

### [test_missing_dates]

Check missing dates in given range.

*Applies to:* Column

```yaml
tests: {{ test_missing_dates(date_column, from, to) }}

Inputs: from = '2024-01-01' # (start of date range)
        to = '2024-01-01' # (end of date range)
```

### [test_missing_date_offset]

Check missing dates from current date to offset.

*Applies to:* Column

```yaml
tests: {{ test_missing_date_offset(date_column, from, to) }}

Inputs: from = '100' 
        to = '1' 
```

## Versions

Available versions of the package.

| Version # | Release Date | Notes |
|-----------|--------------|-------|
| 1.1.1 | July 28, 2025 | Bug fix for test case EXPECT_TABLE_AGGREGATION_TO_EQUAL_OTHER_TABLE |
| 1.1.0 | June 03, 2025 | Simplifying Test Macros |
| 1.0.0 | December 12, 2024 | Initial release of Coalesce test utility. |

---

## Getting an Access Token

## Getting a Access Token

Coalesce requires an access token to make API requests. Go to the **Deploy tab** and click **Generate Access Token**.

## Access Token Regeneration

Access tokens are regenerated after each login. Previously issued tokens remain valid. Tokens never expire across all environments and projects. They can be revoked under certain circumstances.

## Access Token Revoked

Tokens are revoked when:

- User is deleted
- User is disabled
- Password change
- Email address change
- SSO users will need a new token for each new environment created.
- Turning Multi-factor Authentication on and off.

---

## Changing Your Login Email

There currently isn't a way to change your email address in Coalesce. You'll need to create a new account. Talk with your org admin to create a new user.

If you create a new account, there isn't a way to migrate data between accounts.

---

## Change or Forgot Password

## New Password

Enter your old password once, your desired new password, and your new password again to confirm. Then click on **Update Password** to complete the password change.

:::info[Password Requirement]

Passwords must be at least 6 characters.

:::

## Forgot Password

If you've forgotten your current password, you can reset it by logging out and clicking the **Forgot your password?** link on the login page.

## SSO Password Change

If your company uses SSO, you'll need to work with your Administrator to log back in.

---

## Migrating Work Between Coalesce Accounts

Coalesce leverages Git to store and version control your work, which makes migrating work between Coalesce accounts simple and easy.

Let's work with an example scenario where you have a Project in Account A that you wish to migrate to Account B. Here’s a step-by-step guide to help you achieve this:

## Account Migration Steps

1. **Create a New Project:** Begin by creating a new Project and associated Workspace for the Project you intend to migrate from Account A.
2. **Link to Git Repository:** Associate this new Project with the Git repository that contains the Project from Account A.
3. **Reconfigure Your Essential Settings:** For security reasons, certain Coalesce configurations aren't stored in Git. Ensure you:

   - Reset your Git credentials.
   - Re-establish your Data Platform connection.
   - Update your Storage Mappings.
4. **Optional - Paring Down Objects:** If your Git repository has more content than the specific Project you're looking to migrate, you can refine the objects with the following steps:

   a. **Merging with an Existing Project:**

   - Hover over the `...` next to your Project/Workspace created in Step 1.
   - Choose "Copy Objects from Workspace."
   - Find and select the Project and Workspace you wish to combine with the Project you're migrating.
   - Choose all the relevant objects and click "Copy."

   b. **Starting Afresh without Merging:**

   - Initiate an additional new Project and Workspace in Account B.
   - Link this new project to a fresh Git repository.
   - Hover over the `...` adjacent to your Project/Workspace from Step 1.
   - Choose "Copy Objects from Workspace."
   - Locate and select the Project and Workspace where you want to integrate your migrated project.
   - Choose only the objects you wish to bring into your fresh Project and click "Copy."

## Post-migration Steps in Account A

After successfully migrating your Project, if you desire to delete the Project or certain objects from Account A, you have can do so.

---

## Multi-factor Authentication

:::info[Availability]

MFA is only available for username and password login.

:::

Enable MFA (Multi-factor authentication) to keep your account secure. MFA is configured at the account level.

1. Go to **User Settings > Account and Security**.
2. If your email hasn't been verified, resend the email and follow the instructions.
3. Toggle **Multi-Factor Authentication** to on.
4. Choose **Authenticator App** from the drop-down.
5. Toggle MFA Status to **Enabled**.
6. Enter your password.
7. Scan the QR code using your authenticator app then click Next.
8. Enter the six digit code.
9. That’s it. You’re ready to use MFA.

:::info[MFA Confirmation]

After enrollment, you'll get a confirmation email. MFA can only be disabled in the Coalesce app, by an organization administrator.
:::

## Organization Admins

- Users can't turn off MFA. Once enabled, an administrator will need to disable MFA.
- MFA is managed on a per-user basis. MFA can't be turned on and off for the organization in Coalesce.
- Users will need to enable individually.

## MFA and Access Tokens

If an account has an access token and MFA is turned on or off, the access token will no longer work. Get a new token by going to the [Deploy Page][].

## Snowflake Accounts

If you have MFA enabled on your **Snowflake** account, you should use OAuth authentication with Coalesce.

[Deploy Page]: /docs/api/authentication

---

## Authentication Methods

## Authenticating to Coalesce

| Auth Method                           | Web GUI |  API  |  CLI  |
| :------------------------------------ | :-----: | :---: | :---: |
| Username & Password                   |  **X**  |       |       |
| Single Sign-On via Okta               |  **X**  |       |       |
| Single Sign-On via Microsoft Entra ID |  **X**  |       |       |
| Coalesce Token                        |         | **X** | **X** |

## Authenticating Coalesce to Your Data Platform

| Auth Method                                   | Web GUI |  API  |  CLI  | Notes |
| :-------------------------------------------- | :-----: | :---: | :---: | --- |
| Snowflake Username & Password (Cloud Storage)           |  **X**  | **X** |   | Snowflake is moving towards MFA for all password based sign-ins. We recommend using OAuth for user login and Key pair for automation accounts. For the most recent information, see Snowflake's blog post: [Snowflake Will Block Single-Factor Password Authentication by November 2025][].|
| Snowflake Username & Password (Browser/Local Storage) |  **X**  | **X** | **X** | Storing passwords in browser storage is insecure, as anyone with browser access could view them. For Username and Password authentication, use Cloud Storage instead.|
| Snowflake OAuth                        |  **X**  | **X** |       | Proof Key for Code Exchange (PKCE) is required and enforced for all OAuth authorizations.|
| Snowflake Key Pair                                      |  **X**  | **X** | **X** | |
| Databricks User-to-Machine | **X** | | | |
| Databricks Machine-to-Machine |**X** | **X**| **X**| |
| Databricks Token | **X** | **X**| **X**| |

---

## What's Next?

* [Connection Guides][]

---

[Connection Guides]: /docs/category/connection-guides
[Snowflake Will Block Single-Factor Password Authentication by November 2025]: /docs/organization-and-accounts/organization-management/authentication-methods

---

## Organization Mangement

Organization management is available to Organization Admins.

<DocCardList />

---

## How to Assign User Roles

You'll learn how to add users to your organization and assign them roles.

## Add a User to Your Organization

:::info[Organization Administrator]

You must be an organization administrator to add users and assign roles.

:::

1. Go to **Org Settings**.
2. Click **Add a New User**.
3. Fill out the form with the user information. You'll need to select the Role and Organization. Then Save.
4. Users added will receive a password reset email for the organization they were added to.

Once a user is added to the organization, they can then be added to the [project][]. Review the [Org Level Roles][].

## Add a User to a Project

:::info[Add Users to the Organization First]

A user must be added to the organization, before being added to a project.

:::

1. Go to the **Projects** page.
2. Go to **Project Settings > Members**.

Review the [Project][] level roles. Project membership grants access to the project, including all **Workspaces** in that project, as allowed by the member's project role.

## Add a User to an Environment

:::info[Add Users to the Environment First]

A user must be added to the organization, then the project, before being added to a environment. If you don't want them to have project access, add them as a Project Member first.

:::

1. Access the environment settings either by going to the **Build Settings > Environment > Members** or by clicking the **Deploy Tab** and then configure the Environment.

Review the [Environment][] level roles.

  
## New Users

Unless specified when adding a user, new users will be added as Org Members. Org members have limited access and can't see project or environment information.

## Existing Users

If you enable RBAC and have existing users then they'll be given the following roles automatically.

- If a user had the Admin role, they are given the role of Org Admin, Project Admin, and Environment Admin.
- If a user had a non-administrative role, then they are given the role of Org Contributors, Project Admins, and Environment Admin.

## SSO Users

New SSO users are given the role of Org Member.

## Learn How to Manage Your Users Video

<YouTube id="EJ6uiE922H8" />

[Project]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Environment]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Org Level Roles]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles

---

## Role-Based Access Control

Role-Based Access Control (RBAC) is a method for regulating access based on the roles of individual users. In RBAC, roles are assigned to users based on permissions according to the roles. RBAC helps in enforcing the principle of least privilege, reducing the risk of accidental or malicious access to sensitive data. It also streamlines the process of adding, removing, or changing permissions since it only requires updating the user role.

## Understanding User Roles

- **Organization** - They tend to have access to organization level settings such as being able to create projects for the organization. Review [Organization][] Roles for an in-depth explanation of each organization role.
- **Project -** These roles can work in the project they are assigned to. This is where data architects and data engineers will be assigned to build out the data pipeline. Review [Project][] Roles for an in-depth explanation of each organization role.
- **Environment -** Environment roles can deploy pipelines and view documentation. These are good for data and business analysts who need to review information or operations to deploy pipelines. Review [Environment][] Roles for an in-depth explanation of each organization role.

## User Role Assignment Example

Let’s go through an example of assigning a user role. There is a Senior Data Engineer on your team and they need to be able to work on multiple projects.

In the example the engineer is assigned to the Organization as an Org Contributor. They will be able to create projects, assign users to projects, and manage project settings.

They are also added to the projects for Finance, Marketing, and Sales. As an Org Contributor, the data engineer only has full access to the projects they created. They will need to be assigned to other projects. In the Marketing project, they are a Project Architect meaning they can build node types and configure the storage locations. In the Sales Project, they are a Project Member which is the role assigned to give them Environment access. They do not have access to a project, but since the roles are hierarchical, they need to be given Project level permissions first.

At the environment level, in the Finance project, there are two environments: QA and UAT. They are an Environment Admin on both. An Environment Admin can approve deployment and schedule jobs. In the Sales project, there are two environments: Staging and Production. They have no access to the Staging environment, but they are an Environment Reader in Production. An Environment Reader has access to the documentation so they can understand the data pipelines for that environment.

Coalesce RBAC gives you full control over user access in the platform. Assign users the roles they need to perform their roles.

## What's Next?

<DocCardList />

[Organization]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Project]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Environment]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles

---

## RBAC Migration Guide

Use this guide to migrate your users to RBAC and understand the changes for your existing users.

## Existing Users

Existing users will be given the following roles automatically. You'll need to adjust their user access accordingly.

- If a user had an **Admin role**, they are given the role of Org Admin, Project Admin, and Environment Admin.

- If a user had a **non-administrative role**, then they are given the role of Org Contributors, Project Admins, and Environment Admin.

## Licensing

Any user assigned **at least one** of the following roles, will count towards your license allocation.

- Org Admin

- Project Admin

- Project Architect

- Project Contributor

Users who are **only assigned** the following roles are considered read-only users and **will not count** against the license allocation. The user can't be assigned any of the above roles.

- Org Contributor

- Org Member

- Project Member

- Environment Admin

- Environment Reader

## Service Accounts

Service accounts should only be granted from one of the following roles:

If your service account needs to deploy and refresh, assign the Environment Admin role.

- Org Contributor

- Org Member

- Project Member

- Environment Reader

- Environment Admin

---

## RBAC Roles and Permissions

- **Read(R)** - A user can view the data, but not create or update the data.
- **Write (W)** - A user can create something new or modify existing information. This includes, creating, updating, and deleting.

- [Coalesce Roles and Permissions Summary](#coalesce-roles-and-permissions-summary)

<!-- vale off -->

<RBACTable />

## Coalesce Roles and Permissions Summary

|Level|Role|Permissions Summary|Recommended For|
|--- |--- |--- |--- |
|**Organization**|**Organization Administrator**|The creator of the Coalesce App is automatically assigned as organization administrator. Only organization administrators can add other users, including other organization administrators. They have full access to all functionality in Coalesce.|Full administrative control|
|**Organization**|**Organization Contributor**|They can't add new users to the organization.They have access to read documentation, create API tokens, user settings, and Git account information. They will be able to set up a project, configure Git, add members to projects,and oversee work.Only have access to the projects created by them. If there are multiple organization contributors, they will need to share access with the organization contributor.|Managers who decide how each person will contribute to a project.|
|**Organization**|**Organization Member**|This is the default role. They can edit Git account information, create API tokens, and read documentation.|Default Role|
|**Project**|**Project Administrator**|This role can manage projects, but not create them. An organization administrator or contributor can create projects. The role has access to projects, deployments, and environments.• Project roles have no Organization permissions• Project roles must be added to the org first, then the project|Team manager or senior team member to manage projects.|
|**Project**|**Project Architect**|This role can manage certain project information, build nodes, and generate API tokens. Assign this role to a data architect so they can build the needed node types, set storage locations, and create macros.|Senior data architects.|
|**Project**|**Project Contributor**|A project contributor can't edit or create custom nodes or macros. They have read-only access to certain project settings. They have read access to projects, deployments, and environments.|Team members who need access to project information without making changes.|
|**Project**|**Project Member**|Assign this role if you want to add them to the environment.|This role could be either a data engineer or a data platform engineer. The project member would not be actively involved in creating or maintaining data pipelines, but would need access to the environment level.|
|**Environment**|**Environment Admin**|This role manages environment settings, reads project documentation, and deploys either through the API, CLI, or Coalesce App.• Environment roles have no organization permissions• Environment roles have access to view a list of projects• Environment roles must be added to the org, project, and then the environment|Data platform engineer or operations who would approve deployments and schedule jobs.|
|**Environment**|**Environment Reader**|This role only has access to the documentation for the environment they are added to. They have access to certain API functions to get deployment information.|Business analyst or data analyst.|

---

## Service Accounts in Coalesce

Service accounts are non-human accounts used to run automated processes, deployments, and scheduled jobs in Coalesce. Unlike individual user accounts, service accounts aren’t tied to a specific employee. This makes them more reliable for production workloads and easier to manage when team members change.

This guide explains how to configure service accounts, assign the correct roles, set up authentication, and follow security and licensing best practices.

## Role and Permission Recommendations

Service accounts should follow the principle of least privilege. Only assign the roles needed for their intended purpose.

[Recommended roles][]:

- Org Contributor  
- Org Member  
- Project Member  
- Environment Reader  
- Environment Admin (only if the service account must deploy or refresh)

If a service account is responsible for deployments or refreshes, it must have the **Environment Admin** role.

## Authentication Method

For service accounts, **key pair authentication** is strongly recommended. Key pairs provide stable machine-to-machine connections, reduce overhead, and avoid issues with expiring tokens. OAuth or username/password authentication should be avoided when setting up service accounts.

## Use Cases and Best Practices

Service accounts are best suited for tasks where human intervention isn’t required:

- **Production deployments and refreshes**: Keeps these processes separate from personal accounts.  
- **Scheduled jobs and automation**: Ensures jobs continue running even if team members leave.  
- **CI/CD processes**: Allows external tools to automate deployments securely.  

## Environment Strategy

Use personal accounts for development work. Reserve service accounts for higher environments such as QA, UAT, and Production.  

For stronger security isolation, consider creating separate service accounts for each environment.

## Licensing and Management

- Service accounts do not count toward your [Coalesce license allocation][].  
- Most organizations use one or two service accounts.  

## Security Considerations

To keep service accounts secure, follow these best practices:

- Use distribution list emails instead of personal email addresses.  
- Distribute private keys securely using encrypted methods.  
- Assign only the minimum required privileges for the account’s role.  

## Steps to Create a Service Account in Coalesce

Follow these steps to create and configure a service account in Coalesce:

1. Go to the top right of the Coalesce interface and click **Organization Settings**.
2. Click **Add New User**.  
   1. Set the first name as **Service** and last name as **Account** (or use a similar naming convention).  
   2. Use a distribution email address such as `coalesce_svc@yourcompany.com` instead of a personal email.  
3. Assign the service account to the appropriate role based on its intended use. You'll need to set it on the Org, Project, and Environment level. If the service account is used for deployments, ensure it has the necessary [environment permissions][].

    
  
4. Choose your authentication method. We recommend:
   - [Key Pair][] for Snowflake
   - [Machine-to-Machine][] for Databricks
5. If your Coalesce project integrates with Snowflake, create a corresponding service account in Snowflake using SQL:

```sql
   USE ROLE USERADMIN;
   CREATE USER <SERVICE-USER-USERNAME> PASSWORD='<SERVICE-USER-PASSWORD>'
   FIRST_NAME='DCR' LAST_NAME='Service User'
   EMAIL='<SERVICE-USER-EMAIL-ADDRESS>';
```

[Recommended roles]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Coalesce license allocation]: /docs/organization-and-accounts/organization-management/user-licensing
[Key Pair]: /docs/setup-your-project/connection-guides/databricks#machine-to-machine-coalesce-configuration
[Machine-to-Machine]: /docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[environment permissions]: #role-and-permission-recommendations

---

## Google SSO

In this guide, you'll learn how to set up Google as a Single Sign-On provider in Coalesce.

:::info[Google Administrator]

You must be a Google Cloud or Google Workspace Administrator with access to the Google Cloud Console to complete this process.

:::

## Before You Begin

### Check Your Subdomain

Your subdomain is the subdomain of your Coalesce instance. For example, if you log in at `https://yoursubdomain.app.coalescesoftware.io/`, your subdomain is `yoursubdomain`.

To check if you already have a subdomain, go to your organization's [Single Sign-On settings][].

If you don't have a subdomain, you can add one in the **Subdomain** box. Coalesce automatically configures your subdomain based on the name you enter. Check with your IT team before adding it to your organization's settings.

## Google Setup

Google uses OpenID Connect for SSO. You'll create an OAuth 2.0 Client ID in the Google Cloud Console, which produces the Client ID and Client Secret you enter in Coalesce.

### Configure the OAuth Consent Screen

1. In the [Google Cloud Console][], select or create the project you want to use for SSO.
2. If you have not already configured the OAuth consent screen, go to **APIs & Services > OAuth consent screen**, then click **Get Started**.
3. On **App Information**, enter the required details:
   1. **App name**
   2. **User support email**
4. On **Audience**, select **Internal**. This limits the application to users in your organization.
5. On **Contact Information**, enter the email address for someone who needs to be notified of changes by Google.
6. On **Finish**, accept the Google API Services User Data Policy, then click **Create**.

### Configure Credentials

1. Go to **APIs & Services > Credentials**.
2. Click **Create Credentials**, then select **OAuth client ID**.

    

3. Set **Application type** to **Web application** and enter a name, for example `Coalesce SSO`.
4. Under **Authorized redirect URIs**, add the Coalesce callback URL. Replace `yoursubdomain` with the subdomain you created in Coalesce:
   - `https://yoursubdomain.app.coalescesoftware.io/login/callback`
5. Optional: Under **Authorized JavaScript origins**, add your Coalesce login origin:
   - `https://yoursubdomain.app.coalescesoftware.io`

    

6. Click **Create**. Google displays a popup with your Client ID and Client Secret. Copy both values. You'll enter them in the Coalesce SSO configuration. You can also retrieve the Client Secret later from the credential details page.

## Coalesce SSO Configuration

1. Open a new window.
2. Sign in to your Coalesce App using your username and password.
3. Go to **Organization Settings > Single Sign-On**.
4. Use the table below to map each Coalesce field to the Google values you collected:

    | Field | Description |
    | --- | --- |
    | Authority | The system being used for Single Sign-On. Choose **Other**. |
    | Subdomain | The subdomain you created during **Before You Begin**. |
    | Authorization Server | `https://accounts.google.com` |
    | OIDC Client ID | The Client ID from Google. It ends in `.apps.googleusercontent.com`. |
    | Server-Side Authorization | Toggle on. Google blocks browser cross-origin requests to its OpenID configuration and token endpoints, so server-side authorization is required. |
    | Authorization Endpoint | `https://accounts.google.com/o/oauth2/auth`. This field appears when **Server-Side Authorization** is on. |
    | Client Secret | The Client Secret from Google. Required for OIDC providers such as Google that require a client secret for server-side authentication. Click **Edit** to enter or update it. |

    

5. Click **Save**.
6. Log out of Coalesce.
7. Go to your Coalesce login page, for example `https://yoursubdomain.app.coalescesoftware.io/`, and click **Use Single Sign-On** to log in with Google.

If you see an error message instead of the sign-on button, check that you entered the Client ID, Client Secret, and Authorization Endpoint correctly in your Coalesce SSO settings. If the problem persists, reach out to our Support Team.

## What's Next?

- If your SSO appears successful but Coalesce shows a blank or spinning screen, see [Troubleshooting Browser Login Issues][].
- Review [Troubleshooting Common SSO Errors][] for additional SSO error messages and fixes.

[Single Sign-On settings]: /docs/organization-and-accounts/organization-management/single-sign-on
[Google Cloud Console]: https://console.cloud.google.com/
[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Troubleshooting Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors

---

## Single Sign-On

Learn how to enable Single Sign-On for your organization.

## Important Single Sign-on Information

- **Multi-org support**: SSO users part of multiple organizations can log into each organization.
- **Single Sign-On users with username and password**: Users that have been configured directly in Coalesce will be able to login with a username/password even if SSO has been enabled so long as their Coalesce-native user record remains active. If you do not wish to allow username/password-based authentication, you will need to [disable or delete][] the users.
- **Provisioned SSO users**: Single Sign-On (SSO) provisioned users are created just in time, when the user initiates their initial login to Coalesce via you SSO provider. Users are not pre-provisioned prior to initial login.
- **Multiple records for the same person**: While email is used as the unique username for a user within Coalesce, a single email may end up with multiple active user records, each with a unique User ID, within a given Coalesce account. This occurs when a user has been set up directly within Coalesce, as well as provisioned using your single sign-on (SSO) provider. Review [Manage Users][] to see instructions on disabling or deleting extra users.
- SSO users will need a new [token][] for each new environment created.

## Removing SSO

Once SSO has been enabled for an account, there isn't a way to remove SSO. You can change the configuration. Keep in mind, this can have unintended consequences such as locking users out of the account.

:::warning[Access Tokens and SSO]

- Access tokens function independently of SSO. Disabling a user in your SSO platform such as Okta, does not remove the access token. You'll need to also remove the user directly in Coalesce to remove any tokens attached to that user.
- Access tokens are regenerated after each login. Previously issued tokens remain valid. Tokens never expire across all environments and projects.

Access tokens are removed when:

- User is deleted in Coalesce.
- User is disabled in Coalesce.
- Password changed in Coalesce.
- Email address change in Coalesce.
- SSO users will need a new token for each new environment created.
- Turning Multi-factor Authentication on and off.

:::

<DocCardList />

[disable or delete]: /docs/organization-and-accounts/organization-management/users
[Manage Users]: /docs/organization-and-accounts/organization-management/users
[token]: /docs/api/authentication/#getting-your-access-token

---

## JumpCloud SSO

In this guide, you’ll learn how to set up JumpCloud in Coalesce.

:::info[JumpCloud Administrator]

You must be a JumpCloud Administrator to complete this process.

:::

## Before You Begin

### Check Your Subdomain

## JumpCloud Setup

1. In the JumpCloud console, under User Authentication, click **SSO Applications > Add New Application**.
2. Select **Custom Application** from the list.

    

3. Enable **Manage Single Sign-On(SSO)** and select **Configure SSO with OIDC**.

    

4. Enter the **Display Label** and any other information you want about the app. Make sure S**how this application in User Portal** is checked.
5. On the **Configure Your Application** screen, enter the Redirect URL or callback URI and the Login URL. The subdomain is the one you created for Coalesce in the SSO config.
   1. Callback: `https://yoursubdomain.app.coalescesoftware.io/login/callback`
   2. Login: `https://yoursubdomain.app.coalescesoftware.io/login`
6. Set **Client Authentication Type** to **Public (None PKCE)**.

    

7. Scroll down and enable **Attribute Mapping** for both email and profile. This allows Coalesce to receive the email address and name so the account can be set up. The defaults can be left as is.

    

8. Click **Activate**. After activating, you'll get a popup with the Client ID you'll use in the Coalesce SSO configuration.

    

## Coalesce SSO Configuration

1. Open a new window.
2. Sign in to your Coalesce application using username and password.
3. Go to **Organization Settings > Single Sign-On**.
4. Enter in the following information:

    | Field | Description |
    | --- | --- |
    | Authority | The system being used for Single Sign On. Choose **Other**. |
    | Subdomain | The one you created during "Before You Begin.|
    | Authorization Server | `https://oauth.id.jumpcloud.com`|
    | OIDC Client ID | The Client ID from JumpCloud. |
    | Server-Side Authorization (Optional) | Toggle on to add an authorization URL. Use this when the authorization server blocks access to the OpenID configuration or token endpoints.|
    | Authorization Endpoint (Available with Server-Side Authorization ) | The authorization URL to redirect to.|

    

5. Logout and go to your Coalesce login page, for example `https://testapp.app.coalescesoftware.io/`.
6. Click the Single Sign-On button to login. You'll be redirected to the JumpCloud login page if not logged in already, otherwise you will be logged in straight away.

If instead of a button you see an error message, check to make sure you correctly entered all the fields in your Coalesce SSO settings. If the problem persists please reach out to our Support Team.

---

## What's Next?

* If your SSO appears successful but Coalesce shows a blank or spinning screen, see [Troubleshooting Browser Login Issues][].
* [Common SSO Errors][]

---

[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors

---

## Microsoft Entra ID

In this guide, you’ll learn how to set up Microsoft Entra ID in Coalesce.

:::info[Microsoft Entra ID Administrator]

You must be a Microsoft Entra ID Administrator to complete this process.

:::

## Before You Begin

### Get Your Subdomain

### Microsoft Entra ID Permissions

When selecting **Use Single-Sign On** with Microsoft Entra ID, you may be prompted to [grant Coalesce permission][] to:

- Sign you in and read your profile
- Maintain access to data you have given it access to
- Microsoft Graph:
  - `email`
  - `profile`
  - `User.Read`
  - `openid`

These permissions can be pre-approved for future users by an admin in Microsoft Entra ID.

1. Go to **Manage > App Registrations > Your App Registration > API Permissions**.
2. Then select **Add a permission > Microsoft Graph > Delegated Permissions** and then select the desired permissions to pre-approve for the non-admin users.

## Configure Microsoft Entra ID

To use Microsoft Entra ID as your Single Sign-On provider, you'll want to create a new **App Registration** in Azure.

1. Go to the **Overview** panel in Azure Active Directory
2. Click the **+Add** dropdown
3. Select **App Registration**

    

4. On the registration page for this newly created integration, enter the following:
   1. **Name** - this is typically going to be `Coalesce` but any friendly name works
   2. **Supported Account Types** - choose which Account types to support, see the following screenshot.
   3. Make sure you choose **Single Page Application (SPA)**.
   4. **Redirect URI** - The redirect URI should be formatted as follows - `https://mySubdomain.<app_domain>/login/callback`.
      1. Create a subdomain if one hasn’t already been defined for your organization. We recommend choosing a name specific to your organization. If the Subdomain box in your [Single Sign-On][] Settings is blank, you can create one by adding it in the subdomain box.

    

    :::warning[Supported Account Types]

    **Personal Microsoft accounts only** is **not** a supported option for Coalesce Microsoft Entra ID SSO.

    :::

5. Click **Register** to create the integration. You'll now be at a window with all your **App Registration** settings.
6. While still in Azure, go to **Manage > Authentication**.
7. Scroll down to **Implicit grant and hybrid flows**. Select both: Access tokens (used for implicit flows) and ID tokens (used for implicit and hybrid flows) and Save.

    

8. Go to **Manage > Token configuration**.
9. Click **Add optional claim**.
10. Choose **ID** as the Token Type.
11. Select the following:
    1. `email`
    2. `family_name`
    3. `given_name`

    

12. Go back to the **Overview**. You'll need the information to finish configuration in Coalesce.

    

## Configure Coalesce Microsoft Entra ID Settings

1. Open a new window.
2. Sign in to your Coalesce application using username and password.
3. Go to **Organization Settings > Single Sign-On**.
4. Enter in the following information:

    | Field | Description |
    | --- | --- |
    | Authority | The system being used for Single Sign On. Choose **Azure**. |
    | Subdomain | This will be the same as `mySubdomain`. Not the entire redirect URI.|
    | Authorization Server Single for tenant integrations.  |`https://login.microsoftonline.com/[tenantID]/`|
    | Authorization Server for multi-tenant and multi- personal integrations | `https://login.microsoftonline.com/common/`|
    | OIDC Client ID | Refer to the Application (client) ID in the "Essentials" section on the overview page for your App Registration. |
    | Server-Side Authorization (Optional)| Toggle on to add an authorization URL. Use this when the authorization server blocks access to the OpenID configuration or token endpoints.|
    | Authorization Endpoint (Available with Server-Side Authorization ) | The authorization URL to redirect to.|

    

5. Once you've filled out the SSO settings in Coalesce, click **Save**.
6. Log out of Coalesce.
7. Go to your SSO URL, which will be formatted like - `https://mySubdomain.<app_domain>` - and click on the **Use Single Sign-On** button to log in using SSO.

If instead of a button you see an error message, check to make sure you correctly entered all the fields in your Coalesce SSO settings. If the problem persists please reach out to our Support Team.

---

## What's Next?

- If your SSO appears successful but Coalesce shows a blank or spinning screen, see [Troubleshooting Browser Login Issues][].
- [Common SSO Errors][]

---

[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Single Sign-On]: /docs/organization-and-accounts/organization-management/single-sign-on/
[grant Coalesce permission]: https://learn.microsoft.com/en-us/entra/identity-platform/scopes-oidc
[Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors

---

## Okta SSO

In this guide, you’ll learn how to set up Okta SSO in Coalesce.

:::info[Okta Administrator]

You must be a Okta Administrator to complete this process.

:::

## Before You Begin

### Check Your Subdomain

## Create Okta App Integration

To use Okta as your Single Sign-On provider, you'll want to create a new **App Integration** in Okta.

1. Open the admin panel for your Okta organization
2. Click on **Applications**.

    

3. Click on **Create App Integration**.
4. Select **OIDC - OpenID Connect** as the **Sign-in method** and **Single-Page Application** as the **Application Type** and then create the new app integration.

    

5. On the settings page for this newly created integration, enter the following:
    1. **App Integration Name** - this is typically going to be `Coalesce` but any friendly name works
    2. **Sign-in redirect URI** - by default this is `http://localhost:8080/login/callback` which you'll want to change to `https://mySubdomain.<app_domain>/login/callback`. `mySubdomain` is typically the name of your organization.
    3. **Controlled Access** - select whichever setting is appropriate for your organization
6. Click **Save**. You'll now be at a window with all your **App Integration** settings. Keep this browser tab open as you'll need to enter some information from it into Coalesce.

## Configure Coalesce Okta Settings

1. Open a new window.
2. Sign in to your Coalesce application using username and password.
3. Go to **Organization Settings > Single Sign-On**.
4. Enter in the following information:

    | Field | Description |
    | --- | --- |
    | Authority | The system being used for Single Sign On. Choose Okta. |
    | Subdomain | This will be the same as `mySubdomain`. Not the entire redirect URI.|
    | Authorization Server | Refer to the URL you use for your Okta account. Your Authorization Server will be the base URL. For example: `https://<yourcompany>.okta.com` |
    | OIDC clientID | This will be the same as the Client ID field in the settings of your Okta app integration. |
    | Server-Side Authorization (Optional)| Toggle on to add an authorization URL. Use this when the authorization server blocks access to the OpenID configuration or token endpoints.|
    | Authorization Endpoint (Available with Server-Side Authorization ) | The authorization URL to redirect to.|

    

5. Once you've filled out the SSO settings in Coalesce, click **Save**.
6. Log out of Coalesce.
7. Go to your SSO URL, which will be formatted like - `https://mySubdomain.<app_domain>` - and click on the **Use Single Sign-On** button to log in using SSO.

If instead of a button you see an error message, check to make sure you correctly entered all the fields in your Coalesce SSO settings. If the problem persists please reach out to our Support Team.

---

## What's Next?

* If your SSO appears successful but Coalesce shows a blank or spinning screen, see [Troubleshooting Browser Login Issues][].
* [Common SSO Errors][]

---

[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors

---

## Ping Identity SSO

In this guide, you’ll learn how to set up Ping Identity authentication in Coalesce.

:::info[Ping Identity Administrator]

You must be a Ping Administrator to complete this process.

:::

## Before You Begin

### Check Your Subdomain

## Create a Ping Identity Application

1. In Ping go to the **Applications** page, and create a new application.
2. Give the application a name.
3. Select **Single-Page** as the Application Type.
4. Click **Save**.

    

5. After saving, you’ll be taken to the **Application** overview screen.
6. Click on **Configuration**, then edit.
7. Set the following confirmation options:
    1. Select all options under **Response Type**:
        1. Code
        2. Token
        3. ID
    2. **Grant Type**:
        1. Click Authorization Code
        2. PKCE Enforcement is Optional
        3. Click Implicit
    3. Under **Redirect URIs** enter your Coalesce instance URL with login/callback added.
        1. `https://<your Coalesce app domain>/login/callback`
        2. For example: `https://testapp.app.coalescesoftware.io/login/callback`
    4. The other configuration options can be left as default.
8. Click **Save**.
9. After saving, you’ll be taken to the **Application** overview screen.
10. Next, you’ll make sure your allowed scopes are set. Click on **Resources**, then edit.
11. Make sure the following [scopes][] are set:
    1. `openid`
    2. `email`
    3. `profile`

    

12. Make sure your application is turned on by toggling the switch near the X.

## Gather Your Ping SSO Information

You are gathering your subdomain, Authorization Server, and OIDC clientID.

1. On the **Application overview screen**, click URLs to open a drop-down.

    

2. Copy the **Authorization URL**. You only need up to the `/as`. Leave off the trailing slash.
    1. For example: `https://auth.pingone.com/8d472703-1eaf-491b-a425-91aff175d01f/as`.

## Get your OIDC Client ID

1. On the **Application overview screen**, copy the Client ID.

## Configure Coalesce Ping Settings

1. Open a new window.
2. Sign in to your Coalesce application using username and password.
3. Go to **Organization Settings > Single Sign-On**.
4. Enter in the following information:

    | Field | Description |
    | --- | --- |
    | Authority | The system being used for Single Sign On. Choose Other. |
    | Subdomain | This will be the same as **Subdomain**. Not the entire URL.|
    | Authorization Server |`https://auth.pingone.com/8d472703-1eaf-491b-a425-91aff175d01f/as`|
    | Server-Side Authorization (Optional) | Toggle on to add an authorization URL. Use this when the authorization server blocks access to the OpenID configuration or token endpoints.|
    | Authorization Endpoint (Available with Server-Side Authorization ) | The authorization URL to redirect to.|

    

5. Go to your SSO URL, which will be formatted like - `https://mySubdomain.<app_domain>` - and click on the **Use Single Sign-On** button to log in using SSO.

If instead of a button you see an error message, check to make sure you correctly entered all the fields in your Coalesce SSO settings. If the problem persists please reach out to our Support Team.

---

## What's Next?

* If your SSO appears successful but Coalesce shows a blank or spinning screen, see [Troubleshooting Browser Login Issues][].
* [Common SSO Errors][]

---

[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[scopes]: https://docs.pingidentity.com/r/en-us/pingfederate-112/pf_scopes_and_scope_management
[Common SSO Errors]: /docs/faq/general-setup-faq-troubleshooting/common-sso-errors

---

## User Licensing

:::info[Licensing Policies]

This article details Coalesce's current licensing policies and procedures. Coalesce reserves the right to change its licensing policies and procedures in the future. These changes will not be imparted without prior notification to existing customers.

:::

Coalesce is licensed on a per-individual developer basis. **Each developer account that is not disabled or deleted is counted as a license. A developer is defined as an individual who is actively--within the last 90 days--building objects and pipelines in Coalesce Workspaces.**

The Coalesce platform does not prevent an organization from creating additional developers once it has reached its license limit. Rather, license adherence is monitored via auditing by the Coalesce team.

Coalesce allows Environment Admin and Environment reader accounts to be used for deploy and refresh activities to account for organizations that don't wish to use individual developer accounts to do so. **Service accounts used for deploy and refresh activities do not count against the license allocation.**

## RBAC Organizations

If you are using RBAC to manage your users, the following applies for license allocation. To learn more about RBAC, see [Role-Based Access Control][].

Any user assigned **at least one** of the following roles, **will count towards your license allocation.**

- Org Admin
- Project Admin
- Project Architect
- Project Contributor

Users who are **only assigned the following roles** are considered read-only users and **will not count against the license allocation.** The user can't be assigned any of the above roles.

- Org Contributor
- Org Member
- Project Member
- Environment Reader
- Environment Admin

[Role-Based Access Control]: /docs/organization-and-accounts/organization-management/role-based-access-control/

---

## Manage Users

Organizations have the ability to manage users and their roles by going to **User Menu** >**Org Settings** > **Users**.

## Add a New User

To add a new user to your organization, click the **Add New User** button.

In the **Create User** pop-up, enter the new user's account information and select **Submit**.

| Field | Description |
| --- | --- |
| First Name | User's preferred first name. |
| Last Name | User's preferred last name (hyphens are accepted). |
| Enable Password Input | If **unchecked**, an email will be sent to the user so they can set their password. |
| Password | If **Enable Password Input** is **checked**, the user's password will need to be manually entered in this field. |
| Role | There are two options for role designation:  **Admin**: Has access to **Org Settings** such as **Single Sign-On Settings** and management of **Users**   **User**: Does not have access to **Org Settings** |

## Edit User

An **Admin User** has the option to modify a user's first and/or last name as well as the user's designated role (**User** vs **Admin**).

## Disabling, Activating, and Deleting Users

Users can be deleted, disabled, or activated from the **Edit User** screen.

- Deleting a user permanently removes all personally identifiable information (PII), including any user secrets. This action is irreversible.
- Disabling a user will prevent them from logging in, and will log them out shortly if they happen to be logged in at the time. They can be re-enabled or activated at any time.

## Removing SSO Users

- Single Sign-On users with a username and password: Users that have been configured directly in Coalesce will be able to log in with a username and password even if SSO has been enabled or removed, as long as their Coalesce-native user record remains active. If you do not wish to allow username and password-based authentication, you must disable or delete these users.
- Users may still appear in your SSO provider's user list, but they will not be able to log in. These users must be removed manually. We recommend reviewing all deleted users to ensure they have been removed from the SSO provider.

:::warning[Access Tokens and SSO]

- Access tokens function independently of SSO. Disabling a user in your SSO platform does not remove access tokens. You must also remove the user directly in Coalesce to revoke any tokens associated with that user.
- Access tokens never expire across all environments and projects.
- Access tokens are regenerated after each login. Previously issued tokens remain valid.

Access tokens are removed when:

- A user is deleted in Coalesce.
- A user is disabled in Coalesce.
- A user's password is changed in Coalesce.
- A user's email address is changed in Coalesce.
- An SSO user requires a new token for each new environment created.
- Multi-factor Authentication is turned on or off.

:::

## When To Contact Support

Org admins can add users, change roles, and manage access directly in the Coalesce App. Some account and access requests are handled by the Coalesce team and are not self-service.

**Self-service (org admins):**

- Add users to your organization.
- Edit user names and roles (Admin vs User).
- Disable, activate, or delete users.
- Assign RBAC roles at the organization, project, and environment levels.

**Support-assisted (contact Coalesce):**

- New Catalog accounts for new customers.
- New customer organization setup.
- POC account setup and adding specific users.

When requesting new accounts or access, include the following in your support request:

- Customer or organization name.
- Key user email addresses.
- Integrations needed (for example, Snowflake, Sigma, Coalesce Transform).
- Timeline and any organizational context (for example, merger, hierarchy change).

---

## What's Next?

- [Single Sign-On](/docs/organization-and-accounts/organization-management/single-sign-on)

---

---

## User Menu Overview

Towards the top right of the Coalesce interface you'll find a small person icon, through which you can access the **User Menu**. From this menu (and depending on your permissions) you can configure your Git integration, Single Sign On settings, and add/remove/edit users on your account.

## Available User Menu Items

There are two subsections in the User Menu.

### User Settings

- [Git Settings][] - Configuration of your connection to Git
- [Support Information][] - Useful information to provide when contacting Support
- [Change Password][] - You can change your user's login password here

:::info[Org Settings]

Org Settings are only viewable by Admin users.

:::

## Org Settings

- [Users][] - for adding and removing users
- [Single Sign-On][] - configuration of SSO settings
- Preferences - for setting the [Parser][] sample size

[Git Settings]: /docs/git-integration/set-up-your-git-integration
[Support Information]: /docs/get-started/support-information
[Change Password]: /docs/organization-and-accounts/account-management/change-password/
[Users]: /docs/organization-and-accounts/organization-management/users
[Single Sign-On]: /docs/organization-and-accounts/organization-management/single-sign-on/
[Parser]: /docs/build-your-pipeline/parsers/

---

## Coalesce Internal Macros

<!-- vale Coalesce.Headings = NO -->

:::warning[Unexpected Behavior]

Editing internal macros can cause unexpected behavior.

:::

## Using Internal Macros

To use an internal macro, add the macro into the Macros in **Build Settings**. You can make any edits you want, but you need to keep the name of the macro the same.

## get_source_transform()

Gets the source column transform for a node’s columns.

```sql title="get_source_transform"
{%- macro get_source_transform(col) -%}
    {% if col.hashDetails %}
        {{ hash(col.hashDetails.columns, algo=col.hashDetails.algorithm, datatype=col.dataType) }}
    {% elif col.transform | trim != '' %}
        {{ col.transform }}
    {% elif col.sourceColumns[0].node and col.sourceColumns[0].node.name and col.sourceColumns[0].column and col.sourceColumns[0].column.name %}
        "{{ col.sourceColumns[0].node.name }}"."{{ col.sourceColumns[0].column.name }}"
    {% else %}
        NULL
    {% endif %}
{% endmacro -%}
```

## stage()

Defines the individual stages (steps) on node SQL execution.

```sql title="stage"
{% macro stage(stage_name, continue_on_failure=true, type="sql") %}
---
coalesce_stage_start
name: {{ stage_name }}
type: {{ type }}
continue_on_failure: {{ continue_on_failure }}
code:
{% endmacro -%}
```

## get_value_by_column_attributes()

 Filters columns in a node to retrieve only ones that match a provided attribute.

```sql title="get_value_by_column_attributes"
{%- macro get_value_by_column_attribute(column_attribute, value="name") -%}
    {%- set filtered_columns_by_attribute = columns | selectattr(column_attribute) | list -%}
    {%- if filtered_columns_by_attribute | length > 0 -%}
        {{- (filtered_columns_by_attribute | first)[value] -}}
    {%- else -%}
        ## ERR: COLUMN_NOT_FOUND_WITH_ATTRIBUTE_{{ column_attribute }} ##
    {%- endif -%}
{%- endmacro -%}

```

## hash()

Creates a hash of a certain column using a selected algorithm.

```sql title="hash()"
{%- macro hash(columns, algo='MD5', delimiter='||', datatype='CHAR(32)') -%}
    {% for col in columns %}
        {# set algo function #}
        {% set algo_start = algo + '(' %}
        {% set algo_end = ')' %}
        {% if algo | upper == 'SHA256' %}
            {% set algo_start = 'SHA2(' %}
            {% set algo_end = ', 256)' %}
        {% endif %}  
        {# end set algo function #}

        {%- if loop.first %}CAST( {{ algo_start }}{% endif -%}
            NVL(CAST(
                {%  if col is string %} 
                    {{ col }}
                {% elif col.transform | trim != '' %}
                    {{ col.transform }}
                {% elif col.sourceColumns[0] %} 
                    "{{ col.sourceColumns[0].node.name }}"."{{ col.sourceColumns[0].column.name }}"
                {% else %}
                    null
                {%  endif %} 
            AS VARCHAR), 'null')
        {%- if not loop.last %} || {% if delimiter != '' %} '{{ delimiter }}' || {% endif -%} {% endif -%}
        {%- if loop.last %}{{ algo_end }} AS {{ datatype }} ){% endif -%}
    {% endfor %}
{%- endmacro -%}

```

## test_stage()

Use the test stage to expand your node testing capabilities.

Learn more in [How to Use the Coalesce Test Stage Macro][].

```sql
{% macro test_stage(stage_name, continue_on_failure=true, type="sql") %}
coalesce_stage_start
name: {{ stage_name }}
type: {{ type }}Test
continue_on_failure: {{ continue_on_failure }}
code:
{% endmacro -%}
```

## ref_no_link()

Creates a fully qualified name reference to the storage location mapping of another node without an explicit link in the node graph.

Learn more about [Ref Functions][].

```sql title="ref_no_link"
{%- macro ref_no_link(location_name, node_name) -%}
        {%- set foundMapping = { 'flag': false } -%}
        {%- for storage in nodeMetadata.mapping -%}
            {%- if nodeMetadata.mapping[storage].locationName ==  location_name -%}
                {%- if foundMapping.update({'flag': true}) -%}{%- endif -%}
                {{-  location_name | export_ref(node_name, ref_type='ref_no_link') -}}
                "{{- nodeMetadata.mapping[storage].database -}}"."{{- nodeMetadata.mapping[storage].schema -}}"."{{- node_name -}}"
            {%- endif -%}
        {%- endfor -%}
        {%- if not foundMapping.flag -%}
            {{-  location_name | invalid_export_ref(node_name, ref_type='ref_no_link') -}}
        {%- endif -%}
    {%- endmacro -%}

```

## ref_link()

Resolves to a fully qualified name reference to the storage location mapping of a node with an explicit link in the node graph without creating a string in the rendered template.

Learn more about [Ref Functions][].

```sql title="ref_link"

{%- macro ref_link(location_name, node_name) -%}
        {%- set foundMapping = { 'flag': false } -%}
        {%- for storage in nodeMetadata.mapping -%}
            {%- if nodeMetadata.mapping[storage].locationName ==  location_name -%}
                {%- if foundMapping.update({'flag': true}) -%}{%- endif -%}
                {{-  location_name | export_ref(node_name, ref_type='ref_link') -}}
            {%- endif -%}
        {%- endfor -%}
        {%- if not foundMapping.flag -%}
            {{-  location_name | invalid_export_ref(node_name, ref_type='ref_link') -}}
        {%- endif -%}
    {%- endmacro -%}

```

## ref()

Resolves to a fully qualified name reference to the storage location mapping of a node with an explicit link in the node graph.

Learn more about [Ref Functions][].

```sql title="ref"
{%- macro ref(location_name, node_name) -%}
        {%- set foundMapping = { 'flag': false } -%}
        {%- for storage in nodeMetadata.mapping -%}
            {%- if nodeMetadata.mapping[storage].locationName ==  location_name -%}
                {%- if foundMapping.update({'flag': true}) -%}{%- endif -%}
                {{-  location_name | export_ref(node_name, ref_type='ref') -}}
                "{{- nodeMetadata.mapping[storage].database -}}"."{{- nodeMetadata.mapping[storage].schema -}}"."{{- node_name -}}"
            {%- endif -%}
        {%- endfor -%}
        {%- if not foundMapping.flag -%}
            {{-  location_name | invalid_export_ref(node_name, ref_type='ref') -}}
        {%- endif -%}
    {%- endmacro -%}
```

[Ref Functions]: /docs/reference/ref-functions
[How to Use the Coalesce Test Stage Macro]: /docs/build-your-pipeline/build-how-to/how-to-use-the-coalesce-test-stage-macro/

---

## Coalesce Syntax

| Name | Description | Example | Documentation |
| --- | --- | --- | --- |
| **Ref functions** | Generate fully qualified database object names. | Snowflake: `"myDB"."mySCHEMA"."CUSTOMER"`. Databricks or BigQuery: `` `myDB`.`mySCHEMA`.`CUSTOMER` `` | [Ref Functions][] |
| **Helper Tokens** | Provide column or node context to a bulk operation. | Snowflake: `"C_CUSTKEY"`. Databricks or BigQuery: `` `C_CUSTKEY` `` | [Helper Tokens][] |
| **Selector Queries** | Search for or include Nodes in your Graph, Subgraphs, and/or Jobs. | `+{ location: WORK name: DIM_CUSTOMER }+}` = `results in DIM_CUSTOMER + all predecessor and successor objects` | [Search Your Node Data][] |
| **Macros** | Create reusable code in your project. | `{%- macro even_odd(column) -%} CASE WHEN MOD({{ column }}, 2) = 0 THEN 'EVEN' ELSE 'ODD' END {%- endmacro %}` | [Macros][] |
| **Jinja** | Use Jinja with Macros and helper tokens. | `{{hello_goodbye('{{SRC}}')}}` | [Using Jinja with SQL][] |

[Ref Functions]: /docs/reference/ref-functions
[Helper Tokens]: /docs/reference/helper-tokens
[Search Your Node Data]: /docs/reference/selector
[Macros]: /docs/reference/macros
[Using Jinja with SQL]: /docs/reference/jinja/

---

## CONCAT

`CONCAT(string1, string2, ...)` combines two or more strings into a single string.

## Syntax and Parameters

`CONCAT(string1, string2[, string3, ...])`

* **Parameters** - String expressions (at least two required):
  * Can be literal strings, column references, or expressions that result in strings
  * NULL values are treated as empty strings
  * Numbers are automatically converted to strings
  * No limit to number of parameters

## Basic Usage

Simple string combination examples:

```sql
CONCAT('Hello', ' ', 'World') -- Returns "Hello World"
CONCAT('ID-', '12345') -- Returns "ID-12345"
CONCAT('Price: $', 19.99) -- Returns "Price: $19.99"
```

### Common Use Cases

**Creating supplier identifiers:**

```sql
CONCAT('SUP_', "S_SUPPKEY", '_', "S_NATIONKEY")
```

**Formatting contact information:**

```sql
CONCAT("SUPPLIER".S_NAME, ' (', "SUPPLIER".S_PHONE, ')')
```

**Building full supplier details:**

```sql
CONCAT(
    "SUPPLIER".S_NAME, ' - ',
    "SUPPLIER".S_ADDRESS, ' - ',
    'Account Balance: ', "SUPPLIER".S_ACCTBAL
)
```

## Using CONCAT in Coalesce

When using CONCAT in transforms and bulk editing, you can use [Helper Tokens][]:

```sql
CONCAT({{TGT}}, ' - Modified')
CONCAT('New - ', {{SRC_COL}})
```

The Helper Tokens automatically get replaced:

* `{{SRC}}` becomes the fully qualified source Node and column name
* `{{SRC_COL}}` becomes just the source column name

Examples using Helper Tokens:

```sql
-- Adding a prefix
CONCAT('ID-', {{SRC}})

-- Combining multiple columns
CONCAT(
    {{SRC}}, ' (',
    {{ ref('WORK', 'CUSTOMER') }}.email, ')'
)
```

[Helper Tokens]: /docs/reference/helper-tokens

---

## Using Conditionals in Coalesce

Learn how to implement if-else logic in Coalesce using CASE statements, conditional functions, and SQL updates to conditionally transform your data.

## What Are Conditionals?

Logic that executes different actions based on whether a condition is true or false. In Coalesce, you can implement conditionals through [Transforms][], [Pre-SQL][], or [Post-SQL][].

## Using CASE Statements in Transforms

### Basic Syntax

```sql
CASE 
  WHEN condition THEN result
  ELSE default_result
END
```

### Single Condition Example

To update `O_ORDERSTATUS` to `XXX` when `O_CUSTKEY` equals 14784007:

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
CASE 
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784007 THEN 'XXX'
  ELSE {{SRC}}
END
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'XXX'
  ELSE {{SRC}}
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'XXX'
  ELSE {{SRC}}
END
```

  </TabItem>
</Tabs>

### Multiple Conditions

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
CASE 
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784007 THEN 'XXX'
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784008 THEN 'YYY'
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784009 THEN 'ZZZ'
  ELSE {{SRC}}
END
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'XXX'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784008 THEN 'YYY'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784009 THEN 'ZZZ'
  ELSE {{SRC}}
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'XXX'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784008 THEN 'YYY'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784009 THEN 'ZZZ'
  ELSE {{SRC}}
END
```

  </TabItem>
</Tabs>

## Alternative Conditional Functions

Beyond CASE statements, Coalesce supports other conditional functions that can simplify transformations for specific use cases.

### IF/IFF Functions

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
IFF({{SRC_NODE}}."O_CUSTKEY" = 14784007, 'XXX', {{SRC}})
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
IF({{SRC_NODE}}.`O_CUSTKEY` = 14784007, 'XXX', {{SRC}})
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
IF({{SRC_NODE}}.`O_CUSTKEY` = 14784007, 'XXX', {{SRC}})
```

  </TabItem>
</Tabs>

### DECODE Function

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
DECODE({{SRC_NODE}}."O_CUSTKEY", 14784007, 'XXX', {{SRC}})
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE {{SRC_NODE}}.`O_CUSTKEY`
  WHEN 14784007 THEN 'XXX'
  ELSE {{SRC}}
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE {{SRC_NODE}}.`O_CUSTKEY`
  WHEN 14784007 THEN 'XXX'
  ELSE {{SRC}}
END
```

  </TabItem>
</Tabs>

## Using Pre-SQL and Post-SQL for Conditional Updates

Post-SQL executes after data loads, allowing bulk updates based on conditions.

### Basic UPDATE Statement

In the **Post-SQL** box:

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET "O_ORDERSTATUS" = 'XXX'
WHERE "O_CUSTKEY" = 14784007;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'XXX'
WHERE `O_CUSTKEY` = 14784007;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'XXX'
WHERE `O_CUSTKEY` = 14784007;
```

Some `UPDATE` Post-SQL patterns fail on **BigQuery** against views or certain partitioned tables. Run the Post-SQL step in your Project before you schedule it in a Job.

  </TabItem>
</Tabs>

### Multiple Updates Using Stages

Coalesce lets you run multiple SQL statements in Pre-SQL/Post-SQL either by inserting `{{stage('…')}}` or wrapping them in a `BEGIN … END` block.

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET "O_ORDERSTATUS" = 'XXX'
WHERE "O_CUSTKEY" = 14784007;

{{stage('Update Second Condition')}}

UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET "O_ORDERSTATUS" = 'YYY'
WHERE "O_CUSTKEY" = 14784008;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'XXX'
WHERE `O_CUSTKEY` = 14784007;

{{stage('Update Second Condition')}}

UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'YYY'
WHERE `O_CUSTKEY` = 14784008;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'XXX'
WHERE `O_CUSTKEY` = 14784007;

{{stage('Update Second Condition')}}

UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `O_ORDERSTATUS` = 'YYY'
WHERE `O_CUSTKEY` = 14784008;
```

  </TabItem>
</Tabs>

### Creating Conditional Audit Columns

Track data changes by adding new columns with conditional logic. Add a new column, `STATUS_CHANGE_FLAG`, then add the SQL to in Transform.

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
CASE 
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784007 THEN 'MODIFIED'
  WHEN {{SRC_NODE}}."O_CUSTKEY" = 14784008 THEN 'REVIEWED'
  ELSE NULL
END
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'MODIFIED'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784008 THEN 'REVIEWED'
  ELSE NULL
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE 
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784007 THEN 'MODIFIED'
  WHEN {{SRC_NODE}}.`O_CUSTKEY` = 14784008 THEN 'REVIEWED'
  ELSE NULL
END
```

  </TabItem>
</Tabs>

:::info[Adding Columns]

You must create and run the new column before applying Transformations.

:::

## Understanding Pre-SQL, Transforms, and Post-SQL

* **Pre-SQL**: Executes before any data movement begins. Use this for setup tasks like creating backup tables, temporary tables, or logging.
* **Post-SQL**: Executes after the data has been loaded into the target table. These are separate SQL statements (UPDATE, MERGE, etc.) that modify data already in place.
* **Transforms**: Execute as part of the data loading process. When Coalesce builds the SQL, your transform logic becomes part of the SELECT statement that loads data from source to target. All transformations happen during this single data movement operation.

## Choosing the Right Method

**Use Transform when you need:**

* Complex conditional logic
* Row-by-row processing
* Creating derived columns
* Logic integrated into data transformation pipeline

**Use Post-SQL when you need:**

* Simple bulk updates
* Updates based on joined data
* Updates after validation steps
* Temporary data fixes

**Use Pre-SQL when you need:**

* Backup tables before processing
* Logging or audit table setup
* Temporary staging table creation
* Initial state capture before transformations

**Use CASE when you have:**

* Multiple conditions to evaluate
* Range checks or inequalities
* Complex logic with AND/OR operators
* 3+ different scenarios requiring different actions

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
CASE 
  WHEN {{SRC}} > 1000 AND {{SRC}} < 5000 THEN 'Medium'
  WHEN {{SRC}} >= 5000 THEN 'Large'
  ELSE 'Small'
END
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE 
  WHEN {{SRC}} > 1000 AND {{SRC}} < 5000 THEN 'Medium'
  WHEN {{SRC}} >= 5000 THEN 'Large'
  ELSE 'Small'
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE 
  WHEN {{SRC}} > 1000 AND {{SRC}} < 5000 THEN 'Medium'
  WHEN {{SRC}} >= 5000 THEN 'Large'
  ELSE 'Small'
END
```

  </TabItem>
</Tabs>

**Use IF/IFF when you have:**

* Simple binary choices (true/false)
* Single value replacements
* One condition requiring clean readability

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
IFF({{SRC}} IS NULL, 'Unknown', {{SRC}})
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
IF({{SRC}} IS NULL, 'Unknown', {{SRC}})
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
IF({{SRC}} IS NULL, 'Unknown', {{SRC}})
```

  </TabItem>
</Tabs>

**Use DECODE when you have:**

* Exact value matching on a Snowflake Project
* Code-to-description replacements
* Lookup-style transformations

<Tabs groupId="conditionals-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
DECODE({{SRC}}, 
  1, 'Active',
  2, 'Inactive', 
  3, 'Pending',
  'Unknown')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE {{SRC}}
  WHEN 1 THEN 'Active'
  WHEN 2 THEN 'Inactive'
  WHEN 3 THEN 'Pending'
  ELSE 'Unknown'
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE {{SRC}}
  WHEN 1 THEN 'Active'
  WHEN 2 THEN 'Inactive'
  WHEN 3 THEN 'Pending'
  ELSE 'Unknown'
END
```

  </TabItem>
</Tabs>

[Transforms]: /docs/build-your-pipeline/transforms/
[Pre-SQL]: /docs/build-your-pipeline/pre-post-sql/
[Post-SQL]: /docs/build-your-pipeline/pre-post-sql/

---

## NOT IN

`NOT IN` filters rows by excluding values that match a list or subquery result. Use `NOT IN` in the [Join tab][] to refine which rows are included in your Node.

## Syntax

```sql
WHERE column NOT IN (value1, value2, ...)
```

Or with a subquery:

```sql
WHERE column NOT IN (subquery)
```

## Using `NOT IN` in the Join Tab

In the Join tab, add `NOT IN` to your `WHERE` clause to exclude rows matching specific values.

### Exclude a Static List

Filter orders with specific statuses:

```sql
FROM {{ ref('WORK', 'ORDERS') }} "ORDERS"
WHERE "ORDERS"."O_ORDERSTATUS" NOT IN ('F', 'O', 'P')
```

```sql
CASE 
    WHEN `ORDERS`.`O_ORDERPRIORITY` NOT IN ('1-URGENT', '2-HIGH') THEN 'Standard'
    ELSE `ORDERS`.`O_ORDERPRIORITY`
END
```

### Exclude Using A Subquery

Filter orders that don't have any associated line items:

```sql
FROM {{ ref('WORK', 'ORDERS') }} "ORDERS"
WHERE "ORDERS"."O_ORDERKEY" NOT IN (
    SELECT "L_ORDERKEY"
    FROM {{ ref('WORK', 'LINEITEM') }}
    WHERE "L_ORDERKEY" IS NOT NULL
)
```

:::caution[NULL Values Can Return Unexpected Results]

If the subquery contains NULL values, NOT IN may return zero rows. Always add `WHERE column IS NOT NULL` to your subquery to avoid this issue.

:::

## Platform Formatting

<Tabs>
  <TabItem value="bigquery" label="BigQuery" default>

    Use backticks for identifiers and AS for aliases:

    ```sql
    FROM {{ ref('WORK', 'ORDERS') }} AS `ORDERS`
    WHERE `ORDERS`.`O_CUSTKEY` NOT IN (
        SELECT `S_SUPPKEY` 
        FROM {{ ref('WORK', 'SUPPLIER') }}
        WHERE `S_SUPPKEY` IS NOT NULL
    )
    ```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    Use backticks for identifiers:

    ```sql
    FROM {{ ref('WORK', 'ORDERS') }} `ORDERS`
    WHERE `ORDERS`.`O_CUSTKEY` NOT IN (
        SELECT `S_SUPPKEY` 
        FROM {{ ref('WORK', 'SUPPLIER') }}
        WHERE `S_SUPPKEY` IS NOT NULL
    )
    ```

  </TabItem>
  <TabItem value="snowflake" label="Snowflake">

    Use double quotes for identifiers:

    ```sql
    FROM {{ ref('WORK', 'ORDERS') }} "ORDERS"
    WHERE "ORDERS"."O_CUSTKEY" NOT IN (
        SELECT "S_SUPPKEY" 
        FROM {{ ref('WORK', 'SUPPLIER') }}
        WHERE "S_SUPPKEY" IS NOT NULL
    )
    ```

  </TabItem>
</Tabs>

## Alternatives

### Using NOT EXISTS

NOT EXISTS handles NULL values more predictably and often performs better with large data sets:

```sql
FROM {{ ref('WORK', 'ORDERS') }} "ORDERS"
WHERE NOT EXISTS (
    SELECT 1 
    FROM {{ ref('WORK', 'LINEITEM') }} "LINEITEM"
    WHERE "LINEITEM"."L_ORDERKEY" = "ORDERS"."O_ORDERKEY"
)
```

### Using LEFT JOIN with IS NULL

Use a LEFT JOIN in the Join tab to find non-matching rows:

```sql
FROM {{ ref('WORK', 'ORDERS') }} "ORDERS"
LEFT JOIN {{ ref('WORK', 'LINEITEM') }} "LINEITEM"
    ON "ORDERS"."O_ORDERKEY" = "LINEITEM"."L_ORDERKEY"
WHERE "LINEITEM"."L_ORDERKEY" IS NULL
```

## Choosing the Right Method

**Use NOT IN when you have:**

* A small, static list of values.
* Confidence that no NULL values exist in the comparison.

**Use NOT EXISTS when you have:**

* Subqueries that might return NULL values.
* Large data sets requiring optimization.

**Use LEFT JOIN with IS NULL when you need:**

* Performance optimization on very large tables.
* Explicit visibility into the anti-join logic.

---

## What's Next?

* [Join tab][]
* [Helper Tokens][]

---

[Join tab]: /docs/build-your-pipeline/the-build-interface/join-tab
[Helper Tokens]: /docs/reference/helper-tokens

---

## RANK

`RANK()` assigns an integer rank to rows within a group, based on the specified ordering.

## Syntax and Parameters

`RANK() OVER (PARTITION BY partition_column ORDER BY order_column [ASC | DESC])`

* **PARTITION BY** - (optional): Defines the group of rows to rank independently.
* **ORDER BY** - (required): Defines the sort order used to assign rank values.
* **Result** - The rank is an integer. Tied values share the same rank, and subsequent ranks are skipped.
  * `RANK()` always returns an integer (NUMBER).

## Basic Usage

To rank orders for each customer by order date:

```sql
RANK() OVER (
  PARTITION BY "ORDERS"."O_CUSTKEY"
  ORDER BY "ORDERS"."O_ORDERDATE" DESC
)
```

If two orders share the same date, they'll share the same rank, and the next rank is skipped.

```txt
O_CUSTKEY  O_ORDERDATE   RANK
---------  ------------  ----
123        2025-05-06    1
123        2025-05-06    1
123        2025-05-01    3
```

## Using RANK in Coalesce

You can use [Helper Tokens][] to dynamically reference columns:

```sql
// Template
RANK() OVER (
  PARTITION BY {{SRC_PARTITION}}
  ORDER BY {{SRC_ORDER}} DESC
)

//Example using SRC_NODE
RANK() OVER (
  PARTITION BY "{{SRC_NODE}}"."O_CUSTKEY"
  ORDER BY "{{SRC_NODE}}"."O_ORDERDATE" DESC
)
```

## Dense Rank

When you use the `RANK()` function in SQL, it assigns the same rank to rows that are tied in sort order. After a tie, it skips the next rank to maintain position count.

### How Rank Works

You're ranking four values:

```txt
Values:  100   100   90   80
```

Using:

```sql
RANK() OVER (ORDER BY value DESC)
```

You get:

```txt
Ranks:     1     1     3     4
```

* The two `100`s are tied, so both get rank **1**.
* Because rank **2** is “used up” by the tie, the next rank jumps to **3**.
* Then continues normally.

This is standard `RANK()` behavior and is called a **non-dense** ranking.

### How Dense Rank Works

If you don’t want to skip ranks when there are ties, use `DENSE_RANK()` instead:

```sql
DENSE_RANK() OVER (ORDER BY value DESC)
```

```txt
Ranks:     1     1     2     3
```

* Still ranks the tied `100`s as **1**
* But the next distinct value (`90`) gets **2**, not **3**

[Helper Tokens]: /docs/reference/helper-tokens

---

## REPLACE

`REPLACE(string, search_string, replacement_string)` searches for a specific string pattern and replaces it with another string.

## Syntax and Parameters

`REPLACE(string, search_string, replacement_string)`

* **String** - Required. The original string containing the text you want to modify
* **Search String** - Required. The pattern you want to find and replace
* **Replacement string** - Required. The string that will replace the search string
  * Can be an empty string ('') to remove the search string entirely
  * Case-sensitive by default

## Basic Usage

Using `'Supplier#000366631'` as an example:

```sql
REPLACE('Supplier#000366631', 'Supplier#', '') -- "000366631"
REPLACE('Supplier#000366631', '#', '-')  -- "Supplier-000366631"
REPLACE('Supplier#000366631', '000', 'XXX')  -- "Supplier#XXX366631"
```

## Common Use Cases

**Removing prefixes:**

Clean identifier fields by removing standard prefixes.

```sql
-- Basic prefix removal
REPLACE({{SRC}}, 'Supplier#', '') -- 'Supplier#000366631' to '000366631'
REPLACE({{SRC}}, 'CUST_', '') -- 'CUST_12345' to '12345'
REPLACE({{SRC}}, 'ORDER#', '') -- 'ORDER#98765' to '98765'

-- Multiple prefix possibilities
REPLACE(REPLACE({{SRC}}, 'Supplier#', ''), 'SUPP_', '')
```

**Standardizing delimiters:**

Phone numbers, dates, and other formatted fields often need delimiter standardization.

```sql
-- Phone number formatting
REPLACE({{SRC}}, '.', '-') -- '123.456.7890' to '123-456-7890'
REPLACE({{SRC}}, ' ', '-') -- '123 456 7890' to '123-456-7890'

-- Multiple delimiter cleanup
REPLACE(
    REPLACE(
        REPLACE({{SRC}}, '.', '-'), 
        ' ', '-'
    ), 
    '/', '-'
)   

-- 123.456.7890 to 123-456-7890
-- 123 456 7890 to 123-456-7890
-- 123/456/7890 to 123-456-7890

-- Date formatting
REPLACE({{SRC}}, '/', '-') -- '12/31/2023' to '12-31-2023'
```

**Cleaning data:**

Text fields can contain unwanted characters or formatting.

<Tabs groupId="replace-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REPLACE(
    REPLACE({{SRC}}, CHAR(13), ' '), 
    CHAR(10), ' '
)
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
REPLACE(
    REPLACE({{SRC}}, CHR(13), ' '), 
    CHR(10), ' '
)
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REPLACE(
    REPLACE({{SRC}}, CHR(13), ' '), 
    CHR(10), ' '
)
```

  </TabItem>
</Tabs>

```sql
REPLACE({{SRC}}, '  ', ' ')
```

## Using REPLACE in Coalesce

When using REPLACE in transforms and bulk editing, you can use [Helper Tokens][]:

<Tabs groupId="replace-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REPLACE({{SRC}}, 'Supplier#', '')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
REPLACE({{SRC}}, 'Supplier#', '')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REPLACE({{SRC}}, 'Supplier#', '')
```

  </TabItem>
</Tabs>

Expanded examples:

<Tabs groupId="replace-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REPLACE("SUPPLIER"."S_NAME", 'Supplier#', '')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
REPLACE(`supplier`.`s_name`, 'Supplier#', '')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REPLACE(`supplier`.`s_name`, 'Supplier#', '')
```

  </TabItem>
</Tabs>

[Helper Tokens]: /docs/reference/helper-tokens

---

## SUBSTR

`SUBSTR(string, position length)` extracts a substring from a string.

## Syntax and Parameters

`SUBSTR(string, position, length)`

* **String** - (required): The original string you want to extract from.
* **Position** - (required).
  * Positive number. For example 1, 2, 3. Starts from left side
  * Negative number. For example  -1, -2, -3. Starts from right side.
  * Cannot be 0.
  * Position counting starts at 1, not 0.
* **Length** - (optional):
  * If specified: Returns that many characters.
  * If omitted: Returns all remaining characters from the position.

## Using Positive Positions

Positive positions count from the left side.

"TRANSFORM" as an example:

```sql
SUBSTR("TRANSFORM", 1) -- Returns "TRANSFORM" (full string)
SUBSTR("TRANSFORM", 1, 4) -- Returns "TRAN" (first 4 letters)
SUBSTR("TRANSFORM", 2) -- Returns "RANSFORM" (everything after T)
```

When no length is specified, SUBSTR continues to the end of the string. For example,`SUBSTR("TRANSFORM", 1)`:

```txt
T R A N S F O R M
1 2 3 4 5 6 7 8 9  (positions)
↑
Starts here and continues to end
```

## Using Negative Positions

Negative positions count from the right side.

"TRANSFORM" as an example:

```sql
SUBSTR("TRANSFORM", -1) -- Returns "M" (last letter)
SUBSTR("TRANSFORM", -2) -- Returns "RM" (last 2 letters)
SUBSTR("TRANSFORM", -4, 2) -- Returns "FO" (2 letters starting at F)
```

Position counting for "TRANSFORM" from right:

```txt
Position from right:  M(-1)
                     R(-2)
                     O(-3)
                     F(-4)
                     S(-5)
                     N(-6)
                     A(-7)
                     R(-8)
                     T(-9)
```

## Using SUBSTR in Coalesce

When using SUBSTR in transforms and bulk editing, you can use [Helper Tokens][]:

```sql
SUBSTR({{SRC}}, 4)
```

The `{{SRC}}` token automatically gets replaced with the fully qualified source Node and column name. For example:

* If source column is `"CUSTOMER"."FULL_NAME"` it becomes `SUBSTR("CUSTOMER"."FULL_NAME", 4)`.
* If source column is `"ORDERS"."ORDER_ID"` it becomes `SUBSTR("ORDERS"."ORDER_ID", 4)`.

[Helper Tokens]: /docs/reference/helper-tokens

---

## TO_DATE

`TO_DATE(string_expr, format)` converts a string expression into a date.

## Syntax and Parameters

`TO_DATE(string_expr, format)`

* **`string_expr`** - Required. The string or expression you want to convert to a date.
* **`format`** - Optional.
  * Specifies the expected format of the input string.
  * If omitted:
    * On Snowflake: Attempts to auto-detect using the `DATE_INPUT_FORMAT` session parameter.
    * On Databricks: Defaults to Spark's `CAST(expr AS DATE)` behavior, which may return `NULL` for invalid formats.
    * On BigQuery: Uses `PARSE_DATE` or `DATE` casting depending on input type. Specify a format string when the source is not ISO `YYYY-MM-DD`.

## Common Format Elements

These format strings can vary slightly by data platform.

| Format | Description                | Example       |
|--------|----------------------------|---------------|
| `yyyy` | Four-digit year            | `2025`        |
| `MM`   | Two-digit month (01-12)    | `05`          |
| `dd`   | Two-digit day of month     | `06`          |
| `HH`   | Hour in 24-hour format     | `14`          |
| `mm`   | Minutes                    | `30`          |
| `ss`   | Seconds                    | `59`          |

## Basic Examples

<Tabs groupId="to-date-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
TO_DATE('2025-05-06', 'yyyy-MM-dd') -- Returns: 2025-05-06
TO_DATE('06/05/2025', 'dd/MM/yyyy') -- Returns: 2025-05-06
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
TO_DATE('2025-05-06', 'yyyy-MM-dd') -- Returns: 2025-05-06
TO_DATE('06/05/2025', 'dd/MM/yyyy') -- Returns: 2025-05-06
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
PARSE_DATE('%Y-%m-%d', '2025-05-06') -- Returns: 2025-05-06
PARSE_DATE('%d/%m/%Y', '06/05/2025') -- Returns: 2025-05-06
```

  </TabItem>
</Tabs>

## Default Behavior

When no format is specified:

<Tabs groupId="to-date-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
TO_DATE('2025-05-06')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
TO_DATE('2025-05-06')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
DATE('2025-05-06')
```

  </TabItem>
</Tabs>

## Handling Invalid Dates

When parsing date strings, it’s common to encounter values that don’t match the expected format, such as typos, unexpected separators, or completely invalid input like `'April 35'`. How these errors are handled differs by data platform.

### Snowflake

In Snowflake, if the input string does not match the expected format, the standard `TO_DATE()` function throws an error:

```sql
TO_DATE('invalid', 'yyyy-MM-dd')  -- Error: invalid input syntax for type date
```

To prevent your pipeline from breaking due to such errors, Snowflake provides a safe variant:

```sql
TRY_TO_DATE('invalid', 'yyyy-MM-dd')  -- Returns: NULL
```

`TRY_TO_DATE()` works just like `TO_DATE()`, but instead of throwing an error, it returns `NULL` if the input can't be parsed. It is available on Snowflake only.

This is ideal in production transformations where you want to allow processing to continue even with malformed date strings.

### BigQuery

Use `PARSE_DATE` with an explicit format string:

```sql
PARSE_DATE('%Y-%m-%d', '2025-05-06')
```

For safe parsing that returns `NULL` on failure instead of raising an error:

```sql
SAFE.PARSE_DATE('%Y-%m-%d', 'invalid')  -- Returns: NULL
```

BigQuery format elements use `%Y`, `%m`, and `%d` rather than Snowflake's `yyyy` and `MM` tokens. Test format strings against sample values in your warehouse.

### Databricks

Databricks throws an error if the date format or value is invalid and stops the query. Filter out invalid data before running `TO_DATE()`.

| Scenario                         | ANSI Mode ON                      | ANSI Mode OFF               |
| -------------------------------- | --------------------------------- | --------------------------- |
| Invalid date format or value     | Throws error and stops the query  | Returns `NULL` silently     |
| Well-formed date string          | Converts to date                  | Converts to date            |
| Missing value or `'null'` string | Must be handled manually          | Must be handled manually    |

:::info[ANSI Mode]

If you have ANSI mode off, then you can write a case statement to handle errors. Be sure to research and understand what changing ANSI mode means for your organization.

If it's off, you can write a case statement to handle incorrect values.

<Tabs groupId="to-date-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
TRY_TO_DATE({{SRC}}, 'yyyy-MM-dd')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CASE
  WHEN {{SRC}} IS NULL OR LOWER({{SRC}}) = 'null' THEN NULL
  ELSE TO_DATE({{SRC}}, 'yyyy-MM-dd')
END
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CASE
  WHEN {{SRC}} IS NULL OR LOWER(CAST({{SRC}} AS STRING)) = 'null' THEN NULL
  ELSE PARSE_DATE('%Y-%m-%d', CAST({{SRC}} AS STRING))
END
```

  </TabItem>
</Tabs>

* [ANSI compliance in Databricks Runtime][]
* [ANSI mode in Databricks SQL][]

:::

## Using TO_DATE in Coalesce

When using date parsing in transforms or bulk editing, you can use [Helper Tokens][]:

<Tabs groupId="to-date-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
TO_DATE({{SRC}}, 'yyyy-MM-dd')
```

Expanded:

```sql
TO_DATE("CUSTOMER"."SIGNUP_DATE", 'yyyy-MM-dd')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
TO_DATE({{SRC}}, 'yyyy-MM-dd')
```

Expanded:

```sql
TO_DATE(`customer`.`signup_date`, 'yyyy-MM-dd')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
PARSE_DATE('%Y-%m-%d', {{SRC}})
```

Expanded:

```sql
PARSE_DATE('%Y-%m-%d', `customer`.`signup_date`)
```

  </TabItem>
</Tabs>

If the column is already a `DATE` or `TIMESTAMP` type, remove the format argument on Snowflake and Databricks: `TO_DATE({{SRC}})`. On BigQuery, cast directly: `DATE({{SRC}})`.

## What's Next?

* [Snowflake conversion functions][]
* [Databricks `to_date` function][]
* [BigQuery `PARSE_DATE` function][]

[Helper Tokens]: /docs/reference/helper-tokens
[Snowflake conversion functions]:https://docs.snowflake.com/en/sql-reference/functions-conversion
[Databricks `to_date` function]:https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
[BigQuery `PARSE_DATE` function]: https://cloud.google.com/bigquery/docs/reference/standard-sql/date_functions#parse_date
[ANSI compliance in Databricks Runtime]: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-ansi-compliance
[ANSI mode in Databricks SQL]: https://docs.databricks.com/aws/en/sql/language-manual/parameters/ansi_mode

---

## TRY_TO_DOUBLE

Safely cast string values to double precision. If the conversion fails, the expression returns `NULL` instead of raising an error.

## Basic Usage

<Tabs groupId="try-to-double-platform">
  <TabItem value="snowflake" label="Snowflake" default>

`TRY_TO_DOUBLE(expr [, format])` is available on Snowflake only.

```sql
SELECT
  TRY_TO_DOUBLE('40.7128') AS lat,
  TRY_TO_DOUBLE('-74.0060') AS lon;
-- Returns: 40.7128, -74.0060

SELECT
  TRY_TO_DOUBLE('N/A') AS lat,
  TRY_TO_DOUBLE('180W') AS lon;
-- Returns: NULL, NULL
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
SELECT
  TRY_CAST('40.7128' AS DOUBLE) AS lat,
  TRY_CAST('N/A' AS DOUBLE) AS invalid_lat;
-- Returns: 40.7128, NULL
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
SELECT
  SAFE_CAST('40.7128' AS FLOAT64) AS lat,
  SAFE_CAST('N/A' AS FLOAT64) AS invalid_lat;
-- Returns: 40.7128, NULL
```

  </TabItem>
</Tabs>

## JSON Coordinates

When coordinates are stored as strings in JSON:

<Tabs groupId="try-to-double-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
SELECT
  TRY_TO_DOUBLE(variant_col:"lat"::string) AS latitude,
  TRY_TO_DOUBLE(variant_col:"lon"::string) AS longitude
FROM my_table;
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

Extract the JSON field as a string, then cast:

```sql
SELECT
  TRY_CAST(get_json_object(json_col, '$.lat') AS DOUBLE) AS latitude,
  TRY_CAST(get_json_object(json_col, '$.lon') AS DOUBLE) AS longitude
FROM my_table;
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
SELECT
  SAFE_CAST(JSON_VALUE(json_col, '$.lat') AS FLOAT64) AS latitude,
  SAFE_CAST(JSON_VALUE(json_col, '$.lon') AS FLOAT64) AS longitude
FROM my_table;
```

  </TabItem>
</Tabs>

## In Coalesce Transforms

<Tabs groupId="try-to-double-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
TRY_TO_DOUBLE({{SRC}})
```

Expanded:

```sql
TRY_TO_DOUBLE("ORDERS"."LATITUDE")
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
TRY_CAST({{SRC}} AS DOUBLE)
```

Expanded:

```sql
TRY_CAST(`orders`.`latitude` AS DOUBLE)
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
SAFE_CAST({{SRC}} AS FLOAT64)
```

Expanded:

```sql
SAFE_CAST(`orders`.`latitude` AS FLOAT64)
```

  </TabItem>
</Tabs>

## Snowflake Syntax Reference

`TRY_TO_DOUBLE(expr [, format])`

- `expr`: Required. A string VARCHAR or CHAR expression to convert.
- `format`: Optional. A format model for strings with consistent formatting such as commas. See [SQL format models][].

## Best Practices

- **Input type**: Only cast from strings when the column is character data. Numeric columns do not need a safe cast function.
- **VARIANT / JSON on Snowflake**: Convert JSON fields to string first (`::string`) before applying `TRY_TO_DOUBLE`.
- **Error rates**: If many conversions fail, performance may degrade. These functions work best when errors are rare.
- **Format usage (Snowflake)**: Use the optional format model only when input strings have consistent formatting, for example, 1,234.56.
- **Precision caveat**: Conversion from string to floating point may involve rounding typical in floating-point representation.

## What's Next?

- [Snowflake TRY_TO_DOUBLE][]
- [Helper Tokens][]
- [Databricks TRY_CAST][]
- [BigQuery SAFE_CAST][]

[SQL format models]: https://docs.snowflake.com/en/sql-reference/sql-format-models
[Snowflake TRY_TO_DOUBLE]: https://docs.snowflake.com/en/sql-reference/functions/try_to_double
[Helper Tokens]: /docs/reference/helper-tokens
[Databricks TRY_CAST]: https://docs.databricks.com/aws/en/sql/language-manual/functions/try_cast
[BigQuery SAFE_CAST]: https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#safe_casting

---

## TRIM Functions

The TRIM functions remove whitespace or specified characters from strings.

- `TRIM` removes characters from both ends.
- `LTRIM` removes characters from the left (start).
- `RTRIM` removes characters from the right (end).

## Syntax and Parameters

```sql
TRIM(string , characters)
LTRIM(string , characters)
RTRIM(string , characters)
```

- **String**: The input string to trim.
- **Characters**: Optional characters to remove. If omitted, removes whitespace.

## Basic Usage

Using `'  data science  '` and `'/path/to/file/'` as examples:

```sql
-- Trimming whitespace
TRIM('  data science  ') -- Returns "data science"
LTRIM('  data science  ') -- Returns "data science  "
RTRIM('  data science  ') -- Returns "  data science"

-- Trimming specific characters
TRIM('/path/to/file/', '/') -- Returns "path/to/file"
LTRIM('/path/to/file/', '/') -- Returns "path/to/file/"
RTRIM('/path/to/file/', '/') -- Returns "/path/to/file"
```

## Common Use Cases

**Cleaning input data:**

Remove unwanted padding in imported data.

```sql
-- Basic whitespace cleanup
TRIM({{SRC}}) -- "  customer name  " → "customer name"

-- Removing specific characters
LTRIM({{SRC}}, '0') -- "000123" → "123"
RTRIM({{SRC}}, '.') -- "price.00" → "price"

-- Combined trimming for both ends
TRIM({{SRC}}, '/') -- "/path/to/file/" → "path/to/file"
```

**Standardizing identifiers:**

Clean up identifier fields that may have padding.

```sql
-- Remove leading zeros
LTRIM({{SRC}}, '0') -- "000366631" → "366631"

-- Remove trailing spaces from fixed-width fields
RTRIM({{SRC}}) -- "CUSTOMER_ID    " → "CUSTOMER_ID"

-- Remove both leading zeros and trailing spaces
TRIM(LTRIM({{SRC}}, '0')) -- "00012345    " → "12345"
```

## Using TRIM Functions in Coalesce

When using TRIM functions in transforms and bulk editing, you can use [Helper Tokens][]:

```sql
TRIM({{SRC}})
```

The `{{SRC}}` token automatically gets replaced with the fully qualified source Node and column name. For example:

- If source column is `"CUSTOMER"."C_NAME"` it becomes `TRIM("CUSTOMER"."C_NAME")`
- If source column is `"ORDERS"."ORDER_ID"` it becomes `TRIM("ORDERS"."ORDER_ID")`

[Helper Tokens]: /docs/reference/helper-tokens

---

## Run-Time Dates and Times

You can use your warehouse's native date and time functions in [Parameters][] and [Column Transforms][]. Coalesce sends that SQL to the platform when the Job runs, so the database evaluates `CURRENT_DATE()`, `CURRENT_TIMESTAMP()`, offsets, and month-end functions at run time instead of freezing a literal in the editor.

:::tip[Match the column data type]

Set the column **Data Type** to **DATE** or **TIMESTAMP** to match what your expression returns. Mixing types can cause compile errors or implicit casts you did not intend.

:::

## Use Run-Time Dates in Parameters

Define a Parameter value with warehouse SQL. When a Job uses that Parameter, the warehouse resolves the expression for that run.

<Tabs>
  <TabItem value="snowflake" label="Snowflake" default>

    Common choices:

    - `CURRENT_DATE()`
    - `CURRENT_TIMESTAMP()`

    Example: a Parameter set to `CURRENT_DATE()` returns the calendar date for each Job run.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

    In Standard SQL, common choices are:

    - `CURRENT_DATE()`
    - `CURRENT_TIMESTAMP()`

    Example: a Parameter set to `CURRENT_DATE()` returns the calendar date for each Job run.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    Common choices:

    - `current_date()`
    - `current_timestamp()`

    Example: a Parameter set to `current_date()` returns the calendar date for each Job run.

  </TabItem>
</Tabs>

## Use Run-Time Timestamps in Column Transforms

Use the same functions in the **Transform** field or **Column Editor** when you want audit-style fields that refresh every time the Node runs.

<Tabs>
  <TabItem value="snowflake" label="Snowflake" default>

    ```sql
    CURRENT_TIMESTAMP()
    ```

    Cast to `TIMESTAMP_NTZ`, `TIMESTAMP_LTZ`, or `TIMESTAMP_TZ` when you need a specific timestamp variant for governance or downstream tools.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

    ```sql
    CURRENT_TIMESTAMP()
    ```

    Cast to `DATE` if you only need the calendar day.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    ```sql
    cast(current_timestamp() as timestamp)
    ```

    Adjust the cast target if your table expects a specific timestamp type.

  </TabItem>
</Tabs>

These patterns are typical for `CREATED_AT` or `UPDATED_AT` style columns.

## Build Incremental-Style Windows

For run-relative windows such as **yesterday** or T-1, use each engine's date math in filters, Parameters, or transforms.

<Tabs>
  <TabItem value="snowflake" label="Snowflake" default>

    ```sql
    CURRENT_DATE() - 1
    ```

    You can use similar arithmetic for other offsets, for example `CURRENT_DATE() - 7` for a 7-day lookback.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

    ```sql
    DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)
    ```

    Use `INTERVAL` with the unit you need for other offsets.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    ```sql
    date_sub(current_date(), 1)
    ```

    For forward offsets, use `date_add(current_date(), 7)` or the equivalent for your calendar logic.

  </TabItem>
</Tabs>

## Month End From a Source Date

To derive the last calendar day of the month from a row's date column, wrap the source expression your Node already exposes, such as `{{SRC}}` from [Helper Tokens][].

<Tabs>
  <TabItem value="snowflake" label="Snowflake" default>

    ```sql
    LAST_DAY(CAST({{SRC}} AS DATE))
    ```

    `LAST_DAY` accepts an optional second argument for other periods; the default date part is **MONTH**, which returns month end.

    If the source value is `NULL`, `LAST_DAY` returns `NULL`.

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

    ```sql
    LAST_DAY(DATE({{SRC}}))
    ```

    You can pass a second `date_part` when you need a boundary other than month end.

    If the source value is `NULL`, `LAST_DAY` returns `NULL`.

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    ```sql
    last_day(to_date({{SRC}}))
    ```

    `last_day` returns the last day of the month for the given date.

    If the source value is `NULL`, `last_day` returns `NULL`.

  </TabItem>
</Tabs>

## Month End From Today

For logic tied to the Job run date, compute month end from the warehouse's current date in the same dialect.

<Tabs>
  <TabItem value="snowflake" label="Snowflake" default>

    ```sql
    LAST_DAY(CURRENT_DATE())
    ```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

    ```sql
    LAST_DAY(CURRENT_DATE())
    ```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

    ```sql
    last_day(current_date())
    ```

  </TabItem>
</Tabs>

## Calendar Month End and Fiscal Periods

`LAST_DAY` and `last_day` return **calendar** month end. Fiscal period end depends on your organization's calendar. If fiscal months do not line up with calendar months, model those boundaries in a calendar table or mapping and join to it instead of relying on `LAST_DAY` alone.

## Other Common Uses

Run-time dates and times also support:

- **Incremental loads** - Filter or watermark with relative dates so each run uses the correct window without editing literals.
- **Partition keys** - Populate date or timestamp partition columns from `CURRENT_DATE()` or `current_date()` so new data lands in the right slice.
- **Run metadata** - Stamp processing time with `CURRENT_TIMESTAMP()` or `current_timestamp()` in logging or audit tables.

## Limitations

Some testing utilities only accept fixed date strings and reject run-time date functions. For example, `testUtils.test_missing_dates` expects a static value rather than `CURRENT_DATE()`.

---

## What's Next?

- [Column Transforms][] for transform fields, data types, and platform quoting.
- [Helper Tokens][] for `{{SRC}}` and related tokens in transforms and Macros.
- [Macros][] to reuse date logic across Nodes.

---

[Column Transforms]: /docs/build-your-pipeline/transforms
[Helper Tokens]: /docs/reference/helper-tokens
[Macros]: /docs/reference/macros
[Parameters]: /docs/deploy-and-refresh/parameters

---

## Using Math in Transformations

Mathematical operations in SQL allow you to perform calculations on numeric columns. This guide covers the four basic arithmetic operations: addition (+), subtraction (-), multiplication (*), and division (/).

## Data Type Rules

When performing math operations between different numeric types:

* FLOAT * INTEGER = FLOAT
* DECIMAL * INTEGER = DECIMAL
* FLOAT * DECIMAL = FLOAT
* INTEGER / INTEGER = DECIMAL
* All operations return NULL if either input is NULL

## Addition (+)

```sql
-- Simple addition showing full reference vs column name
{{SRC}} + 100  -- Full reference
{{SRC_COL}} + 100  -- Just column name

-- Using source node name in a comment
-- Adding tax to {{SRC_NODE}} amount
{{SRC}} + ({{SRC_NODE}}."TAX_RATE" * {{SRC}})

-- Reference current node
-- Common uses
{{SRC}} + "ADJUSTMENTS"."AMOUNT"  -- Add adjustments
{{SRC}} + ("BASE"."AMOUNT" * 0.08)  -- Add 8% tax
```

## Subtraction (-)

```sql
-- Basic subtraction
{{TGT}} - 50

-- Subtracting columns
{{SRC}} - "ORDERS"."DISCOUNT_AMOUNT"

-- Calculate differences
{{SRC}} - "PREVIOUS"."BALANCE"  -- Change in balance
{{SRC}} - "COSTS"."AMOUNT"  -- Profit calculation

-- Common uses
{{SRC}} - ("ORDERS"."AMOUNT" * "DISCOUNTS"."RATE")  -- Apply discount
{{SRC}} - "RETURNS"."AMOUNT"  -- Subtract returns
```

## Multiplication (*)

```sql
-- Basic multiplication
{{SRC}} * 2

-- Multiplying columns
{{SRC}} * "ORDERS"."QUANTITY"

-- Common uses
{{SRC}} * -1  -- Negate amounts
{{SRC}} * 0.453592  -- Convert pounds to kilograms
{{SRC}} * (1 + "PRICING"."MARKUP_PCT")  -- Apply markup
```

## Division (/)

```sql
-- Basic division
{{SRC}} / 100  -- Convert to hundreds
{{SRC}} / 12  -- Monthly average

-- Dividing columns
{{SRC}} / "SALES"."TOTAL_UNITS"  -- Per unit calculation

-- Common uses
{{SRC}} / NULLIF("TOTALS"."COUNT", 0)  -- Safe division with NULLIF
{{SRC}} / 1000000  -- Convert to millions
```

## Controlling Precision

When using calculations, you might need to control the number of decimal places in the results. For example, you might want exactly two decimal places for currency or four decimal places for percentages. SQL provides two main functions to help: `ROUND()` for simple decimal control, and `CAST()` for more precise data type formatting.

### ROUND

`ROUND()` lets you specify how many decimal places you want to keep in a number.

```sql
ROUND(number, decimal_places)
```

`number`: The number you want to round.
`decimal_places`: How many decimal places to keep.

```sql
-- Round to 2 decimal places
ROUND({{SRC}} * "ORDERS"."QUANTITY", 2)

-- Round to whole numbers
ROUND({{SRC}} / 12, 0)

-- Round percentage calculations
ROUND({{SRC}} / "TOTALS"."AMOUNT", 4)  -- 4 places for percentages
```

### CAST

`CAST()` converts a number into a specific data type and format. This is helpful when you need to:

* Control both the total number of digits and decimal places.
* Ensure consistent data types in your calculations.
* Convert between different number formats.

```sql
CAST(number AS data_type(total_digits, decimal_places))
```

* `number`: The value you want to convert.
* `data_type`: The type of number you want.
* `total_digits`: The total number of digits to allow.
* `decimal_places`: How many digits after the decimal point.

For example, DECIMAL(18,2) means:

* Up to 18 total digits
* 2 decimal places

```sql
-- Cast as decimal with 2 decimal places
CAST({{SRC}} + O_TOTALPRICE AS DECIMAL(18,2))

-- Cast as integer
CAST({{SRC}} / 100 AS INTEGER)

-- Cast with specific precision
CAST({{SRC}} * "RATES"."MULTIPLIER" AS DECIMAL(38,6))
```

## Using Math in Coalesce

When using math operations in transforms and bulk editing, you can use [Helper Tokens][]:

```sql
{{SRC}} * 1.08  -- Add 8% to source value
{{TGT}} / 100   -- Convert target to percentage
```

Examples using Helper Tokens:

```sql
-- Basic calculations
{{SRC}} + 100  -- Add flat amount
{{SRC}} * 2    -- Double the value

-- Combining with other columns
{{SRC}} * "TAX_RATE"
```

## Best Practices

* Use parentheses to make order of operations clear.
* Handle division by zero with `NULLIF()`.
* Consider precision needs when choosing between `ROUND()` and `CAST()`.
* Use appropriate scale for financial calculations, such as `DECIMAL(38,6)`.
* Test with edge cases (very large numbers, very small numbers, negative numbers).

[Helper Tokens]: /docs/reference/helper-tokens

---

## Cron Reference

Cron expressions are a way to schedule tasks automatically. You can run scripts, commands, or software at specific times. Coalesce uses Cron for Scheduled Jobs.

## Structure of a Cron Expression

A Cron expression is a string made of five fields separated by a space. The fields must be written in the following order. Fields 1- 6 are required.

1. **Seconds** (`0-59`): Specifies the exact second when the task should start.  
   1. **Example:** `0` indicates the task runs at the beginning of the minute.  
2. **Minutes** (`0-59`): The minute in hour when the task should run.  
   1. **Example:** `15` means the task runs at the 15th minute of the hour.  
3. **Hours** (`0-23`): The hour of the day when the task should run. Based on 24 hour time.  
   1. **Example:** `14` indicates the task runs at 2:00 PM.  
4. **Day of Month** (`1-31`): The day of the month to run.  
   1. **Example:** `1` means the task runs on the 1 st day of the month.  
5. **Month** (`1-12`): The month the task should run.  
   1. **Example:** `5` means the task runs in May.  
6. **Day of Week** (`0-6`): This field defines the days of the week when the task should run.  
   1. Example: `0` means the task runs on Sundays.

### Special Characters

Special characters are used to build your expression.

- **Asterisk** (`*`):  Represents all possible values for a field.  
  - Example: `*` in the month field means every month.  
- **Hyphen** (`-`) : Specifies a range of values.  
- Example: `10-12` in the hours field means the task runs at 10:00 AM, 11:00 AM, and 12:00 PM.  
- **Comma** (`,`) : Specifies a list of values.  
- Example: `0,6,5` in the day of the week field means it runs every Sunday, Saturday, and Friday.  
- **Slash (`/`)**: Specifies intervals.  
- Example: `/15` in the minutes field means the task runs every 15 minutes.

## Examples of Cron Expressions

### Run Daily at Midnight

```text
0 0 * * *
```

- `0 0` - The task runs at 0 minutes and 0 hours (midnight).  
- `*` - The task runs every day of the month.  
- `*` - The task runs every month.  
- `*`- The task runs every day of the week.

### Run Every 15 Minutes

```text
*/15 * * * *
```

- `*/15 - The task run every 15 minutes`  
- `*` - The task runs every hour.  
- `*` - The task runs every day of the month.  
- `*` - The task runs every month.  
- `*` - The task runs every day of the week.

### Run at 09:00 Every Day of the Month and on Monday

```text
0 9 1-7 * 1
```

- `0 9` - The task runs at 9:00 AM.  
- `1-7` - First seven days of the month  
- `*` - The task runs every month.  
- `1` - The task also runs on Monday.

### Run at 22:00 on Monday and Friday

```text
0 22 * * 1,5
```

- `0` - Any second  
- `22` - Run at 22:00  
- `*` -  Any day  
- `*` - Any month  
- `1,5` - Monday and Friday

## Troubleshooting Cron Jobs

- Question Marks (`?`) are currently not accepted in the Cron Expressions in Coalesce.  
- `SUN-SAT` and `JAN-DEC`  alternative values are not accepted.  
- Make sure your fields are in the correct order. Check using [Crontab](https://crontab.guru/) to make sure.  
- Avoid specifying days that don’t exist, for example February 30th.

## Resources

- [Crontab](https://crontab.guru/) - Generate expression

---

## HTTP 413 Request Entity Too Large During Git Commit

You may see **HTTP Error: 413** when Coalesce pushes version control changes to your repository. The status means the push request exceeded a size limit on your Git host or on a reverse proxy in front of it. This page explains when the error appears, how to commit smaller changes from Coalesce, and when your Git administrator needs to adjust infrastructure limits.

## Description

When you commit from the Coalesce UI, the app sends version control operations through a secure HTTPS proxy to your Git provider. Coalesce supports GitHub, GitLab, Bitbucket, and Azure DevOps. The provider, or a corporate reverse proxy on the path to it, accepts only HTTP requests up to a configured maximum body size. If your commit packs too much metadata into one push, the server responds with **413** and the commit does not complete.

You may see an error similar to:

```txt
Failed to Push to remote: Error: [HttpError] HTTP Error: 413
```

The same HTTP status can appear when Coalesce creates or updates a remote branch, not only when you click **Commit and Push** in the Git modal.

## Common Symptoms

You may notice one or more of these patterns:

- **Commit and Push** fails after you stage many files at once.
- A lighter commit succeeds, but a heavier commit from the same Workspace fails.
- **Duplicate Workspace** fails when you choose a commit that contains a large metadata snapshot. An earlier commit works.
- The failure happens consistently for one repository or Git host, especially after bulk Node edits, **Packages** changes, or environment mapping updates.

## Possible Causes

- You staged **too many files in one commit**. The Git modal selects all changed files by default.
- The commit includes **large metadata**, such as many Nodes, **Packages** configuration, Jobs, macros, or environment mappings. See [What Gets Committed][] for what Coalesce writes to Git.
- You combined several large changes in one push, such as a **Marketplace package** install plus widespread Node edits.
- Your **Git platform or reverse proxy** enforces a low HTTP body limit. This is common with **Azure DevOps** deployments and self-hosted Git behind a corporate reverse proxy or load balancer.

## Possible Solutions

### Reduce What You Commit From Coalesce

Use this workflow when the error appears during **Commit and Push**:

1. Open the **Git** modal and review the file list under **Changes**.
2. **Unstage** files you do not need in this commit. Leave only one logical unit of work selected.
3. Click **Commit and Push**, then repeat for remaining changes without closing the modal if more files are still listed.
4. Follow [Version Control Best Practices][]: frequent, small commits scoped to a single task.
5. Avoid combining package installs, mass Node edits, and environment changes in one push when possible.

For step-by-step staging behavior, see [Git Commits][].

:::tip[Stage fewer files by default]

The Git modal selects every changed file when you open it. Unstage files before your first **Commit and Push** when you changed many Nodes or **Packages** in one session.

:::

### Duplicate Workspace or Branch Operations

If duplicating a Workspace fails on one commit but succeeds from an earlier one, the starting commit may be too large for your Git host to accept in one push. Choose a **lighter commit** as the source, or split pending work into smaller commits before you duplicate. See [When To Use Copy Objects, Duplicate, or Create New Project][].

### Work With Your Git Administrator

If smaller commits still fail, ask your Git or platform team to review **HTTP request body limits** for:

- **Azure DevOps**, including organization or proxy settings for large HTTPS pushes
- **Self-hosted Git** behind a reverse proxy or corporate load balancer

Administrators typically raise proxy body-size limits on **each layer** between clients and the Git server. Coalesce Support cannot change settings on your Git infrastructure.

Version control from the Coalesce App uses **HTTPS** to your Git provider. Resolving HTTP 413 is about commit size and Git infrastructure limits.

## Contact Coalesce Support

If you still cannot commit after you reduce commit size and your Git team confirms infrastructure limits, contact [Coalesce Support][] with:

- The full error message, including **HTTP Error: 413**
- Your Git provider: GitHub, GitLab, Bitbucket, or Azure DevOps
- What you were doing: **Commit and Push**, **Duplicate Workspace**, new branch, or similar
- Rough scope of the change, such as number of Nodes or files staged

## What's Next?

- [Git Commits][]
- [What Gets Committed][]
- [Version Control Best Practices][]
- [Expired Version Control Credentials][]

[Coalesce Support]: mailto:support@coalesce.io
[Expired Version Control Credentials]: /docs/git-integration/solving-git-errors/access-token-expired
[Git Commits]: /docs/git-integration/git-commits
[Version Control Best Practices]: /docs/git-integration/version-control-best-practices
[What Gets Committed]: /docs/git-integration/what-gets-committed
[When To Use Copy Objects, Duplicate, or Create New Project]: /docs/get-started/coalesce-fundamentals/copying-workspace-and-project-data

---

## Blank or Empty Column Names (Plan Validation Failure)

## Error

Deployment fails at plan generation or validation with an error related to blank or empty column names. The failure may occur even if the Node has deployed successfully in the past.

## Possible Causes

- **Legacy Nodes**: The Node was created before Coalesce added build and plan validation. Columns with empty names may have been allowed previously.
- **CLI or version upgrade**: After upgrading the CLI or Coalesce version, stricter validation can surface existing invalid column names.
- **Manual or bulk edits**: Column names were cleared or left blank during manual edits or bulk operations.

## Possible Solutions

**Fix the Node in the Build tab:**

1. Open the affected Node in the **Build** tab.
2. In the column mapping grid, locate any columns with blank or empty names.
3. Give each column a valid, non-empty name.
4. Save your changes, commit, and redeploy.

:::tip[Finding Blank Columns]

If the Problem Scanner or validation doesn't point to a specific column, check each output column in the Node's mapping grid. Look for empty cells in the column name field.

:::

**If you can't edit the Node:**

In rare cases, the Node editor may block changes (for example, due to an invalid source column reference). If that happens:

1. Copy the transform logic from the affected Node.
2. Clear the transform (remove all logic).
3. Remove the invalid or blank column reference.
4. Paste the transform logic back and re-add the correct source column if needed.
5. Give all columns valid names.
6. Commit and redeploy.

## Prevention

Coalesce now validates column names in the Build tab. When creating or editing Nodes, ensure every output column has a non-empty name before committing.

---

## Duplicate Row Detected During Dml Action Row Values

## Error

`“Duplicate row detected during DML action Row Values:[]”`

The error received is actually a Snowflake error. This error results from when there is more than one row that matches the set merge key attempting to update/replace the same record in the target table.  
The merge does not know with what record to replace the destination record with since it is only able to determine the uniqueness of the record on the primary key.  
This can usually be resolved be re-evaluating the set of table columns that make up the business/merge keys to include another unique identifier.

This error is returned from Snowflake. It happens when there is more than one row that matches the set merge key attempting to update or replace the same record in the target table. The merge does not know with what record to replace the destination record with since it is only able to determine the uniqueness of the record on the primary key.

This can usually be resolved be re-evaluating the set of table columns that make up the business or merge keys to include another unique identifier.

Table one contains Ross and Monica using the LAST_NAME as the business or merge key. You will get an error if you try to merge into Table two since there are two records that share the same business key attempting to merge into Table two that contains one record with that same business key.

## Solution

You can do one of two solutions.

1. You can fix this by using the ID column as our business key or selecting a composite business key as `first_name` and `last_name` in the merge.
2. Set a session variable in the pre-sql.
   1. `alter session set ERROR_ON_NONDETERMINISTIC_MERGE = true;`
   2. This solution may result in data loss since it will pick records at random to include or exclude in the merges.

---

## Error: GraphExecutor halted and finished

## Error

If you receive the "GraphExecutor halted and finished; encountered failures for node: \[node ID\]; no nodes skipped" error when running a Nodes, Subgraph, Job, etc. within a Workspace, this is typically due to the objects you are trying to complete a Run operation on in Coalesce not yet being created in the target location in Snowflake.

You may also see a message in the results pane that say `Table <fully qualified table name> does not exist or not authorized.`

## Solution

In your Workspace, first execute the Create operation for the objects you are trying to execute the Run on. Once the Create has completed successfully, you may execute the Run operation. In the Workspace, the Create operation executes the Data Definition Language (DDL) for your objects, while the Run operation executes the Data Manipulation Language (DML) for your objects. These operators are comparable to Deploy (Create) and Refresh (Run) for your Coalesce Environments.

---

## Spaces in METADATA$FILENAME

## Error

When working with external tables, you may see unexpected spaces in `METADATA$FILENAME` values. This can result in incorrect file path references, failed lookups, or broken downstream logic that depends on clean metadata values.

## Possible Causes

- **CSV parsing configuration:** File headers contain extra spaces or incorrect quoting, which affects how metadata columns are populated.
- **File pattern and stage setup:** Incorrect file path selection or column mapping in the external table configuration introduces extra whitespace.
- **External table column mapping:** Spaces appear in `METADATA$FILENAME` due to how the external table parses and maps source files.
- **Metadata extraction formatting:** Extra spaces are generated between database names or during file processing.

## Possible Solutions

- Review your external table configuration and confirm that file patterns match the expected file paths without extra whitespace.
- Check CSV formatting options such as quoting, delimiters, and header row handling. Make sure headers don't contain leading or trailing spaces.
- Validate column mapping in your external table to ensure `METADATA$FILENAME` is correctly populated.
- Use a `TRIM()` function on the `METADATA$FILENAME` column to remove unwanted spaces if the source files can't be corrected.

---

## Error: Field Is Not a Valid GROUP BY Expression

## Error

The error `field is not a valid group by expression` occurs when a SQL query violates GROUP BY rules. This happens when you use aggregate functions like SUM, COUNT, or MAX in your SELECT statement, but non-aggregated columns aren't included in the GROUP BY clause.

## Possible Causes

* **Non-aggregated columns not in GROUP BY:** When you use aggregate functions in your SELECT statement, any column that isn't aggregated must be included in the GROUP BY clause.
* **Missing GROUP BY clause:** If you're using aggregate functions but haven't specified which columns to group by.
* **Column reference issues:** The error can occur due to how columns are referenced or aliased in the query.

## Possible Solutions

* **Add missing columns to GROUP BY:** In the Join tab of your Node, add a GROUP BY clause that includes all non-aggregated columns from your SELECT statement.
* **Use GROUP BY ALL:** Coalesce supports the "GROUP BY ALL" syntax which automatically groups by all non-aggregated columns in your SELECT statement.
* **Apply aggregate functions:** Convert non-aggregated columns to use aggregate functions like MAX(), MIN(), or FIRST_VALUE() in the column transforms.
* **Remove unnecessary columns:** If certain columns don't need to be in the output, remove them from the mapping to avoid GROUP BY conflicts.

---

## Error Codes Dictionary

Your comprehensive reference for all API and Coalesce app errors. Search by error code or browse to find detailed explanations, causes, and step-by-step instructions.

<ErrorTable spreadsheetId="1tne4FWXGSG49qKys68qvQ9YjJ7V3oIgRkCJhvUbuAiQ" />

---

## Invalid Column Metadata (Deployment Failure With No Failed Stages)

## Error

A deployment fails but no individual stages show as failed. This can occur when a Node references a source column that no longer exists, was renamed, or has invalid metadata. The failure may appear at plan generation or during deployment without clear stage-level errors.

## Possible Causes

- **Invalid source column reference:** A downstream Node references a column that was removed or renamed in the source.
- **Metadata drift:** Column metadata in Coalesce is out of sync with the actual source table or view.
- **Stale cache:** Cached metadata from a previous deployment doesn't match the current state.

## Possible Solutions

**Workaround when the transform references a bad source column:**

1. Copy the transform logic from the affected Node.
2. Clear the transform (remove all logic).
3. Remove the invalid source column reference from the Node.
4. Paste the transform logic back.
5. Re-add the correct source column if needed.
6. Commit and redeploy.

If the problem persists, verify that source tables and views exist and that column names and types match what Coalesce expects. Consider running a metadata sync or refreshing source definitions.

---

## Error: Invalid Identifier

## Error

If you receive an `invalid identifier` error when running or creating a Node in Coalesce, this typically means your Node is referencing a column or table that doesn't exist in your data platform. The error message will usually indicate which line in the generated SQL is causing the issue.

## Possible Causes

* **Wrong identifier quoting for your Project platform**: Coalesce generates double-quoted identifiers on Snowflake Projects and backtick-quoted identifiers on Databricks and BigQuery Projects. Hand-written identifiers that use the wrong style, or Snowflake examples copied into a Databricks or BigQuery Project without updating quotes, often cause this error. See [Quoted Identifiers in Coalesce][].
* **Case sensitivity issues**: Mixed-case column names must match the quoting style for your platform. On Snowflake, wrap identifiers in double quotes or use uppercase unquoted names. On Databricks and BigQuery, wrap identifiers in backticks when case matters.
* **Column doesn't exist in source table**: The column you're referencing may have been removed, renamed, or never existed in the upstream source table.
* **Source table changes**: The upstream source table structure may have changed since you last synced columns in Coalesce.
* **Missing or incorrect table aliases**: If you're using aliases in joins or references, make sure they match exactly what you've defined in your SQL.
* **Upstream dependency issues**: The table or Node you're referencing may not have been created successfully in your data platform yet.

## How to Troubleshoot

* **Check the SQL tab**: Look at the generated SQL to see exactly which column or identifier is causing the issue. The error message will provide a line number to help you locate the problem.
* **Match platform quoting**: Confirm hand-written identifiers in Transforms and the Join tab use double quotes on Snowflake and backticks on Databricks or BigQuery. Verify that [helper tokens][] and [ref functions][] expanded with the expected quoting style.
* **Resync columns**: Go to your source Node and click Resync Columns to refresh the column metadata from your data platform. This ensures Coalesce has the latest column information.
* **Verify column names**: Double-check that the column names in your mapping match exactly what exists in the source table, including case sensitivity.
* **Check upstream dependencies**: Make sure any upstream Nodes have been created successfully before trying to reference them. Run the Create operation on upstream Nodes if needed.
* **Review table aliases**: If using aliases in your SQL, verify they match exactly what you've defined and are used consistently throughout your query.

[Quoted Identifiers in Coalesce]: /docs/faq/build-faq-troubleshooting/double-quoted-identifiers
[helper tokens]: /docs/reference/helper-tokens
[ref functions]: /docs/reference/ref-functions

---

## IP Address Is Not Allowed to Access Snowflake

To fix this error, you can try the following steps:

- Verify that IP addresses have been added to the allow list correctly by contacting Snowflake support.
- Check if there is a delay in the application of the allow list. Sometimes there is a delay in allow list to be applied.
- Try connecting to Snowflake from a different network to see if the issue is specific to your network.
- Check if there are any firewall rules or network security groups that are blocking the connection to Snowflake.
- If you are using a VPN or proxy, make sure that it is configured correctly and that it is not causing any issues with the connection to Snowflake. If none of these steps resolve the issue, you may need to contact Snowflake support for further assistance.
- Check that your network policy is at **account** level and not user level.

---

## Jobs Fail After Changing Authentication or Credentials

## Description

This error occurs when you update authentication settings or credentials for a service user in Coalesce, but scheduled Jobs continue to run with the old settings. The Job Scheduler does not automatically refresh authentication details.

## Cause

The Job Scheduler is still using the old authentication method or credentials.  

## Solution

* Update the credentials in the Environment with the failed Job.
* If your organization rotates credentials use a [third-party or CI/CD](/docs/deploy-and-refresh/third-party-devops-tools) tool to manage updates.

---

## No Stages Rendered

`No Stages Rendered`

## Description

The "No Stages Rendered" error typically indicates there is an incorrect or missing configuration within a Node that is preventing the template from rendering properly. This error can occur in any template, check the error message for the location.

## Possible Causes

* **Dimension Nodes** - The "Business Key" may not be configured in the Node Configuration. You need to select the business key columns and move them to the selected side using the arrow button.
* **View Nodes** - The "Override Create SQL" option has been enabled, but the SQL in the Create SQL tab is incorrect, missing elements, or doesn't include a proper "CREATE OR REPLACE VIEW" statement. Validate that your SQL statement would execute correctly if pasted directly in your data platform.
* **Node Type Switching Issues** - When you switch a Node from one type to another, the old metadata may still be present, causing rendering issues. The solution is to create a new Node from scratch rather than switching types.
* **Missing Required Configuration** - Check for any required fields marked with red asterisks in the Node configuration. All required fields must be properly configured for the template to render.

## Possible Solutions

* **Check Node Configuration** -  Look for any missing required fields (marked with red asterisks) in the Node properties.
* **Validate SQL** -  If using custom SQL, ensure it's syntactically correct and would run in your data platform.
* **Review Template Errors** -  Click the arrow at the bottom of the Node to expand error details.
  
  

* **Recreate Problem Nodes** -  If switching Node types, delete and recreate the Node rather than converting it.
* **Check Dependencies** -  Ensure all referenced objects exist and are properly configured  The error essentially means the Jinja template cannot generate the necessary SQL stages due to missing or incorrect configuration parameters.

## Next Steps

If errors continue after troubleshooting, create a [support ticket](mailto:support@coalesce.io). Include the full error message, deployment ID, and steps to reproduce the issue to help the support team investigate.

---

## An error occurred refreshing your OAuth refresh_token

## Error

`An error occurred refreshing your OAuth refresh_token.  Please ensure your OAuth Security Integration is still valid or reauthenticate`

This can occur because:

- User is deleted
- User is disabled
- Password change
- Email address change

## Solution

Check your Snowflake OAuth configuration. By default in the code provided in the Coalesce documentation, this is set to expire every 90 days. However, this may have been set in a shorter duration by whomever set up this security integration.

You will need a Snowflake ACCOUNTADMIN to run `DESCRIBE SECURITY INTEGRATION ;`in your Snowflake instances and send you the results. In the result set, take a look at the `OAUTH_REFRESH_TOKEN_VALIDITY` parameter. Its values are displayed in seconds. Confirm that the value set matches the amount of time you expect this token to remain valid. If not, have the ACCOUNTADMIN recreate the integration with your required value. Note, you will have to re-configure your OAuth integration in Coalesce if the security integration is recreated on the Snowflake side.

---

## Permission Issues When Adding Sources

## ERROR: Permission Issues When Adding Sources

You may encounter permission errors when trying to add sources in your Workspace. These issues are usually related to Snowflake role permissions, Workspace configuration, or authentication.

## Description

This error occurs when your Snowflake role or Workspace settings prevent you from accessing source tables or storage mappings. It can also be caused by incorrectly configured warehouses or authentication tokens.

## Possible Causes

* **Snowflake Permissions**
  * Missing `USAGE` permissions on the database or schema
  * Missing `SELECT` permissions on source tables
  * No access to the warehouse specified in your credentials

* **Workspace Configuration**
  * User credentials not saved after testing
  * Storage Mappings not configured or invalid
  * Warehouse not specified in User Credentials

* **Authentication Problems**
  * OAuth authentication not refreshed
  * Incorrectly configured integration
  * Expired authentication tokens

* **Storage Location Issues**
  * Mappings point to databases or schemas that don’t exist
  * Role lacks access to mapped storage locations
  * Red/invalid storage location errors

## Possible Solutions

* **Check Your Snowflake Permissions**
  * Ensure your role has `USAGE` on the database and schema
  * Confirm `SELECT` access on all source tables
  * Verify warehouse access for your role

* **Update Workspace Configuration**
  1. Go to **Build Settings → Development Workspace → Edit Workspace → User Credentials**
  2. Click **Test Connection** to confirm authentication
  3. Click **Save** after successful authentication
  4. Verify that Storage Mappings are correct and saved

* **Verify Warehouse Settings**
  * Make sure a warehouse is specified in your User Credentials
  * Confirm the warehouse exists and is accessible by your role

* **Re-authenticate if Needed**
  * Disconnect and reconnect OAuth authentication
  * Refresh authentication tokens
  * Confirm OAuth integration is configured properly

* **Check Storage Mappings**
  * Ensure all mapped locations exist
  * Confirm your role has access to mapped databases and schemas
  * Resolve red/invalid mapping errors

## Common Troubleshooting Steps

1. Save changes to credentials and refresh the page.
2. Check for empty schemas—red errors may indicate missing objects, not permissions.
3. Confirm that all source tables and views exist in the specified location.

## Next Steps

If these steps don’t resolve the issue, the problem may be specific to your Snowflake setup or security policies. Work with your Snowflake administrator to grant the required permissions.

---

## Primary Key and Schema Change Failures

## Error

Deployment fails when attempting to change primary keys, column data types, or other schema elements. Snowflake and some other data platforms restrict or disallow certain schema changes (for example, altering column types or dropping primary key constraints) through standard DDL.

## Possible Causes

- **Primary key changes:** Adding, removing, or modifying primary key constraints may not be supported in the same way across platforms.
- **Column data type changes:** Altering column types (for example, VARCHAR to NUMBER) can require special handling or may be blocked.
- **Platform-specific DDL limits:** Each data platform has its own rules for schema evolution.

## Possible Solutions

- **Make changes off-platform:** For column data type changes, consider making the change directly in your data platform (for example, using `ALTER TABLE` in Snowflake), then sync metadata in Coalesce so it reflects the new structure.
- **Recreate the object:** In some cases, dropping and re-creating the table or view with the new schema may be necessary. Ensure you have a backup or can re-create data before doing this.
- **Check platform documentation:** Review your data platform's documentation for supported schema change operations and any workarounds.

---

## Template Rendering Error: No First Item

`Template rendering error: No first item, sequence was empty`

## Description

This error occurs when a Node's template expects a sequence or list with at least one item, but the sequence is empty. The issue is usually related to configuration problems within the Node.

## Possible Causes

* **Storage Location Issues** - The first sequence in a Node's create template is empty due to incorrect or missing storage location configuration.  
* **Missing Parameters** - A required parameter was added in the Workspace but not configured in the Node.  
* **Empty Configuration Values** - A configuration field expects a value but none exists.  
* **Incorrectly Structured Source Tables** - The source table is empty or incorrectly configured.

## Possible Solutions

* **Check Recently Added Nodes** - Focus on Nodes that were recently added or modified, since they are often the source of the error.  
* **Verify Storage Locations** - Ensure all storage locations are configured and mapped correctly.  
* **Review Node Parameters** - Confirm all required parameters are defined in both the Workspace and the target Environment.  
* **Validate Source Configurations** - Make sure source tables exist and contain the expected data structure.  
* **Recreate Problem Nodes** - If the error persists, delete and recreate the problematic Node.  
* **Retry Deployment** - In some cases, redeploying can resolve temporary template rendering issues.  

## Next Steps

If errors continue after troubleshooting, create a [support ticket](mailto:support@coalesce.io). Include the full error message, deployment ID, and steps to reproduce the issue to help the support team investigate.

---

## Incoming Request Blocked During Snowflake OAuth Setup

`Incoming Request Blocked`

You may see the message "Incoming request blocked" when setting up Snowflake OAuth with Coalesce. The error appears after OAuth authentication succeeds but before the connection is established.

## Description

This error occurs when Coalesce’s IP addresses are not added to the account-level network policy in Snowflake. While OAuth authentication completes successfully, the connection test fails because the JavaScript driver requires account-level access.

## Common Symptoms

You may see the following sequence of messages in Coalesce:

- `OAuth authentication confirmed! Testing connection...`
- `Error establishing OAuth connection`

In Snowflake login history, this error appears as:

- Error code: `390422`  
- Message: `INCOMING_REQUEST_BLOCKED`

## Possible Causes

- Required Coalesce IP addresses are missing from the account-level network policy.  
- IP addresses were added only to a user-level network policy, which the JavaScript driver does not use.  

## Possible Solutions

1. Get the list of required IP addresses from the [Network Requirements documentation](/docs/setup-your-project/setup-requirements/network-requirements).  
2. You can either [create the policy](https://docs.snowflake.com/en/user-guide/network-policies#create-a-network-rule) using Snowsight or SQL.
   1. Snowsight: Go to **Governance & security > Network policies**.
   2. SQL:

    ```sql
    GRANT USAGE ON DATABASE securitydb TO ROLE network_admin;
    GRANT USAGE ON SCHEMA securitydb.myrules TO ROLE network_admin;
    GRANT CREATE NETWORK RULE ON SCHEMA securitydb.myrules TO ROLE network_admin;
    USE ROLE network_admin;

    CREATE NETWORK RULE cloud_network TYPE = IPV4 VALUE_LIST = ('11.11.11.11/11');
    ```

:::info

The IP address must be added at the account level. User-level policy won’t fix this error.

:::

---

## What's Next?

- [Troubleshooting IP Address Access Issues](/docs/reference/error-library/ip-address-is-not-allowed-to-access-snowflake)
- [Controlling network traffic with network policies](https://docs.snowflake.com/en/user-guide/network-policies#create-a-network-rule)

---

---

## Template Rendering Errors - Jinja and Parse Issues

When a Node's Jinja or configuration templates are invalid, Coalesce can fail during **Deploy** or **Preview** with a template rendering error. This page describes frequent parse-style problems, what they usually mean, and where to look first. For deployment workflow and isolation tips, see [Troubleshooting Template Rendering Errors][]. For warehouse date functions in SQL, use [Using Dynamic Dates][], which covers warehouse functions rather than Jinja parse errors.

## How Template Errors Appear

Template rendering runs before generated SQL is sent to the warehouse. Mistakes in Jinja syntax, braces that are doubled or unclosed, invalid filters, bad JSON produced by templates, incorrect macro calls, or misuse of [Helper Tokens][] often show up in this phase. The error text might reference a line or character position in generated SQL or in template output, which helps you locate the template that produced it.

## Where Templates Run in the Node Editor

Use this list to decide which part of the Node to open first when you are tracing an error:

- **Create** - SQL and templates used when the object is created or recreated.  
- **Run** - SQL and templates used when the Node runs its load or transform logic.  
- **Pre-SQL** - SQL run before the main operation; Jinja is evaluated here when the Node is built or run, depending on your Node type and **Package**.  
- **Post-SQL** - SQL run after the main operation, with the same templating behavior as **Pre-SQL** for your type.

If the message looks like warehouse SQL, for example a `SELECT` the Node generates, the faulting template is usually in **Create** or **Run**. If it references statements you only defined in **Pre-SQL** or **Post-SQL**, open those editors on the same Node first.

## When the Error Points to a Character Position

Some messages include a position in a line of generated SQL or rendered output. Treat that output as the result of your templates, not necessarily the file where you typed the mistake. Work backward from the shown fragment to the Node field that could emit that text: **Create**, **Run**, **Pre-SQL**, **Post-SQL**, column **Transform**, or a configuration field driven by Jinja. Undo recent edits in that area, simplify the expression, and redeploy or preview again.

## Common Causes

These issues often trigger a template rendering error before SQL reaches the warehouse:

- **Unclosed or doubled `{{` and `}}`** - The renderer expects balanced expression delimiters. Extra `}` characters or missing `}}` often produce parse failures.  
- **Invalid filter or expression syntax** - Typos in filter names, wrong pipe syntax, or broken expressions after editing filters or nested `{{ }}` can yield messages such as **expected name or number** from the template engine.  
- **`{% set %}` and control blocks** - Missing `{% endif %}`, `{% endfor %}`, or invalid assignments inside `{% %}` break parsing before SQL is built.  
- **Invalid JSON from templates** - Configuration fields that must stay valid JSON can break when Jinja emits a trailing comma, an unquoted bare word, or an empty value. See [Unexpected RIGHT_BRACE][] for JSON brace issues tied to templates.  
- **Macro misuse** - Wrong argument counts, quoting, or calling a macro that itself emits invalid SQL or JSON.  
- **Helper tokens inside strings or JSON** - [Helper Tokens][] look like `{{SRC}}` but follow Coalesce replacement rules. Nesting them inside strings or JSON without the quoting pattern your macro expects can break Jinja or JSON. See [Macros and Tokens][] for the recommended `{{ my_macro('{{SRC}}') }}` style.

## Possible Solutions

Try these steps before you escalate:

- **Narrow the failing surface** - Deploy or preview a smaller change set, or 1 Node, so the error stays tied to 1 template. The workflow in [Troubleshooting Template Rendering Errors][] matches this step.  
- **Validate Jinja structure** - Compare your expressions with [Jinja Syntax][] and the official [Jinja Template Documentation][] for full grammar. Fix unclosed blocks, filters, and `{% %}` pairs.  
- **Separate template errors from warehouse errors** - If the message is a warehouse **SQL compilation error** and there is no template rendering prefix, the template phase already succeeded. Fix object names, permissions, or warehouse SQL in the rendered statement instead of chasing Jinja delimiters.  
- **Check JSON-like configuration** - For fields on a Node or in a custom Node Type that store JSON, preview the rendered value mentally: strings in double quotes, no trailing commas, braces balanced. Use [Unexpected RIGHT_BRACE][] when the error mentions JSON state or stray `}`.  
- **Confirm parameters and Packages** - Missing **Workspace** parameters or outdated **Packages** can surface as template failures when templates expect values. Align with the parameter checks in [Troubleshooting Template Rendering Errors][].  
- **Review helper and ref patterns** - Use [Helper Tokens][] for column context in transforms and [Ref Functions][] for fully qualified object names and DAG-safe references. Do not substitute one for the other when the docs call for a specific pattern.

:::info[Template phase versus SQL compilation and error messages]

Template rendering expands Jinja and Coalesce-specific tokens into SQL and configuration text. The warehouse then parses that SQL. A clean template render followed by a warehouse error is a SQL compilation issue, not a Jinja parse problem.

Exact error strings can change with renderer or Package versions. Focus on stable patterns such as unclosed `{{`, invalid JSON from a template, or unexpected `}` in JSON, and use the official [Jinja Template Documentation][] when you need authoritative syntax.

:::

## What's Next?

- Continue with [Troubleshooting Template Rendering Errors][] for deployment isolation and parameter checks.
- Create a [support ticket][] when you still need help. Include the full error message, deployment ID, and steps to reproduce the issue.

---

[Troubleshooting Template Rendering Errors]: /docs/reference/error-library/troubleshoot-template-rendering-errors
[Using Dynamic Dates]: /docs/reference/common-transformations/using-dynamic-dates
[Helper Tokens]: /docs/reference/helper-tokens
[Unexpected RIGHT_BRACE]: /docs/reference/error-library/unexpected-brace
[Macros and Tokens]: /docs/reference/jinja/use-jinja-to-build-a-macro#macros-and-tokens
[Jinja Syntax]: /docs/reference/jinja/jinja-syntax
[Jinja Template Documentation]: https://jinja.palletsprojects.com/en/stable/templates/
[Ref Functions]: /docs/reference/ref-functions
[support ticket]: mailto:support@coalesce.io

---

## Error: Cannot clone transient object

## Error

If you receive an error stating that transient objects cannot be cloned when deploying Nodes in Coalesce, this typically occurs due to Snowflake's fundamental restrictions on transient table operations. The error may appear when Coalesce attempts to use its default clone-based deployment strategy on transient tables.

You may also encounter this error when an object in your project previously referenced a different schema location that has since been deleted. During deployment, Coalesce attempts to clean up objects from their old locations, which can cause failures if the previous schema no longer exists.

```txt
SQL compilation error: Transient object cannot be cloned to a permanent object. name=OperationFailedError, code=002120, data={"errorCode":"002120","age":0,"sqlState":"0A000","queryId":"01bef863-0002-4f27-0000-83f907f70bce"}
```

## Possible Causes

* Cloning restrictions are the primary cause. Transient objects cannot be cloned into permanent objects in Snowflake due to platform limitations.
* Schema location changes can trigger this error. When an object previously referenced a different schema location and that schema has been deleted, deployment attempts to clean up the old location and encounters errors.
* Node type changes may not deploy properly. Changing Node types from regular tables to transient tables sometimes results in tables remaining as regular tables instead of becoming transient.

## Possible Solutions

* Drop the problematic table manually in Snowflake and then run the deployment again. This recreates the table with the correct configuration.
* Use [advanced deployment strategies][] for transient tables. Advanced deployment strategies are designed to handle transient table limitations.
* Leverage Marketplace solutions. New Nodes that support transient tables are available in the Marketplace under [Base Node Types Advanced Deploy][].
* Review schema mappings before deployment. Ensure all referenced schemas exist and are properly configured in your Build Settings to avoid cleanup errors on deleted schemas.

[advanced deployment strategies]: /docs/build-your-pipeline/user-defined-nodes/deployment-strategies
[Base Node Types Advanced Deploy]: /docs/marketplace/package/coalesce_snowflake_base-node-types-advanced-deploy

---

## Troubleshooting Template Rendering Errors

Template rendering errors occur when a Node's template or configuration cannot be processed correctly during deployment. These errors often appear alongside SQL compilation issues or missing object references. This guide covers common causes and resolution strategies.

## Understanding the Error

To diagnose template rendering errors:

* Go to the **Deploy** tab and open the failed deployment.  
* Double-click on the error to view the full error message and SQL.  
* Look for recurring error types such as `Template rendering error`, `SQL compilation error`, or `Object does not exist`.  

## Common Template Rendering Errors

* **Missing or Invalid Parameters**  
  * Verify that all required parameters are defined in the Node configuration.  
  * Confirm parameter values are formatted correctly. For example, wrap string parameters in quotes.  
  * Ensure custom Node Types are installed from Packages.  

* **Node Type Issues**  
  * If you changed a Node Type, create a new Node instead of converting an existing one.  
  * Confirm that Node Type packages are installed and up to date.  
  * Check for missing or invalid Node Type definitions.  

## SQL Compilation Errors

* **Object Dependencies**  
  * Ensure upstream objects exist before deploying dependent objects.  
  * Confirm referenced tables, views, and schemas are accessible.  
  * Verify object names match exactly, as they are case-sensitive.  

* **Column and Schema Issues**  
  * Confirm all referenced columns exist in source tables.  
  * Check for typos in column or table names.  
  * Escape special characters in column descriptions when necessary.  

## Storage and Environment Issues

* **Storage Location Mappings**  
  * Verify storage locations are mapped correctly in the target environment.  
  * Confirm database and schema mappings point to existing objects.  
  * Ensure the deployment user has the required permissions.  

* **Authentication Issues**  
  * Re-authenticate your connection if you encounter OAuth or credential errors.  
  * Confirm service account permissions for automated deployments.  
  * If network policies are enabled, ensure IP addresses are whitelisted.  

## Resolution Strategies

* **Incremental Troubleshooting**  
  * Deploy a smaller subset of changes first.  
  * Break large commits into smaller, testable pieces.  
  * Deploy a single Node at a time to isolate the issue.  

* **Environment-Specific Solutions**  
  * Use a clean test environment to isolate problems.  
  * Select **Clear target environment** if errors persist.  
  * Drop and recreate problematic objects manually if needed.  

* **Recovery Options**  
  * Redeploy after fixing the underlying issue.  
  * Use **Retry from failure** to continue from the last failed step.  
  * Roll back to a previous successful deployment if necessary.  

## Next Steps

If errors continue after troubleshooting, create a [support ticket](mailto:support@coalesce.io). Include the full error message, deployment ID, and steps to reproduce the issue to help the support team investigate.

---

## Unexpected RIGHT_BRACE

`Unexpected RIGHT_BRACE("}"') in state COLON' with DeployIssueTracker.monitor`

## Description

There is a syntax problem the JSON (JavaScript Object Notation).

## Possible Causes

* **Incorrect JSON Formatting** - An extra closing brace `}` has been added, or a required opening brace `{` is missing.
* **Template rendering errors** - A dynamic template, like Jinja, can create invalid JSON if a variable is empty, incorrect, or if the template logic itself has an error.
* **Parameter section formatting** - Problems with JSON structure in the [parameters](/docs/deploy-and-refresh/parameters) section of Nodes.

## Possible Solutions

* **Mismatched Braces** - Manually count your opening `{` and closing `}` braces to make sure they match. Look for any extra `}`, brackets.
* **Escaping Quotes** - Make sure any quotation marks `"` within a string value are properly escaped with a backslash (`\"`).
* **Trailing Commas** - Ensure there are no commas after the last item in an object or array.
* **Verify the Parameters Section** - Double-check the JSON structure inside the `parameters` section of your **Nodes**.
* **Look for Unclosed Jinja References** -  If you're using Jinja templating, ensure all references (for example, `{{ params.my_param }}`) are correctly written and closed.

---

## User Not in Environment

## Error

If you receive the error, "User not in environment" when deploying via the API, this is typically due to using an API token for a user who has not yet proven their Snowflake OAuth authentication for the environment and is an easy fix.

With Snowflake OAuth connectivity, while it can be configured by a single Admin/individual user for a Workspace and/or Environment, each individual user who wishes to connect via it will have to prove their authorization to use the connection individually. In other words, the OAuth authentication channel is opened via a shared trust that each individual who wishes to leverage it must prove their access to.

## Solution

The "User not in environment" error is received during deployment via the API when the user whose API token is being used has not yet proven this access. This is corrected via the following steps:

1. The affected user should login to the Coalesce UI.
2. Launch the main workspace that contains the Environment you are deploying to.
3. Open the Build Settings, and navigate to Environments
4. Edit the Environment you are deploying to
5. Navigate to OAuth Settings and confirm they are configured. Do not change them if so; configure them if not.
6. Navigate to User Credentials and confirm your authentication type is set to Snowflake OAuth, and your Role and Warehouse are filled in if your user does not have a default role and/or warehouse.
7. Click Authenticate to prove you have access via the OAuth connection.

After completing the above, your "User not in environment" error should resolve itself when you retry deploying via the API.

---

## AI Assistant(Glossary)

The **AI Assistant** in **Coalesce Catalog** helps you discover assets and documentation through conversational search. It uses Catalog metadata such as descriptions, **Owners**, **Lineage**, **Metrics**, and **Catalog glossary terms** as context.

The in-app assistant complements **Catalog MCP**, which connects the same metadata to external tools such as Claude or Cursor.

Responses respect your Catalog permissions. The assistant summarizes what Catalog already indexes; it does not run warehouse queries unless integrated workflows explicitly allow it.

See [AI Assistant context][] for supported asset types and [MCP Integration][] for external assistants.

[AI Assistant context]: /docs/catalog/ai-assistant/ai-assistant/ai-assistant-context
[MCP Integration]: /docs/catalog/developer/mcp-integration

---

## castor-extractor

**castor-extractor** is a Python package of command-line tools for **client-managed extraction**. Commands such as `castor-extract-*` read metadata from supported warehouses and BI platforms and write local files.

You upload those files with `castor-upload` after optional validation with `castor-file-check`. Catalog never connects to your source during client-managed sync; it processes the files you send.

Install extras per source (for example Tableau or Snowflake) and run extraction on a schedule that matches your governance needs.

See [Castor Extractor][] for installation, configuration, and upload steps.

[Castor Extractor]: /docs/catalog/developer/castor-extractor

---

## Catalog Glossary Term

A **Catalog glossary term** is documentation your organization creates inside **Coalesce Catalog** under **Knowledge**. It defines business language such as "active customer" or "net revenue" and links to related tables, **Metrics**, and **Domains**.

This docs site glossary at `/docs/reference/glossary` defines **Coalesce product vocabulary**: Workspace, Deploy, Lineage, and similar terms used across Transform and Catalog documentation.

Both help readers understand data, but they serve different audiences:

- **Catalog glossary terms**: your company's business definitions, editable in the Catalog app.
- **Docs site glossary**: Coalesce platform terms, maintained in documentation and available as inline `<Definition>` tooltips.

See [Knowledge Pages][] to author Catalog glossary terms and [Document a Documentation Initiative][] for rollout guidance.

[Knowledge Pages]: /docs/catalog/use-cases/document/documentable-assets/knowledge-pages
[Document a Documentation Initiative]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-3-create-key-metrics-and-glossary-terms

---

## Catalog MCP

**Catalog MCP** exposes **Coalesce Catalog** to MCP-compatible AI tools such as Claude, Cursor, and Dust. Your assistant can search tables, read descriptions, traverse **Lineage**, and list **Catalog glossary terms** using your API token and region endpoint.

MCP complements the in-app **AI Assistant**. Use MCP when you want Catalog context inside your IDE or chat tool without opening the Catalog UI.

Configure the MCP server URL for your region, US or EU, and authenticate with a Catalog API token from **Settings > API**. See the [Developer guide][] for token creation and rotation.

See [MCP Integration][] for setup and tool reference.

[MCP Integration]: /docs/catalog/developer/mcp-integration
[Developer guide]: /docs/catalog/developer

---

## Client-Managed Extraction

**Client-managed extraction** means you run metadata collection inside your network and deliver files to Catalog. Catalog ingests the upload instead of pulling live from the source system.

Choose this path when firewalls, security policy, or trial setup prevent a persistent Catalog connection. You control scheduling, credentials, and network access.

The typical flow uses **castor-extractor** CLIs to write JSON or CSV files, optional validation, then upload to Catalog's ingestion bucket.

See [Castor Extractor][] for installation and [Quick Set Up][] for which integrations support client-managed mode.

[Castor Extractor]: /docs/catalog/developer/castor-extractor
[Quick Set Up]: /docs/catalog/quick-set-up

---

## Coalesce Catalog

**Coalesce Catalog** centralizes metadata from warehouses, BI tools, transformation platforms, and other integrations. You search assets, read documentation, explore **Lineage**, and collaborate on definitions without switching tools.

Catalog complements **Coalesce Transform**:

- Transform builds and runs pipelines in the warehouse.
- Catalog discovers what exists, who uses it, and how assets connect.

Catalog pulls descriptions, owners, query history, popularity, and quality signals where integrations support them. Teams use Catalog for onboarding, governance, and self-serve analytics.

See the [Catalog user guide][] to get started.

[Catalog user guide]: /docs/catalog

---

## Column-Level Lineage

**Column-level lineage** in **Coalesce Transform** shows how output columns depend on upstream columns inside your pipeline. It helps you assess blast radius when renaming sources or changing transformations.

Transform derives column lineage from Node SQL, **Node Type** templates, and supported patterns in **Node Type V2** annotations. Complex SQL such as nested CTEs may lineage partially until patterns are supported.

Column-level lineage inside Transform complements **Field-level lineage** in **Coalesce Catalog**, which spans warehouse and BI assets across integrations.

See [Getting Started with Node Type V2][] for annotation patterns that improve lineage capture.

[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes

---

## Create Template

The **Create Template** belongs to a **Node Type** and generates DDL when you **Deploy**. It creates tables and views, adds columns, and applies structural changes defined for that type.

Each **Node** inherits Create behavior from its Node Type unless overrides apply in the Node editor. In a **Workspace**, deploy runs Create Templates when you click **Create**.

Pair Create Template changes with a deploy before expecting **Refresh** to succeed on new columns or objects.

See [Nodes and Node Types][] and [Create and Run Templates][].

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Create and Run Templates]: /docs/build-your-pipeline/user-defined-nodes/create-run-template

---

## DAG

A Directed Acyclic Graph, or DAG, is a concept from mathematics and computer science that describes a specific type of graph.

- **Directed**: Connections between points in the graph, known as nodes, have a direction. In a directed graph, each connection, or edge, points from one node to another, indicating a one-way relationship.
- **Acyclic**: There are no cycles in the graph. A cycle occurs when you can start at one node and follow a path of edges that eventually loops back to the starting node. In a DAG, loops are not allowed.
- **Graph**: A graph is a collection of nodes, which can represent tasks, events, or states, and edges, which represent relationships between those nodes.

## Key Characteristics of a DAG

- **Directionality**: Each edge has a direction, showing the relationship from one node to another.
- **No Cycles**: Once you have left a node, you cannot return to it through the edges.
- **Topological Ordering**: Because a DAG has directed edges and no cycles, you can list the nodes in a linear order. This order respects the direction of the edges, meaning for every directed edge from node A to node B, A appears before B in order.

### Common Uses

Directed acyclic graphs are useful when tasks must run in a specific order.

When working in Coalesce, you use a DAG to build your pipeline. In the graph you have nodes and the connections between them. Each connector is one way, so the first nodes execute, followed by the next nodes. `CUSTOMER > STG_CUSTOMER > DIM_CUSTOMER`.

---

## Deploy Enabled

**Deploy Enabled** is a **Node** option that includes or excludes the Node from **Deploy** operations.

When Deploy Enabled is off, the Node may not receive DDL updates on deploy. Depending on **Node Type** settings, deploy can remove warehouse objects associated with disabled Nodes.

Teams disable deploy for experimental Nodes or objects they want to keep in the graph without managing in the warehouse. Review Node Type documentation before toggling Deploy Enabled in shared **Environments**.

See [Node Definition][] for related Node options.

[Node Definition]: /docs/build-your-pipeline/user-defined-nodes/node-definition

---

## Deploy

**Deploy** applies data definition language for your pipeline: create and alter tables, views, and columns generated from **Node Types** and **Create Templates**.

Deploy is required when you:

- Build a new pipeline.
- Add Nodes or change column structure.
- Update **Node Type** definitions that affect object shape.

In a **Workspace**, deploy is labeled **Create** and follows Workspace rules, including drop and recreate for some structural changes. In an **Environment**, deploy uses Environment rules, often `ALTER` and cleanup of removed objects.

You must deploy before you **Refresh**. Structure comes first, then data.

See [Understanding Deploy, Refresh, and Jobs][] for the full lifecycle.

[Understanding Deploy, Refresh, and Jobs]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh

---

## Description

A **Description** in **Coalesce Catalog** is text documentation on an asset: tables, columns, dashboards, terms, and other objects. Descriptions can come from extraction, manual edits, or **Sync back** from other tools.

Good descriptions explain business meaning, grain, refresh expectations, and known caveats. Catalog surfaces descriptions in search, asset pages, and **AI Assistant** context.

For **Coalesce Transform** customers, descriptions can flow between Catalog and Transform so warehouse comments stay aligned with version-controlled Node metadata after deploy.

See [Knowledge Pages][] for documentation practices and [Sync Back][] for Transform workflows.

[Knowledge Pages]: /docs/catalog/use-cases/document/documentable-assets/knowledge-pages
[Sync Back]: /docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce

---

## Dimension

A **Dimension**, also called a Dimension Node, creates a table of descriptive attributes used in analytics. Examples include customer profiles, product catalogs, and calendar tables.

Dimension Nodes are one of the default **Node Types** in Coalesce. They follow the same deploy and refresh model as other Nodes: **Create Template** for DDL at deploy time, **Run Template** for DML at refresh time.

Model dimensions upstream of **Fact** Nodes so joins and aggregations stay consistent across your pipeline.

See [Nodes and Node Types][] for modeling guidance.

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes

---

## Domain

A **Domain** in **Coalesce Catalog** groups data assets and documentation by business subject area. Examples include Customer, Finance, and Product.

Domains organize **Knowledge** pages, **Metrics**, **Catalog glossary terms**, and related assets so stewards and consumers browse by function rather than by warehouse schema alone.

Most organizations define fewer than ten top-level domains. You can start with department names and refine into domain hierarchies as documentation matures.

See [Knowledge Pages][] for structure examples.

[Knowledge Pages]: /docs/catalog/use-cases/document/documentable-assets/knowledge-pages

---

## Environment

An **Environment** is the deployment target for your data pipeline. You deploy DDL and run refreshes against the warehouse credentials and **Storage Mappings** configured for that Environment.

Environments are designed for steady-state operation:

- Structural changes use `ALTER` where possible instead of drop and recreate.
- Deleted objects are cleaned up in the warehouse.
- You do not develop directly in an Environment. Work happens in a **Workspace**, then flows through version control into an Environment.

Common Environment types include development, QA, staging, and production. Each can use different **Storage Mappings**, parameters, and connection settings.

See [Environments][] for configuration details.

[Environments]: /docs/get-started/coalesce-fundamentals/environments

---

## Fact

A **Fact**, also called a Fact Node, stores measurable business events at a defined grain, such as order lines, payments, or daily account balances. Fact tables usually hold numeric measures and foreign keys to **Dimension** tables.

Fact Nodes are a default **Node Type**. Like other Nodes, they deploy structure with DDL and load data on refresh.

Place Fact Nodes after staging logic so measures reflect validated, business-ready inputs.

See [Nodes and Node Types][] for default Node Types.

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes

---

## Field-Level Lineage

**Field-level lineage** traces individual columns through your stack. It answers which upstream fields populate a dashboard metric column or which transformations touch a warehouse column.

Catalog derives field lineage from query history, integration parsers, and supported transformation metadata. Coverage depends on the source technology and SQL patterns in use.

Field-level lineage complements **asset-level lineage**, which links whole tables and dashboards. Start with asset lineage for orientation, then drill to fields for impact analysis or debugging.

See [Lineage][] and [MCP Integration][] for API access to field lineage tools.

[Lineage]: /docs/catalog/navigate/lineage
[MCP Integration]: /docs/catalog/developer/mcp-integration

---

## Graph

The **Graph**, also called the **Node Graph**, is the canvas where you build and view your pipeline. Each **Node** appears as a box, and connectors show which Nodes must run before others.

The Graph represents the same directed structure as your **DAG**, a directed acyclic graph. You use the Graph to add Nodes, trace lineage inside Transform, and open **Subgraphs** for focused views.

When you hear "the graph" in Coalesce Transform, it refers to this pipeline view in your **Workspace** or **Environment**, not the Catalog lineage graph.

See [Graphs and Subgraphs][] for organization patterns.

[Graphs and Subgraphs]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs

---

## Integration

An **Integration** in **Coalesce Catalog** is a configured connection to a system Catalog reads metadata from. Examples include Snowflake, Databricks, Tableau, Power BI, dbt, and **Coalesce Transform**.

Integrations power search, **Lineage**, ownership, popularity, and optional **Sync back** flows. Administrators configure credentials and visibility in **Settings > Integrations**.

Some technologies use Catalog-managed connections. Others use **Client-managed extraction** when Catalog cannot reach the source directly.

See [Catalog integrations][] for supported technologies and setup paths.

[Catalog integrations]: /docs/catalog/integrations

---

## Job

A **Job** groups **Nodes** for targeted **Refresh** runs. Instead of refreshing every Node, you define which part of the **Graph** to run using **Selectors**.

Coalesce supports:

- **Predefined Jobs** saved in the app with IDs you can schedule.
- **Ad-hoc Jobs** built with one-time include or exclude selectors for testing.
- **Full pipeline refresh** when you need every Node.

Jobs connect to **Subgraphs** and operational patterns such as nightly incremental loads versus weekly full rebuilds.

See [Understanding Deploy, Refresh, and Jobs][] and [Scheduling Jobs in Coalesce][].

[Understanding Deploy, Refresh, and Jobs]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh
[Scheduling Jobs in Coalesce]: /docs/deploy-and-refresh/refresh/scheduling-jobs-in-coalesce

---

## Lineage(Glossary)

**Lineage** in **Coalesce Catalog** maps relationships between tables, dashboards, BI models, and other assets. It answers questions such as which sources feed a report and which dashboards depend on a table.

Catalog supports:

- **Asset-level lineage** between tables, dashboards, and models.
- **Field-level lineage** between columns where parsers and integrations provide column detail.

Use lineage views such as **Sources**, **Parents**, and **Children** to walk upstream and downstream. Lineage graphs focus on active, useful paths rather than every historical query.

Lineage in Catalog is separate from the **Graph** in Transform, though Transform-built tables can appear in Catalog lineage when the Coalesce integration is configured.

See [Lineage][] in Catalog docs.

[Lineage]: /docs/catalog/navigate/lineage

---

## Manual Lineage Importer(Glossary)

The **manual lineage importer** adds **Lineage** edges that automatic parsers miss. Use it when critical dependencies live in custom code, opaque views, or systems with limited parser coverage.

You prepare lineage inputs according to Catalog's importer format, then upload through the developer workflow documented for your deployment. Imported lineage appears alongside extracted lineage in asset views.

Manual lineage is a supplement, not a replacement, for integration-based extraction. Keep imports updated when pipelines change.

See [Manual Lineage Importer][] for format and upload steps.

[Manual Lineage Importer]: /docs/catalog/developer/catalog-apis/manual-lineage-importer

---

## Metric

A **Metric** in **Coalesce Catalog** is a documented KPI or measurable business concept in the **Knowledge** section. Metrics explain definition, calculation intent, and ownership, and they link to implementing assets.

Metrics differ from **Catalog glossary terms**, which define vocabulary, and from raw warehouse columns, which hold data. A single Metric might reference several tables or dashboards.

Teams often document Metrics alongside **Domains** during a documentation initiative so analysts share one definition of revenue, churn, or conversion.

See [Knowledge Pages][] and [Create Key Metrics and Glossary Terms][].

[Knowledge Pages]: /docs/catalog/use-cases/document/documentable-assets/knowledge-pages
[Create Key Metrics and Glossary Terms]: /docs/catalog/use-cases/document/how-to-approach-a-documentation-initiative/step-3-create-key-metrics-and-glossary-terms

---

## Node Type V2(Glossary)

**Node Type V2**, also called a V2 Node Type, is Coalesce's SQL-first authoring model for **Node Types**. V2 types set `version: 2` in the YAML definition and configure behavior through SQL annotations in each **Node** file instead of large config item panels.

V2 **Node Types** still use **Create Templates** for **Deploy** and **Run Templates** for **Refresh**. They support patterns such as `ref()`, CTEs, and column-level metadata for lineage inside Transform.

Create V2 types from **Build Settings > Node Types > Create Node Type > V2**. Duplicate existing V2 types when you want a variation without starting from scratch.

See [Getting Started with Node Type V2][] for a full walkthrough.

[Getting Started with Node Type V2]: /docs/build-your-pipeline/v2-node-types/getting-started-v2-nodes

---

## Node Type

A **Node Type** controls how **Nodes** behave. Each type includes:

- A YAML **Node definition** with UI fields, naming, and defaults.
- A **Create Template** for DDL at **Deploy**.
- A **Run Template** for DML at **Refresh**.

Coalesce ships default types such as **Source**, **Stage**, **Dimension**, **Fact**, and **View**. You can duplicate defaults, build custom types, or author **Node Type V2** SQL-first types.

Manage types in **Build Settings > Node Types**. Enabled types appear when you add Nodes to the **Graph**.

See [Nodes and Node Types][] and the [Node Type Editor][].

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Node Type Editor]: /docs/build-your-pipeline/user-defined-nodes/node-definition

---

## Node

A Node in Coalesce is the visual representation of database objects such as tables and views that form the building blocks of your data pipeline. Nodes are classified by Node Type.

Each Node is comprised of Node Properties and Options that you can configure according to your requirements, as well as elements where you define Node contents. For example, the Mapping View defines column and transformation specifications, and the Join Editor specifies relationships between sourcing Nodes.

Nodes are classified into Node Types. Each Node Type has a specification or definition, a create template, and a run template. Those templates drive the options available to Nodes of that type and how SQL is generated. They also control how each Node behaves during deployment, tied to the create template and DDL, and during refresh, tied to the run template and DML.

Node types control the creation and data manipulation of all database objects within your data pipeline. Node types are defined by Jinja/SQL templates and a YAML **Node definition** with numerous customization options in the **Node Type Editor**.

---

## Owner

An **Owner** in **Coalesce Catalog** identifies who maintains an asset. Ownership can be set manually, imported from source systems, or inferred where integrations support owner detection.

Owners appear on asset pages and in search filters. They support governance workflows: routing questions, prioritizing documentation, and assigning stewardship.

For **Coalesce Transform** integrations, owner detection may require API access and version control permissions configured in the integration setup.

See [Coalesce Transform integration][] for owner-related requirements.

[Coalesce Transform integration]: /docs/catalog/integrations/transformation/coalesce

---

## Package

A **Package** adds reusable **Node Types**, macros, and patterns to your **Project**. Install Packages from the **Coalesce Marketplace** or internal feeds through **Build Settings > Packages**.

Packages accelerate patterns such as incremental loading, snapshots, and platform-specific optimizations. After installation, new **Node Types** appear in **Build Settings > Node Types** and in the add-Node menu.

Packages version with your **Project**. Test Package upgrades in a **Workspace** before deploying to production **Environments**.

See [Manage Packages][] and the [Coalesce Marketplace][] for installation steps.

[Manage Packages]: /docs/marketplace/manage-packages
[Coalesce Marketplace]: /docs/marketplace

---

## Project

A **Project** is how Coalesce organizes your transformation work. Think of it as a folder that groups related pipelines, settings, and team access.

Each Project connects to **one version control repository**. Within a Project you create **Workspaces** for development and **Environments** for deployment targets such as QA and production.

Projects are often grouped by department, business area, or compliance boundary. For example, an organization might have separate Projects for data foundations, finance analytics, and revenue operations.

See [Projects][] in Coalesce Fundamentals for setup steps.

[Projects]: /docs/get-started/coalesce-fundamentals/projects

---

## Refresh

**Refresh** executes data manipulation language across your pipeline: inserts, merges, truncates, and updates that populate tables with transformed data.

Refresh runs the **Run Template** for each included **Node**. You can refresh the full graph or a subset through a **Job**.

In a **Workspace**, refresh is labeled **Run** and is typically on-demand while you develop. In an **Environment**, refresh can be manual, scheduled with a **Job Schedule**, or triggered through automation.

Refresh assumes objects already exist from a prior **Deploy**.

See [Understanding Deploy, Refresh, and Jobs][] for scheduling options.

[Understanding Deploy, Refresh, and Jobs]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh

---

## Run Template

The **Run Template** belongs to a **Node Type** and generates DML when you **Refresh**. It implements load logic: merges, inserts, truncates, and transforms that populate tables.

Each **Node** uses its type's Run Template during refresh. In a **Workspace**, refresh runs Run Templates when you click **Run**.

Run Templates assume warehouse objects from the **Create Template** already exist.

See [Nodes and Node Types][] and [Create and Run Templates][].

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Create and Run Templates]: /docs/build-your-pipeline/user-defined-nodes/create-run-template

---

## Selector

A **Selector** is the filter expression that defines **Job** scope. Selectors identify which **Nodes** to include or exclude when you refresh part of the **Graph**.

Predefined Jobs store selector logic with a Job ID you can schedule. Ad-hoc Jobs accept one-time selectors for testing or incident response.

Selectors often reference **Subgraphs**, tags, Node names, or Node Types depending on your Job definition syntax.

See [Understanding Deploy, Refresh, and Jobs][] and the [Selector reference][].

[Understanding Deploy, Refresh, and Jobs]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh
[Selector reference]: /docs/reference/selector

---

## Source Node

A **Source Node** points at data that already lives in your warehouse. Coalesce does not create the underlying source object. You register it so downstream **Stage**, **Dimension**, **Fact**, and **View** Nodes can read from it.

Source Nodes are mostly read-only in the Node editor. You can adjust metadata such as description or **Storage Location** mapping, but transformations happen in Nodes downstream.

In onboarding, teams often connect Source Nodes to landing-zone schemas fed by ingestion tools before building staging and mart layers.

See [Nodes and Node Types][] for default Node Types.

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes

---

## Stage

A **Stage**, also called a Stage Node, sits between sources and final marts. Use Stage Nodes to cleanse, join, or reshape data before **Dimension** or **Fact** Nodes.

By default, Stage Nodes truncate and reload on each run so the table mirrors the current source batch. You can change load behavior with Node options such as **Truncate Before** or incremental strategies from **Packages**.

Stage Nodes are a common first custom layer after Source Nodes in new pipelines.

See [Nodes and Node Types][] for Stage behavior and alternatives.

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes

---

## Storage Location

A **Storage Location** is a logical name for where **Nodes** of a given layer live, such as bronze, silver, or gold. It separates organization in Coalesce from the physical database and schema in the warehouse.

Each **Node** maps to exactly one Storage Location. One Storage Location can contain many Nodes.

**Storage Mappings** bind each Storage Location to real database and schema values per **Workspace** or **Environment**. Mapping definitions for Environments are stored in version control.

See [Storage Locations and Storage Mappings][] for examples and best practices.

[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings

---

## Storage Mapping

A **Storage Mapping** connects a **Storage Location** to a concrete database and schema in your warehouse for a given **Workspace** or **Environment**.

Mappings let the same pipeline code target different physical schemas in development, QA, and production. Incorrect mappings are a common cause of "missing" tables after deploy or refresh.

Best practice: map development **Workspaces** to scratch schemas separate from **Environment** schemas. Avoid sharing the same schemas between an active Workspace and an Environment unless the Workspace is read-only.

See [Storage Locations and Storage Mappings][] for configuration steps.

[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings

---

## Subgraph

A **Subgraph** lets you focus on part of your **Graph**. Adding a Node to a Subgraph creates a reference to the same Node, not a copy. A Node can belong to multiple Subgraphs.

Teams use Subgraphs to:

- Split large pipelines by domain or squad.
- Scope **Jobs** to the Nodes that matter for a schedule.
- Reduce visual clutter while developing.

Subgraphs do not change execution order. Dependencies still follow the full **DAG**.

See [Graphs and Subgraphs][] and [Using Subgraphs to Build Your Pipeline][].

[Graphs and Subgraphs]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs

---

## Sync Back(Glossary)

**Sync back** propagates documentation authored in **Coalesce Catalog** to connected systems. Supported targets depend on your integrations and may include **Coalesce Transform**, Snowflake, dbt, Looker, Tableau, and BigQuery.

For **Coalesce Transform**, sync back typically creates a version control branch with description updates on Nodes so you can review and merge before **Deploy** writes comments to the warehouse.

Sync back is an admin action from integration settings. Prerequisites often include API credentials and repository access for Transform customers.

After sync back to Transform, **Deploy** to the target **Environment** keeps warehouse object comments consistent with your pipeline metadata.

See [Sync Back to Coalesce][] and [Sync back troubleshooting][].

[Sync Back to Coalesce]: /docs/catalog/integrations/transformation/coalesce/sync-back-to-coalesce
[Sync back troubleshooting]: /docs/catalog/integrations/sync-back/sync-back-troubleshooting

---

## Tag(Glossary)

A **Tag** in **Coalesce Catalog** is a label applied to assets for search, filtering, and governance. Tags help teams mark PII, deprecated tables, cost centers, or campaign data sets.

Tag support varies by integration and workflow. Some sync and extraction paths preserve tags from source systems; others rely on manual tagging in Catalog.

Use tags with **Domains** and **Owners** to narrow large catalogs to the assets a team owns or certifies.

See [Tag Settings][] for organization-wide tag configuration.

[Tag Settings]: /docs/catalog/administration/tag-settings

---

## Transform MCP

**Transform MCP** is Coalesce's hosted Model Context Protocol server. MCP-compatible AI tools such as Claude, Cursor, and VS Code can list **Projects** and **Environments**, inspect runs and **Nodes**, and trigger refreshes through your Coalesce account.

The MCP endpoint lives on your regional Coalesce App host. See [About Transform MCP][] for setup, the URL pattern, authentication steps, and tool reference.

For Catalog search and lineage through MCP, see [Catalog MCP][].

[About Transform MCP]: /docs/coalesce-ai/mcp/about-mcp
[Catalog MCP]: /docs/reference/glossary/catalog-mcp

---

## Transform Source

A **Transform source** in **Coalesce Catalog** is metadata from a transformation platform connected to Catalog. For **Coalesce Transform**, Catalog maps warehouse tables to Coalesce **Nodes**, surfaces descriptions, and marks Coalesce-built tables in **Lineage**.

Transform sources help analysts see which tables pipeline code maintains versus raw landing zones. They enable description retrieval, quality test surfacing, and **Sync back** to Transform when configured.

Setup requires warehouse integration plus Transform API and often version control access for full features.

See [Coalesce Transform integration][] for capability matrix and setup.

[Coalesce Transform integration]: /docs/catalog/integrations/transformation/coalesce

---

## User-Defined Node

A **User-Defined Node**, or UDN, is any **Node** whose **Node Type** was customized or created outside the built-in defaults. UDNs include duplicated default types and entirely new types built in the **Node Type Editor**.

UDNs let teams encode organization-specific patterns: naming conventions, incremental strategies, platform-specific DDL, and governance columns.

UDNs follow the same **Deploy** and **Refresh** lifecycle as default Nodes. Store definitions in version control and test in a **Workspace** before promoting to **Environments**.

See [User-Defined Nodes][] and [Node Definition][].

[User-Defined Nodes]: /docs/build-your-pipeline/user-defined-nodes/
[Node Definition]: /docs/build-your-pipeline/user-defined-nodes/node-definition

---

## Version Control(Glossary)

**Version control** is how Coalesce tracks changes to pipeline metadata. Each **Project** links to one Git repository holding Nodes, **Node Types**, macros, and Environment **Storage Mapping** definitions.

You develop in a **Workspace** on a branch, commit changes, then deploy to **Environments** tied to protected branches or promotion flows.

Version control separates sandbox experimentation from production deployment. Catalog **Sync back** to Transform also relies on repository access to push description updates on a branch.

See [Setup Version Control][] to connect a repository.

[Setup Version Control]: /docs/setup-your-project/setup-version-control

---

## View

A **View**, also called a View Node, publishes a SQL view in your warehouse. Views do not store data separately. They execute their defining query when queried.

View Nodes suit presentation layers, security filters, or logic you do not need to materialize. Deploy still creates or replaces the view object in the warehouse.

View Nodes are one of the built-in **Node Types**. You can duplicate the type or author custom types when you need different deploy behavior.

See [Nodes and Node Types][] for when to use views versus tables.

[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes

---

## Workspace

A Workspace is a sandbox environment where you can complete the development of your data, refine, and perform preliminary validation before merging them into the codebase. Each workspace has its own graph, storage locations, macros, node types, connection configuration, and Git branch. You can create multiple workspaces to work on different tasks and merge them into the codebase using Environments.

- All Coalesce development is completed in a Workspace.
- Objects are created and data is loaded and refreshed manually on-demand, either for individual objects or for subsets of or the full data pipeline.
- When a change is made to an object, such as adding or dropping a column or renaming a column, the object is dropped and recreated rather than altered.
- Deleted objects are not cleaned up with DROP in the backend instance.

---

## Helper Tokens

<!-- markdownlint-disable MD036 -->

Use helper tokens when creating Macros or writing transformations.

Given a staging node using the fully qualified database `"GENERAL_DATA"`, schema `"WORK"`, and table `CUSTOMERS`.

## SRC

Resolves to the current source.

**Syntax**

```sql
{{SRC}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"CUSTOMER"."C_PHONE"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`CUSTOMER`.`C_PHONE`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`CUSTOMER`.`C_PHONE`
```

  </TabItem>
</Tabs>

## SRC_COL

Resolves to source column name of the column. This can be helpful if you column names have been changed. For example the column name is `Customer_Number`, but the source is `C_PHONE`.

**Syntax**

```sql
{{SRC_COL}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"C_PHONE"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`C_PHONE`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`C_PHONE`
```

  </TabItem>
</Tabs>

## SRC_NODE

Resolves to source node name of the column.

**Syntax**

```sql
{{SRC_NODE}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"CUSTOMER"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`CUSTOMER`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`CUSTOMER`
```

  </TabItem>
</Tabs>

## TGT

Resolves to current fully qualified node and column.

**Syntax**

```sql
{{TGT}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"STG_CUSTOMER_Macro"."C_PHONE"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`STG_CUSTOMER_Macro`.`C_PHONE`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`STG_CUSTOMER_Macro`.`C_PHONE`
```

  </TabItem>
</Tabs>

## TGT_COL

Resolves to current column. If the column name is different from the source it will resolve to the current name. For example, the source name is `C_PHONE`, but the current name is `CUSTOMER_PHONE`. It will resolve to `CUSTOMER_PHONE`.

**Syntax**

```sql
{{TGT_COL}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"CUSTOMER_PHONE"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`CUSTOMER_PHONE`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`CUSTOMER_PHONE`
```

  </TabItem>
</Tabs>

## TGT_NODE

Resolves to current node.

**Syntax**

```sql
{{TGT_NODE}}
```

**Example**

<Tabs groupId="helper-tokens-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
"STG_CUSTOMER_Macro"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
`STG_CUSTOMER_Macro`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
`STG_CUSTOMER_Macro`
```

  </TabItem>
</Tabs>

See [Quoted Identifiers in Coalesce][] for how Coalesce chooses quoting per Project platform.

[Quoted Identifiers in Coalesce]: /docs/faq/build-faq-troubleshooting/double-quoted-identifiers

---

## Using Jinja with SQL

Jinja can be used with SQL to expand functionality. Create control statement and use variables to write scripts that can be used in Coalesce.

In these guides you’ll go over some basic syntax and build some Macros.

## Referencing Nodes and the Current Object

Jinja templates often need **fully qualified object names** (for this Environment), **DAG-safe references to other Nodes**, or a **shorthand for the Node you are editing**. Those use **`ref()`**, **`ref_link()`**, **`ref_no_link()`**, and **`{{this}}`**, not generic Jinja variables alone. See [Ref Functions][] for syntax, examples (including Post-SQL), and how **`{{this}}`** relates to **`ref_no_link()`**.

## Where to Use Jinja in Coalesce

- [Custom Node Types][]
- [Out-of-the-Box Node Types][]
- [Node Config Options][]
- [Setting Default Parameters][]
- [Architecture][]
- [Macros][]
- [Bulk Editing][]

[Custom Node Types]: /docs/build-your-pipeline/user-defined-nodes/
[Out-of-the-Box Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Node Config Options]: /docs/build-your-pipeline/user-defined-nodes/node-config-options#generic-ui-elements
[Setting Default Parameters]: /docs/deploy-and-refresh/parameters/rtp-default#using-parameters-in-a-node-type-template
[Architecture]: /docs/architecture-and-security/architecture#template-renderer
[Macros]: /docs/reference/macros
[Bulk Editing]: /docs/build-your-pipeline/column-bulk-editor/
[Ref Functions]: /docs/reference/ref-functions

---

## Using Regular Expressions with Jinja

Regular expressions are useful for pattern matching and string manipulation. You can combine Jinja's templating features with SQL regex functions to create reusable data transformations.

## What Is a Regular Expression?

A **regular expression (regex):** A sequence of characters that defines a search pattern. Use regex when you need to match patterns in text rather than exact strings. This helps with data cleaning, validation, and transformation tasks such as removing special characters, validating formats, and extracting substrings.

## Common Regex Functions

These SQL regex functions work well with Jinja templating. The examples use [helper tokens][] to reference columns dynamically.

### REGEXP_REPLACE

Searches for a pattern and replaces it with another value. Use this function when you need to clean, format, or transform string data.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_REPLACE({{SRC}}, '-', '')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_replace({{SRC}}, '-', '')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

BigQuery patterns follow [RE2 syntax][].

```sql
REGEXP_REPLACE({{SRC}}, '-', '')
```

  </TabItem>
</Tabs>

**Example:** Remove all dashes from a phone number column. If `{{SRC}}` resolves to `"CUSTOMER"."C_PHONE"` or `` `CUSTOMER`.`C_PHONE` `` with value `555-123-4567`, the output is `5551234567`.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_REPLACE({{SRC}}, '\\s+', ' ')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_replace({{SRC}}, '\\s+', ' ')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REGEXP_REPLACE({{SRC}}, r'\\s+', ' ')
```

  </TabItem>
</Tabs>

### REGEXP_LIKE

Returns true if a string matches a specified pattern. Use this function for validation or filtering rows based on patterns.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_LIKE({{SRC}}, '^[0-9]+$')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_like({{SRC}}, '^[0-9]+$')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REGEXP_CONTAINS({{SRC}}, r'^[0-9]+$')
```

  </TabItem>
</Tabs>

**Example:** Filter rows where city names start with "San".

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
WHERE REGEXP_LIKE({{SRC}}, '^San.*')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
WHERE regexp_like({{SRC}}, '^San.*')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
WHERE REGEXP_CONTAINS({{SRC}}, r'^San.*')
```

  </TabItem>
</Tabs>

### REGEXP_SUBSTR

Extracts the substring that matches a pattern. Use this function to pull specific parts of a string.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_SUBSTR({{SRC}}, '\\w+')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_substr({{SRC}}, '\\w+')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REGEXP_EXTRACT({{SRC}}, r'\\w+')
```

  </TabItem>
</Tabs>

**Example:** Extract a year from a date string column.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_SUBSTR({{SRC}}, '[0-9]{4}')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_substr({{SRC}}, '[0-9]{4}')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REGEXP_EXTRACT({{SRC}}, r'[0-9]{4}')
```

  </TabItem>
</Tabs>

### REGEXP_INSTR

Returns the position of the first character in the substring that matches the pattern. Use this function to find where a pattern occurs.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_INSTR({{SRC}}, '[0-9]')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_instr({{SRC}}, '[0-9]')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
REGEXP_INSTR({{SRC}}, r'[0-9]')
```

  </TabItem>
</Tabs>

### REGEXP_COUNT

Returns the number of times a pattern occurs in a string. Use this function to count matches.

<Tabs groupId="jinja-regex-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
REGEXP_COUNT({{SRC}}, '[0-9]')
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
regexp_count({{SRC}}, '[0-9]')
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
ARRAY_LENGTH(REGEXP_EXTRACT_ALL({{SRC}}, r'[0-9]'))
```

  </TabItem>
</Tabs>

## Macro: Remove Non-Numeric Characters

This macro removes all non-numeric characters from a column. It's useful for cleaning phone numbers, IDs, or any field that should contain only digits.

Enter the following macro into **Build Settings > Macros**.

```sql
{%- macro remove_non_numeric(column) -%}
   REGEXP_REPLACE({{ column }}, '[^0-9]', '')
{%- endmacro %}
```

You can use this by entering this in the Transform field `{{remove_non_numeric('{{SRC}}')}}`

**Example output:**

| Input               | Output     |
| ------------------- | ---------- |
| 555-123-4567        | 5551234567 |
| (800) 555-1234      | 8005551234 |
| Phone: 123.456.7890 | 1234567890 |

## Macro: Clean Email Addresses

This macro removes whitespace and converts email addresses to lowercase for consistent data storage.

```sql
{%- macro clean_email(column) -%}
   LOWER(REGEXP_REPLACE({{ column }}, '\\s', ''))
{%- endmacro %}
```

**Usage:** Enter `{{clean_email('{{SRC}}')}}` in the transforms field.

## Macro: Extract Domain from Email

This macro extracts the domain portion from an email address.

```sql
{%- macro extract_email_domain(column) -%}
   REGEXP_SUBSTR({{ column }}, '@([A-Za-z0-9.-]+)', 1, 1, 'e')
{%- endmacro %}
```

**Example output:**

| Input                   | Output        |
| ----------------------- | ------------- |
| `user@example.com`      | `example.com` |
| `john.doe@company.org`  | `company.org` |

## Macro: Validate Email Format

This macro checks if a column value matches a basic email pattern. Use it in a WHERE clause or conditional logic.

```sql
{%- macro is_valid_email(column) -%}
   REGEXP_LIKE({{ column }}, '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$')
{%- endmacro %}
```

**Usage in a filter:**

```sql
WHERE {{ is_valid_email('"CUSTOMER"."EMAIL"') }}
```

## Macro: Mask Sensitive Data

This macro masks all but the last 4 characters of sensitive data such as credit card numbers or social security numbers.

```sql
{%- macro mask_sensitive(column) -%}
   REGEXP_REPLACE({{ column }}, '.(?=.{4})', '*')
{%- endmacro %}
```

**Example output:**

| Input            | Output           |
| ---------------- | ---------------- |
| 1234567890123456 | ************3456 |
| 123-45-6789      | *******6789      |

## Using Regex with Conditional Logic

You can combine regex functions with Jinja conditionals for more complex transformations.

```sql
{%- macro categorize_phone(column) -%}
   CASE 
      WHEN REGEXP_LIKE({{ column }}, '^1?800') THEN 'Toll Free'
      WHEN REGEXP_LIKE({{ column }}, '^1?900') THEN 'Premium'
      ELSE 'Standard'
   END
{%- endmacro %}
```

## Common Regex Patterns

Here are some patterns you can use in your macros:

| Pattern    | Matches                  |
| ---------- | ------------------------ |
| `[0-9]`    | Any digit                |
| `[^0-9]`   | Any non-digit            |
| `[A-Za-z]` | Any letter               |
| `\\s`      | Any whitespace           |
| `\\w`      | Any word character       |
| `^`        | Start of string          |
| `$`        | End of string            |
| `.`        | Any single character     |
| `*`        | Zero or more of previous |
| `+`        | One or more of previous  |

## Escaping Special Characters

When you use regex patterns in SQL strings, you need to escape backslashes. Use `\\` in your patterns where you would normally use `\`.

```sql
-- Single backslash in regex needs double backslash in SQL
REGEXP_REPLACE({{SRC}}, '\\d+', 'NUMBER')

-- To match a literal backslash, use four backslashes
REGEXP_REPLACE({{SRC}}, '\\\\', '/')
```

## What's Next?

- [Template Designer Documentation][]
- [String Functions (Regular Expressions)][]
- [Helper Tokens][]
- [Ref Functions][]

[RE2 syntax]: https://github.com/google/re2/wiki/Syntax
[Template Designer Documentation]: https://jinja.palletsprojects.com/en/stable/templates/
[String Functions (Regular Expressions)]: https://docs.snowflake.com/en/sql-reference/functions-regexp
[Helper Tokens]: /docs/reference/helper-tokens
[Ref Functions]: /docs/reference/ref-functions

---

## Jinja Syntax

Jinja is a templating engine for Python. It’s most frequently used in Django and Flask applications. In Coalesce, you can use it to expand SQL capabilities.

These are just some of the syntax options available with Jinja. See [Template Designer Documentation][] for more.

## Variables

Variables in Jinja are defined by double curly braces. Jinja replaces the `{{ variable_name }}` with the value.

```sql title="variable"
{{ variable_name }}
```

:::tip[Coalesce object names in templates]

Plain `{{ variable_name }}` is standard Jinja. For database objects that must track [Storage Mappings][] and the DAG, Coalesce provides **`ref()`**, **`ref_link()`**, **`ref_no_link()`**, and **`{{this}}`**. See [Ref Functions][] for full detail.

:::

## Statement

Statements control logic and flow. For example, starting a loop.

```sql title="Statement"
{% %}

//For Loop Example
{% set cookies = ['chocolate chip', 'oatmeal', 'sugar'] %}

{%- for cookies in cookies -%}

I love {{cookies}} cookies!

{% endfor %}

```

## Comments

Comments are not evaluated when the code is run. Comments are a good place to put information such as what the code does and the expected output.

```sql title="Comments"
{# Some comment info here
More comment information
#}
```

## Whitespace

Jinja will print any whitespace in the code block. To prevent that, adding a dash to the left or right of the percent sign will remove the whitespace.

## Filters

Filters are designed by a pipe, `|`. They change the look, format, or even output new data.

```sql title="Filters"
{{ name | capitalize }}
```

## Set

Set a variable value.

```sql title="Set"
{% set company_name = "Coalesce" %}
```

We support all builtin [Jinja filters][].

## For Loop

A for loop will loop over a sequence until a condition is met. For example, you have a list of cookies, chocolate chips, oatmeal, and sugar. For every cookie in the list, print out “I love cookies”. “I love cookies” will be printed three times.

```sql title="For Loop"
{% set cookies = ['chocolate chip', 'oatmeal', 'sugar'] %}
{%- for cookies in cookies -%}
    I love {{cookies}} cookies!
{% endfor %}

I love chocolate chip cookies!
```

## If/Else

Use if/else to provide instructions such as if this condition is met, then to this action.

```sql title="If/Else"
{% set color = 'orange' %}
{% if color == 'orange' %}

I love orange!

{% else %}

I only want orange

{% endif %}
```

## Lists

Lists are formatted using brackets, `[]`. Lists are indexed starting at 0. For example, chocolate chip has an index of 0.

```sql title="Lists"
{% set cookies = ['chocolate chip', 'oatmeal', 'sugar'] %}

```

## Dictionary

Dictionaries use `key:value` pairs.

```sql title="Dictionary"
{% set company = {
    'name': 'Coalesce',
    'type': 'Data Transformation',
    'trial': 30
} %}

{{ company.trial }}
```

## Comparison Operators

Jinja inherits the comparison operators from Python. Comparison operators are used to compare values and determine if they equate to true or false.

[Available comparison operators][]

**Example:**

In this example, a single equal, =, is used to set a variable. Use a single equals to set variables, dictionaries, and lists. A double equal, ==, is used to compare if the color is orange. In the example below, it is orange, it evaluates to true.

```sql title"Comparison Operators"
{% set color = 'orange' %}
{% if color == 'orange' %}

I love orange!

{% else %}

I only want orange

{% endif %}
```

## Jinja Syntax Summary

Because Jinja inherits from Python, you have many syntax options available. There are a few common ones in this guide. You can also use SQL statements such as CASE WHEN. This makes Jinja a very powerful tool in Coalesce.

- [https://www.w3schools.com/sql/sql\_case.asp][]
- [http://jinja.quantprogramming.com/][]
- [Available comparison operators][]

[Template Designer Documentation]: https://jinja.palletsprojects.com/en/stable/templates/
[Available comparison operators]: https://www.w3schools.com/python/gloss_python_comparison_operators.asp
[Ref Functions]: /docs/reference/ref-functions
[Storage Mappings]: /docs/get-started/coalesce-fundamentals/environments
[https://www.w3schools.com/sql/sql\_case.asp]: https://www.w3schools.com/sql/sql_case.asp
[http://jinja.quantprogramming.com/]: http://jinja.quantprogramming.com/
[Jinja filters]: https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-filters

---

## Use Jinja and SQL to Build a Macro

You’ll learn how to build a simple macro and a macro that uses another macro.

## What Is a Macro?

A [macro][] is a reusable block of code that you can define once and use multiple times. Macros can help keep your code DRY(Don’t Repeat Yourself).

## Macro Syntax

Macros are defined by using the keyword macro. Notice in the example, `{{ flower_shop(type = 'white') }}`, only type was supplied, not flower. It used the default `flower = ‘rose’` in the macro definition.

```sql

{%- macro flower_shop(type, flower = 'rose') -%}

Today I want a {{ type }} {{ flower }}.

{%- endmacro -%}

{{ flower_shop(type = 'white') }} //Today I want a white rose. 

{{ flower_shop('purple', 'orchid') }}// Today I want a purple orchid.
```

## Macro Uppercase Example

You can use this macro to change all the column data to UPPERCASE.

1. To run this macro, go to **Build Settings > Macros**.
2. Copy and paste the macro there.
3. In your Node Editor, either double click on the **Transforms** field or select the column and open the Column Editor.
4. Enter `{{uppercase_col_data('{{SRC}}')}}` or enter `{{uppercase_col_data('"CUSTOMER"."C_COMMENTS"')}}`.
    1. SRC is a token that refers to the fully qualified source column name.
5. Validate the function. Then **Create and Run** to see the data as UPPERCASE.

```sql
{%- macro uppercase_col_data(column) -%}
   UPPER({{ column }})
{%- endmacro %}
```

## Using a Macro with Another Macro

To keep your code from repeating itself, you can refer to another macro with another. In this example, we create a macro to remove dashes. Then another macro that uses `remove_dashes` and formats a phone number to standard American number, 123-456-7890.

This means the `remove_dashes` macro is available to be used and the `format_phone_number` can be used separately.

It uses the `CUSTOMER.C_PHONE` column.

1. To run this macro, go to **Build Settings > Macros**.
2. Copy and paste the macro there.
3. In your Node Editor, either double click on the Transforms field or select the column and open the Column Editor.
4. Enter `{{format_phone_number('{{SRC}}')}}` or enter `{{format_phone_number('"CUSTOMER"."C_PHONE"')}}`.
    1. SRC is a token that refers to the fully qualified source column name.
5. Validate the function. Then **Create and Run** to see the phone number formatted.

```sql
{% macro remove_dashes(column_name) %}
    REPLACE({{ column_name }}, '-', '')
{% endmacro %}

{% macro format_phone_number(column_name) %}
    CONCAT(
        LEFT({{ remove_dashes(column_name) }}, 3), '-',
        SUBSTRING({{ remove_dashes(column_name) }}, 4, 2), '-',
        RIGHT({{ remove_dashes(column_name) }}, 4)
    )
{% endmacro %}
```

## Macros and Tokens

Macros can also use helper tokens. Helper tokens are shorthand in Coalesce that lets you refer to a column or node without needing to type the whole name.

For example, you have a macro named `hello_goodbye`.

- Longhand - `{{hello_goodbye('"CUSTOMER"."C_PHONE"')}}`
- Shorthand - `{{hello_goodbye('{{SRC}}')}}`. This uses SRC to refer to fully qualified source column name.

Check out [Helper Tokens][].

For **fully qualified table, view, or procedure names**, including **`{{this}}`** for the current Node, use the patterns in [Ref Functions][] (not helper tokens alone).

[macro]: /docs/reference/macros/
[Helper Tokens]: /docs/reference/helper-tokens
[Ref Functions]: /docs/reference/ref-functions

---

## Macros

## What Are Macros?

Macros are functions that take in arguments and return a string. These strings can then be used in node templates, transformations, and join strings to produce strings consistently throughout your project. The strings returned ultimately represent the data platforms SQL commands (see Example SQL Generated by the Macro below).

This allows your team to build reusable code that is defined in a single location within the macro but reused repeatedly throughout your project.

They can be defined within a **Node Template** or in **Build Settings** **Macros**.

:::info[Macro Tutorial]

For a tutorial on building Macros with Jinja see [Use Jinja to Build a Macro][].

:::

## How To Use Macros

Macros are defined in Jinja 2 syntax and are populated with the data platforms SQL code. For information on writing in Jinja, refer to the [Official Jinja Documentation][].

Macros can be used nearly anywhere in the application; however, the metadata context given to the macro can vary from place to place.

Below you'll find a simple macro that receives a number as an argument and returns a string 'ODD' or 'EVEN'.

```python title="Simple macro"
{%- macro even_odd(column) -%}
    CASE WHEN MOD({{ column }}, 2) = 0 THEN 'EVEN' ELSE 'ODD' END
{%- endmacro %}
```

If you wanted to use it as a transform in a node, you would call it using the syntax `{{ function(‘“schema”.”table”’) }}`. So to use the above one would enter `{{even_odd('"CUSTOMER"."C_NATIONKEY"')}}` or `{{even_odd('{{SRC}}')}}` as a shorthand.

:::warning[Data Types]

Make sure the data type is VARCHAR or another valid SQL string type, otherwise you may receive an error.

:::

Below you'll find the SQL that was generated by the `STG_CUSTOMER` Node above, including the macro and its transformation.

```sql
EXPLAIN USING text
SELECT
  "CUSTOMER"."C_CUSTKEY" AS "C_CUSTKEY",
  "CUSTOMER"."C_NAME" AS "C_NAME",
  "CUSTOMER"."C_ADDRESS" AS "C_ADDRESS",
  CASE
    WHEN MOD("CUSTOMER"."C_NATIONKEY", 2) = 0 THEN 'EVEN'
    ELSE 'ODD'
  END AS "C_NATIONKEY",
  "CUSTOMER"."C_PHONE" AS "C_PHONE",
  "CUSTOMER"."C_ACCTBAL" AS "C_ACCTBAL",
  "CUSTOMER"."C_MKTSEGMENT" AS "C_MKTSEGMENT",
  "CUSTOMER"."C_COMMENT" AS "C_COMMENT"
FROM
  "COALESCE_SAMPLE_DATABASE"."TPCH_SF001"."CUSTOMER" "CUSTOMER"
```

To use a macro in a node template simply call it as you would a regular function, inside double curly braces, passing in any required parameters.

### Macros in Custom Nodes (UDN)

You can reference macros in Node by adding it to the create template, run template or referencing it in the Node definition.

```sql title="Hash column macro"
{%- macro hash_columns(columns_list) -%}
    SHA2(CONCAT(
        {%- for column in columns_list -%}
            COALESCE(CAST({{ column }} AS STRING), '')
            {%- if not loop.last -%},{%- endif -%}
        {%- endfor -%}
    ), 256)
{%- endmacro -%}

{%- macro ref_raw(location_name, node_name) -%}
    {%- raw -%}{{ ref('{% endraw %}{{- location_name }}{% raw %}', '{% endraw %}{{ node_name }}{% raw %}') }}{% endraw %}
{%- endmacro -%}
```

```yaml title="Node definition using macro"
systemColumns:
  - displayName: "HASH_KEY"
    transform: "{{ hash_columns(config.businessKeyColumns.split(',')) if config.hashBusinessKeys else 'NULL' }}"
    dataType: "VARCHAR(64)"
    placement: "beginning"
    attributeName: "isHashKey"
```

```sql title="Run template using macro"
{# Run Template #}
{% if config.materializationType == 'table' %}
    {{ stage('Insert Data') }}
    INSERT INTO {{ ref_no_link(node.location.name, node.name) }}
    (
        {%- for col in columns %}
            "{{ col.name }}"
            {%- if not loop.last -%},{%- endif %}
        {%- endfor %}
    )
{% endif %}

SELECT
    {%- for col in columns %}
        {%- if col.transform %}
            {{ col.transform }} AS "{{ col.name }}"
        {%- else %}
            {{ get_source_transform(col) }} AS "{{ col.name }}"
        {%- endif %}
        {%- if not loop.last -%},{%- endif %}
    {%- endfor %},
    
    {# Use the hash_columns macro to generate a hash key #}
    {% if config.hashBusinessKeys and config.businessKeyColumns %}
        {{ hash_columns(config.businessKeyColumns.split(',')) }} AS "HASH_KEY"
    {% endif %}
FROM {{ sources[0].join.joinCondition.split('FROM')[1].strip() }}
```

## Macros Run Order

Depending on where macros are used in the app determines the precedence.

Macros are run in the following order:

1. Coalesce internal macros - Runs first
2. Workspace
3. Packages
4. Package Configuration
5. Macros in Node Templates - Runs last

This means that if there is a macro in a Package and one in a Node template that contradicts or edits the same data, the macro in the Package will run, then the macro in the Node Template will run.

* If there are naming collisions, the last macro that is run, is the one that will take effect.
* In a Node (including Package Node Types), they are run starting with the Node Definition, Create Template, then Run Template.
* The order macros appear in a file is the order they are run. They will be run from top to bottom.

[Official Jinja Documentation]: https://jinja.palletsprojects.com/en/stable/
[Use Jinja to Build a Macro]: /docs/reference/jinja/use-jinja-to-build-a-macro

---

## Ref Functions

<!-- markdownlint-disable MD036 -->
<!--vale Coalesce.Headings = NO -->

These are used to generate database object names specific to the current environment and to establish relationships in the DAG for orchestrating node execution.

## When to Use Each Function

Choose the function based on whether you need a fully qualified name, a DAG dependency, or both. The table below summarizes when to use each option.

| Function | Use when you need to | Output | Creates DAG link? |
| ---------- | ------------------------ | -------- | ------------------- |
| `ref()` | Reference another Node in your SQL and track it as a dependency | Fully qualified name (for example, `"myDB"."mySCHEMA"."CUSTOMER"` on Snowflake or `` `myDB`.`mySCHEMA`.`CUSTOMER` `` on Databricks or BigQuery) | Yes |
| `ref_link()` | Force execution order without referencing the Node in SQL | Empty string | Yes |
| `ref_no_link()` | Reference a Node in SQL without adding a DAG dependency | Fully qualified name | No |
| `{{this}}` | Reference the current Node (for example, in tests or Post-SQL) | Fully qualified name of current object | N/A |

**Quick examples:**

- **JOINs, FROM clauses, SELECT in views:** use `ref()` so the graph shows the dependency.
- **Execution order only:** for example, when Node A must run before Node B but B does not query A, use `ref_link()` in the Join tab or SQL.
- **Self-reference** on the current Node with UPDATE or MERGE, **Pre-SQL or Post-SQL backups**, **subqueries where you don't want lineage**, or **external objects such as user-defined Nodes:** use `ref_no_link()` with explicit location and Node names, or **`{{this}}`** when you mean the current Node. Both resolve like a no-link reference to that object. Do not use **`ref()`** to refer to the Node you are defining, because that would imply a dependency on yourself.

## ref()

`ref` is used to resolve the fully qualified name of a Node from its logical Storage Location and Node name. Using `ref()` will also add reference to your graph as a dependency.

This is most commonly used in join conditions to render the fully qualified Node name.

**Syntax**

```sql
ref('<location_name>','<node_name>')
```

**Resolves To**

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
{{ ref('STG','CUSTOMER') }} = "myDB"."mySCHEMA"."CUSTOMER"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
{{ ref('STG','CUSTOMER') }} = `myDB`.`mySCHEMA`.`CUSTOMER`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
{{ ref('STG','CUSTOMER') }} = `myDB`.`mySCHEMA`.`CUSTOMER`
```

  </TabItem>
</Tabs>

**ref() Examples**

**Adding a Node to your graph**

The most basic example is adding a Node to your graph. In the following example, `STG_CUSTOMER` was from the source CUSTOMER Node, `FROM {{ ref('WORK', 'CUSTOMER') }}`.

'WORK' points to the [Storage Mapping][] `WORK`. The Storage Mapping `WORK` in the workspace points to the fully qualified database and schema, `SNOWFLAKE_SAMPLE_DATA.TPCH_SF10`.

Because `WORK` is a dynamic reference that points to a fully qualified database name and schema in the Storage Mapping, if you want to deploy to a new environment, then the reference will point to the fully qualified name of the database or schema in the Storage Location of the environment deployed to. In the QA Environment below, `WORK` is pointing to a different database and schema.

**JOINs and FROM clauses**

Use `ref()` whenever you query another Node so the DAG shows the dependency:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
SELECT o."O_ORDERKEY", c."C_NAME"
FROM {{ ref('WORK', 'ORDERS') }} o
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} c ON o."O_CUSTKEY" = c."C_CUSTKEY"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
SELECT o.`O_ORDERKEY`, c.`C_NAME`
FROM {{ ref('WORK', 'ORDERS') }} o
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} c ON o.`O_CUSTKEY` = c.`C_CUSTKEY`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
SELECT o.`O_ORDERKEY`, c.`C_NAME`
FROM {{ ref('WORK', 'ORDERS') }} o
INNER JOIN {{ ref('WORK', 'CUSTOMER') }} c ON o.`O_CUSTKEY` = c.`C_CUSTKEY`
```

  </TabItem>
</Tabs>

**CREATE VIEW**

When creating a view that selects from other Nodes, use `ref()` for the source in the SELECT so Coalesce can track lineage. For the view name (the current Node), use `ref_no_link()` to avoid a cyclical dependency:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
CREATE OR REPLACE VIEW {{ ref_no_link('WORK', 'V_ORDER_SUMMARY') }} AS
SELECT * FROM {{ ref('WORK', 'ORDERS') }}
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
CREATE OR REPLACE VIEW {{ ref_no_link('WORK', 'V_ORDER_SUMMARY') }} AS
SELECT * FROM {{ ref('WORK', 'ORDERS') }}
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
CREATE OR REPLACE VIEW {{ ref_no_link('WORK', 'V_ORDER_SUMMARY') }} AS
SELECT * FROM {{ ref('WORK', 'ORDERS') }}
```

  </TabItem>
</Tabs>

**ref() Summary**

- Uses [Storage Locations][] to dynamically point to your data platform's database and schema.
- To use in multiple environments, the ref location name should be the same. For example, use location WORK in both QA and Testing environments.

## ref_link()

Use `ref_link()` to add a reference to the graph as a dependency. `ref_link()` doesn’t create a string in the template. This means it will link between the Nodes, but not refer to any object.

This is most commonly used to create a dependency on the execution of a Node, without directly referencing that Node in your SQL. For example, to force a Node to run before the current Node, add `ref_link()` in the Join tab of the downstream Node.

**Syntax**

```sql
ref_link('<location_name>','<node_name>')
```

**Resolves To**

```sql
{{ ref_link('STG','CUSTOMER') }} resolves to (empty string)
```

**When to use ref_link()**

- You need Node B to run before Node A, but Node A does not query Node B (for example, B builds a table that A expects to exist).
- You want Subgraphs to show links between Nodes even when the reference is not in the main query.
- You need execution order without adding the Node to your FROM or JOIN clause.

**Example**

Add `ref_link()` as a standalone line in your SQL. It outputs nothing but creates the DAG link:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
FROM {{ ref('TGT_STAGE', 'STG_TRANSACTIONS') }} "STG_TRANSACTIONS"
WHERE "DOCDATE" >= '2021-01-01'
{{ ref_link('TGT_STAGE', 'STG_TRANSACTIONS_DELETES') }}
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
FROM {{ ref('TGT_STAGE', 'STG_TRANSACTIONS') }} `STG_TRANSACTIONS`
WHERE `DOCDATE` >= '2021-01-01'
{{ ref_link('TGT_STAGE', 'STG_TRANSACTIONS_DELETES') }}
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
FROM {{ ref('TGT_STAGE', 'STG_TRANSACTIONS') }} `STG_TRANSACTIONS`
WHERE `DOCDATE` >= '2021-01-01'
{{ ref_link('TGT_STAGE', 'STG_TRANSACTIONS_DELETES') }}
```

  </TabItem>
</Tabs>

Here, `STG_TRANSACTIONS_DELETES` must run before this Node (for example, to populate a table used elsewhere), but this query does not select from it. The `ref_link()` ensures the dependency appears in the graph and controls execution order.

## ref_no_link()

`ref_no_link()` is used to resolve the fully qualified name of a Node from its logical Storage Location and Node name. Refer to any object in the DAG without creating a connection. This is most commonly used when referencing the fully qualified name of the current Node within its own template. Without `ref_no_link`, a cyclical dependency would be created, which is not allowed. `ref_no_link()` is good for when you need to reference something once, but don’t need to track its lineage.

**Syntax**

```sql
ref_no_link('<location_name>','<node_name>')
```

**Resolves To**

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
{{ ref_no_link('STG','CUSTOMER') }} resolves to "myDB"."mySCHEMA"."CUSTOMER"
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
{{ ref_no_link('STG','CUSTOMER') }} resolves to `myDB`.`mySCHEMA`.`CUSTOMER`
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
{{ ref_no_link('STG','CUSTOMER') }} resolves to `myDB`.`mySCHEMA`.`CUSTOMER`
```

  </TabItem>
</Tabs>

**When to use ref_no_link()**

- **Self-reference**: Referencing the current Node in its own template (for example, UPDATE, MERGE, or Post-SQL). Using `ref()` would create a cyclical dependency.
- **Pre-SQL or Post-SQL backups**: Copying data from another Node without adding a DAG dependency.
- **Subqueries or filters**: Using data from another Node in a WHERE or SELECT without tracking lineage.
- **External objects**: Referencing user defined Nodes or other objects that are not Coalesce Nodes. See [Use Snowflake User Defined Functions (UDF)][] for an example.
- **Incremental loading**: Referencing a high-water mark table in a filter without creating a link.

**Example 1: Subquery in a WHERE clause**

You want to reference a set of values from the `DIM_CUSTOMER_REF_LINK` Node in the `STG_NATION` query without creating a link between the Nodes:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
FROM {{ ref('WORK', 'NATION') }} "NATION"
WHERE "NATION"."CUSTOMER_ID" IN (
  SELECT "TABLE2"."CUSTOMER_ID"
  FROM {{ ref_no_link('WORK', 'DIM_CUSTOMER_REF_LINK') }} "TABLE2"
)
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
FROM {{ ref('WORK', 'NATION') }} `NATION`
WHERE `NATION`.`CUSTOMER_ID` IN (
  SELECT `TABLE2`.`CUSTOMER_ID`
  FROM {{ ref_no_link('WORK', 'DIM_CUSTOMER_REF_LINK') }} `TABLE2`
)
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
FROM {{ ref('WORK', 'NATION') }} `NATION`
WHERE `NATION`.`CUSTOMER_ID` IN (
  SELECT `TABLE2`.`CUSTOMER_ID`
  FROM {{ ref_no_link('WORK', 'DIM_CUSTOMER_REF_LINK') }} `TABLE2`
)
```

  </TabItem>
</Tabs>

**Example 2: Self-reference in Post-SQL**

Updating the current Node's table in Post-SQL must not use `ref()` on that same Node (that would create a cyclical dependency). Use `ref_no_link()` with explicit names, or `{{this}}`, which resolves the same way for the Node being built:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET "STATUS" = 'PROCESSED'
WHERE "ORDER_DATE" < CURRENT_DATE
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `STATUS` = 'PROCESSED'
WHERE `ORDER_DATE` < CURRENT_DATE
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
UPDATE {{ ref_no_link('WORK', 'STG_ORDERS_TRANSFORM') }}
SET `STATUS` = 'PROCESSED'
WHERE `ORDER_DATE` < CURRENT_DATE
```

  </TabItem>
</Tabs>

Equivalent when `STG_ORDERS_TRANSFORM` is the current Node:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
UPDATE {{this}}
SET "STATUS" = 'PROCESSED'
WHERE "ORDER_DATE" < CURRENT_DATE
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
UPDATE {{this}}
SET `STATUS` = 'PROCESSED'
WHERE `ORDER_DATE` < CURRENT_DATE
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
UPDATE {{this}}
SET `STATUS` = 'PROCESSED'
WHERE `ORDER_DATE` < CURRENT_DATE
```

  </TabItem>
</Tabs>

**Example 3: Pre-SQL backup**

Creating a backup table from another Node without adding a DAG dependency:

```sql
CREATE OR REPLACE TABLE DOCUMENTATION.DEV.SUPPLIER_BACKUP AS
SELECT * FROM {{ ref_no_link('WORK', 'STG_SUPPLIER') }};
```

You can also find example usage in the [Stage-Base Templates section][] of our [User-defined Nodes (UDNs)][] article.

**Summary**

- It doesn’t add a relationship in your DAG.
- You can use it with filter statements like WHERE or SELECT. Good for infrequent use cases and incremental loading.

## this

The `{{this}}` template variable resolves to the **fully qualified name of the current Node’s object** (database, schema, and object name as defined by [Storage Mapping][] in the active Environment). Use it anywhere you would otherwise spell out the current Node with `ref_no_link(node.location.name, node.name)` in user-defined node templates. In [Hydrated Metadata][] for those node types, the `this` field is exactly that `ref_no_link` expression; `{{this}}` is the shorthand in SQL templates.

**Syntax**

`{{this}}`

**When to use `{{this}}`**

- **DDL** for the object the Node creates (for example, `CREATE OR REPLACE TABLE {{this}}`).
- **Post-SQL or Pre-SQL** that reads or writes the current Node’s table or view (for example, `UPDATE {{this}}` after a load). Same dependency behavior as `ref_no_link()` for that Node: no extra DAG edge to yourself.
- **Tests and custom test SQL** that need a `FROM` clause targeting the object under test.
- **Stored procedure** DDL where the procedure should live at the mapped location for the Node. See [Stored procedures][].

Do **not** use `ref()` with the current Node’s location and name in those cases, because Coalesce treats that as a graph dependency on yourself.

**Database and schema only**

`{{this}}` always includes the object (table, view, or similar). If you need only the mapped **database** and **schema** in Jinja (for example, for logging or auxiliary DDL), loop over `storageLocations` and use `location.database` and `location.schema`. See [Access Hydrated Metadata][].

**Example: test SQL**

There are several ways to use `{{this}}`. In this example you have a Customer table, and you only want rows where `C_MKTSEGMENT = 'BUILDING'`. To validate, create a test that looks for the value you don’t want, FURNITURE.

```sql
SELECT *
FROM {{this}}
WHERE  C_MKTSEGMENT = 'FURNITURE'
```

The test uses `{{this}}` for the fully qualified object. When the test runs, that expands to the correct name for the Environment.

The compiled SQL looks like:

<Tabs groupId="ref-functions-platform">
  <TabItem value="snowflake" label="Snowflake" default>

```sql
SELECT 1 WHERE EXISTS (
           SELECT *
FROM "TESTING"."DEV"."STG_CUSTOMER_THIS"
WHERE  C_MKTSEGMENT = 'FURNITURE'
```

  </TabItem>
  <TabItem value="databricks" label="Databricks">

```sql
SELECT 1 WHERE EXISTS (
           SELECT *
FROM `TESTING`.`DEV`.`STG_CUSTOMER_THIS`
WHERE  C_MKTSEGMENT = 'FURNITURE'
```

  </TabItem>
  <TabItem value="bigquery" label="BigQuery">

```sql
SELECT 1 WHERE EXISTS (
           SELECT *
FROM `TESTING`.`DEV`.`STG_CUSTOMER_THIS`
WHERE  C_MKTSEGMENT = 'FURNITURE'
```

  </TabItem>
</Tabs>

Using `{{this}}` keeps the same template valid across Environments and Workspaces.

[Storage Mapping]: /docs/get-started/coalesce-fundamentals/environments
[Storage Locations]: /docs/get-started/coalesce-fundamentals/environments
[Stage-Base Templates section]: /docs/build-your-pipeline/user-defined-nodes/join-template/#stage-based-templates
[User-defined Nodes (UDNs)]: /docs/build-your-pipeline/user-defined-nodes/
[Use Snowflake User Defined Functions (UDF)]: /docs/build-your-pipeline/build-how-to/how-to-use-snowflake-user-defined-functions-udf
[Hydrated Metadata]: /docs/build-your-pipeline/user-defined-nodes/hydrated-metadata#this
[Access Hydrated Metadata]: /docs/build-your-pipeline/user-defined-nodes/access-hyrdated-metadata#dot-notation-in-jinja-templates
[Stored procedures]: /docs/guides/stored-procedures

---

## Selector Queries

Coalesce supports searching or filtering for **Nodes** through the **Graph**, **Node Grid**, and **Column Grid** view, which can be helpful when searching for specific ones in a large DAG. **The Node Selector** is also available in the following parts of the application:

- **Browser** -Filter nodes within the entire graph
- **Subgraphs** - Filter nodes within the [Subgraph][]
- **Jobs** - Define a set of nodes to be [refreshed][]
- **API** and **CLI** - Manually refresh nodes

<figure>
  
  <figcaption>Click the <FilterSVG/> icon to expand the filters.</figcaption>
</figure>

## Search Syntax

Below you'll find a table with the available attributes and their effects, along with a screenshot of an example DAG with which you can try the provided usage examples. These will work for both **Include** and **Exclude** searches.

You can also use multiple attributes together, such as:

```sql
{name: *SUPPLIER* location: BI nodeType: Dimension }
```

| Attribute | Description | Example |
| --- | --- | --- |
| `name` | Selects a set of nodes matching the specified name | `{name:STG_SUPPLIER}` |
| `nodeID` | Selects a node with the specified nodeID | `{nodeID:98}` |
| `location` | Selects a set of nodes matching the specified storage location | `{location:STAGING}` |
| `nodeType` | Selects a set of nodes matching the specified node type | `{nodeType:"Dimension"}` |
| `subgraph` | Selects a set of nodes that exist in the specified subgraph | `{subgraph:"My Dimensions"}` |

For naming Subgraphs, using the `subgraph` attribute with Jobs, and related Build workflows, see [Using Subgraphs to Build Your Pipeline][].

Below you'll find a table with the available operators and their effects.

| Operator | Effect | Example Usage |
| --- | --- | --- |
| `OR` | Selects results for either term, like an OR statement. | `{name:STG_*} OR {name:*REGION}` |
| `AND` | Selects results for both terms, like an AND statement. | `{location:STAGING} AND {STG_REGION}` |
| `+` | Select predecessors or successors to a node, depending on placement at beginning or end. | `+{name:STG_SUPPLIER}` selects predecessors  `{name:STG_SUPPLIER}+` selects successors |
| `*` | Wildcard character, represents a multiple (zero or more) characters, including space. | `{name:STG*}` |
| `?` | Wildcard character, represents a single (exactly one) characters, including space | `{name:STG?}` |
| `""` | Allows you to search for key words that have spaces | `{nodeType:"My Custom UDN"}` |

[Subgraph]: /docs/get-started/coalesce-fundamentals/graphs-and-subgraphs
[Using Subgraphs to Build Your Pipeline]: /docs/build-your-pipeline/using-subgraphs
[refreshed]: /docs/deploy-and-refresh/refresh/

---

## What's My ID?

Coalesce APIs, the CLI (COA), and [Transform MCP][] often require numeric or string IDs for **Environments**, **Projects**, **Workspaces**, **Nodes**, **Jobs**, and runs. This page shows where to find each ID in the app and which format each integration expects.

For Transform MCP tool names that use each ID, see the ID type table below and [Available Tools][].

## ID Types at a Glance

| ID | Format | Common uses |
| --- | --- | --- |
| **Environment** ID | String | REST and MCP environment and node reads; `runs_start`, `runs_cancel` |
| **Project** ID | String | REST and MCP project reads; `environments_create`; `runs_list` filters |
| **Workspace** ID | String | REST workspace nodes; `nodes_list_workspace`, `nodes_get_workspace` |
| **Node** ID | String | REST and MCP node reads |
| **Run** ID | **Integer** | REST run reads; MCP `runs_get`, `runs_results`, `runs_retry`, `runs_cancel`, `runs_status` |
| **Job** ID | String | MCP `runs_start` when scoping to a **Job** |
| **Org** ID | String | Support and account context |

:::warning[Run ID is numeric, not a URL UUID]

The scheduler API names the numeric run identifier `runCounter`. Transform MCP tools expose it as `runID`. A run URL may contain a UUID, but that UUID is **not** the run ID REST or MCP tools accept.

:::

## Environment ID

**In the app:**

- Go to **Build Settings > Environments**.

    

- On the **Deploy** page, select an **Environment**. The ID appears in the URL or **Environment** details.

    

**Via API or Transform MCP:**

- Call [List Environments][] or the `environments_list` MCP tool and read the `id` field from the response.

## Project ID

**In the app:**

- On the **Deploy** page, filter by **Project** and read the ID from the browser URL.

    

**Via API or Transform MCP:**

- Call [List Projects][] or the `projects_list` MCP tool and read the `id` field from the response.

## Workspace ID

**In the app:**

- Go to **Build Settings > Workspaces**.

    

**Via API or Transform MCP:**

- Call [List Projects][] with the `includeWorkspaces` query parameter, or call `projects_get` with `includeWorkspaces: true`, and read **Workspace** IDs from the nested response.

## Node ID

**In the app:**

- In the **Node** editor for the **Node** you are editing.

    

- In run results for an **Environment**.

    

**Via API or Transform MCP:**

- Call [List Nodes][] with an **Environment** ID, or `nodes_list_environment` / `nodes_list_workspace` for MCP.
- Call [List Run Results][] for a run.

## Run ID

**In the app:**

- Open run results for an **Environment** on the **Deploy** page. The numeric run counter appears in run details.

    

**Via API or Transform MCP:**

- Call [List Runs][] or the `runs_list` MCP tool. Use the numeric run identifier from each run object, not the UUID in the browser URL.

```text
# Correct for runs_get, runs_status, and similar tools
runID: 42

# Wrong — UUID from a run URL is not runID
runID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
```

## Job ID

To scope a refresh run to a **Job** (REST [Start Run][] or MCP `runs_start`):

- Open **Deploy** and select an **Environment**.

    

Or call `projects_get` with `includeJobs: true` and read **Job** IDs from the nested **Workspace** data.

## Org ID

- Open your user profile in Coalesce and click **Support Information**.

    

## Tips for Transform MCP

When prompting an AI assistant to call Transform MCP tools:

- Ask it to call `projects_list` or `environments_list` first if you do not know the IDs.
- For run operations, say "use the numeric run counter" if the assistant passes a UUID from a URL.
- For full-environment refreshes, the assistant must pass a `jobID` or `includeNodesSelector`, or set `confirmRunAllNodes: true`.

## What's Next?

- [Available Tools][]
- [Connect to Transform MCP][]
- [Coalesce API reference][]

[Transform MCP]: /docs/coalesce-ai/mcp/about-mcp
[Available Tools]: /docs/coalesce-ai/mcp/mcp-available-tools
[Connect to Transform MCP]: /docs/coalesce-ai/mcp/mcp-quickstart
[Coalesce API reference]: /docs/api/
[List Environments]: /docs/api/coalesce/get-environments
[List Projects]: /docs/api/coalesce/get-projects-in-org
[List Nodes]: /docs/api/coalesce/get-nodes
[List Run Results]: /docs/api/coalesce/get-run-results
[List Runs]: /docs/api/coalesce/get-runs
[Start Run]: /docs/api/runs/startrun

---

## Step 4: Add a Data Source

This page explains how to turn tables and views that already live in your connected data platform into **Source Nodes** on the Build canvas. You'll confirm prerequisites, walk through **Add Sources**, and know where to look when expected tables do not show up. You'll also see how **Marketplace** packages add optional Node patterns on top of the standard source flow.

## Before You Begin

Use this checklist before you open **Add Sources**:

- You have a [Project][] with version control configured, and a [Workspace][] where you can develop.
- Your Workspace is connected to your data platform with valid credentials. Platform-specific steps, including **Test Connection** where the wizard exposes it, are in [Create a Workspace][] and the [Connection guides][].
- You've configured **Storage Locations** and **Storage Mappings** for the Workspace so Coalesce knows which databases and schemas map to your logical locations. See [Storage Locations and Storage Mappings][].
- The tables or views you want to model already exist in the warehouse context your credentials can read. Coalesce Transform models existing objects; it does not replace your usual ingestion or loading tools that land data in the warehouse.

## How Source Nodes Relate to Your Warehouse

A **Source Node** is the Coalesce representation of a table, external table, or view that already exists in your data platform. After you add sources, downstream Nodes read from those objects during refresh. Source metadata is treated as read-only input to the pipeline; you change location or description where the product allows, not the underlying warehouse definition here. For a full tour of Node types, see [Nodes and Node Types][].

## How to Add Sources From the Build Screen

On the **Build** screen, add existing warehouse tables and views as **Source Nodes** with these steps:

1. Open **Build** for your Workspace.
2. Select the `+` control, then **Add Sources**.

    

3. Select the sources to add, then confirm with **Add Sources**. You can inspect a preview for each selected object.

    

:::info[Modifying Source Nodes and tables]

Source Nodes reflect warehouse objects you selected. Treat the underlying source data as read-only in this workflow; you're not editing raw table definitions inside Coalesce.

:::

## If Source Tables Do Not Appear in Add Sources

Work through these checks in order:

1. **Confirm Storage Locations and mappings** - If mappings are missing or point at the wrong database or schema, objects outside the mapped context will not list as expected. Revisit [Storage Locations and Storage Mappings][] and your Workspace settings.
2. **Confirm the Workspace connection** - In **Build Settings**, open **Workspace** settings and verify account URL, authentication, role, and warehouse as required for your platform. Run **Test Connection** when it is available, then save. For Environment-specific deploy and refresh credentials, see [Create Your Environments][].
3. **Confirm platform grants** - The role Coalesce uses needs read access to the databases and schemas that hold your source data. Your administrator can align grants with the paths you mapped.
4. **Confirm you're in the right Project and Workspace** - Sources are scoped to the Workspace connection and mappings. Switching Workspaces or branches can change which objects are visible.

## What's Next?

- [Building Your Pipeline][] - Lay out Stages, Dimensions, Facts, and the rest of your graph from your new sources.
- [Using Coalesce Marketplace to Build First Class Data Pipelines][] - Practice package-driven patterns after your sources are on the canvas.
- [Nodes and Node Types][] - Deepen how Source Nodes interact with other Node types.

---

[Project]: /docs/setup-your-project/create-your-project
[Workspace]: /docs/get-started/coalesce-fundamentals/workspaces
[Create a Workspace]: /docs/setup-your-project/create-a-workspace
[Connection guides]: /docs/category/connection-guides
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[Nodes and Node Types]: /docs/get-started/coalesce-fundamentals/nodes
[Create Your Environments]: /docs/setup-your-project/create-your-environments
[Using Coalesce Marketplace to Build First Class Data Pipelines]: /docs/guides/using-coalesce-marketplace-to-build-first-class-data-pipelines
[Building Your Pipeline]: /docs/build-your-pipeline/the-build-interface/

---

## Step 6: Adding Users and Setting Permissions

You can add your team to start collaborating on building data pipelines.

<BestPractices title="User and Permissions Best Practices">
- If your service account needs to deploy and refresh, assign the Environment Admin role.
</BestPractices>

## Setting User Permissions

Each user will need a permission level set. Coalesce uses [Role-Based Access Control (RBAC)][] with three permission levels:

### Role Types

* **Organization** - They tend to have access to organization level settings such as being able to create projects for the organization. Review [Organization][] Roles for an in-depth explanation of each organization role.
* **Project** - These roles can work in the project they are assigned to. This is where data architects and data engineers will be assigned to build out the data pipeline. Review [Project][] Roles for an in-depth explanation of each organization role.
* **Environment** - Environment roles can deploy pipelines and view documentation. These are good for data and business analysts who need to review information or operations to deploy pipelines. Review [Environment][] Roles for an in-depth explanation of each organization role.

## Adding Users

Users must be added to your organization before they can access projects or environments. After a user is added, you'll need to give them user permissions.

:::info[Organization Administrators]

Only Organization administrators can add users.

:::

### Username and Password

1. Go to **Org Settings** > **Users**.
2. Click **Add New User**.
3. Fill in user details and select an initial role.
4. Users will receive an email to set their password.

### SSO

If you are using SSO, they are [provisioned][] when first logging into Coalesce. New SSO users are given the role of Org Member.

1. Go to **Org Settings** > **Users**.
2. Click on **Actions** next to the user, and select **Edit**.
3. Select the role the user should have.

<ButtonLink to="/docs/organization-and-accounts/organization-management/single-sign-on">
  Learn more about SSO
</ButtonLink>

## Service Account Users

Service accounts are non-human accounts used to run automated processes, deployments, and scheduled jobs in Coalesce. They are usually assigned an email address such as `automated-coalesce@yourcompany.com`. We recommend using another authentication method, such as key pair or machine-to-machine authentication. For initial setup, using a username and password is sufficient. Make sure to change it before moving to production.

1. Go to **Org Settings** > **Users**.
2. Click **Add New User**.
3. Fill in user details and select an initial role.
4. The service account will receive an email to set the password.

### Role and Permission Recommendations

Service accounts should follow the principle of least privilege. Only assign the roles needed for their intended purpose.

Recommended roles:

* Org Contributor  
* Org Member  
* Project Member  
* Environment Reader  
* Environment Admin (only if the service account must deploy or refresh)

If a service account is responsible for deployments or refreshes, it must have the **Environment Admin** role.

<ButtonLink to="/docs/organization-and-accounts/organization-management/role-based-access-control/service-accounts">
  Learn more about service accounts
</ButtonLink>
---

## What's Next?

* [**Role-Based Access Control (RBAC)**][] - Go over all roles before assigning to users.
* [**SSO**][] - Review our SSO implementations.
* [**Organization Management**][] - Learn about authentication methods, SSO, and account management.

---

[**Role-Based Access Control (RBAC)**]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Role-Based Access Control (RBAC)]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[provisioned]: /docs/organization-and-accounts/organization-management/single-sign-on/#important-single-sign-on-information
[Organization]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Project]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[Environment]: /docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles
[**SSO**]: /docs/organization-and-accounts/organization-management/single-sign-on/
[**Organization Management**]: /docs/organization-and-accounts/organization-management

---

## Step 5: Authentication to Coalesce

You'll need to decide how your team will sign in to Coalesce. We offer multiple methods.

## Username and Password Authentication

Username and password is the default authentication method. This is how you team will sign in, if no other options are configured. A few considerations:

- Requires users to set and remember a password.
  - Cloud storage (recommended for better security)
  - Browser storage (not recommended)
- Can be stored using either browser storage (not recommended) or cloud storage.
- Supports [multi-factor authentication (MFA)][].

## Single Sign-on (SSO)

Coalesce offers integration with multiple identity providers:

- [Microsoft Entra ID][]
- [Okta][]
- [JumpCloud][]
- [Ping Identity][]

---

## What's Next?

- [**Microsoft Entra ID**][]
- [**Okta**][]
- [**JumpCloud**][]
- [**Ping Identity**][]
- [**Multi-factor Authentication (MFA)**][]
- If your login page shows a blank or unresponsive screen after signing in, check [Troubleshooting Browser Login Issues][]
 for network and browser configuration fixes.

---

[Microsoft Entra ID]: /docs/organization-and-accounts/organization-management/single-sign-on/microsoft-entra-id
[**Microsoft Entra ID**]: /docs/organization-and-accounts/organization-management/single-sign-on/microsoft-entra-id
[Okta]: /docs/organization-and-accounts/organization-management/single-sign-on/okta-sso
[**Okta**]: /docs/organization-and-accounts/organization-management/single-sign-on/okta-sso
[JumpCloud]: /docs/organization-and-accounts/organization-management/single-sign-on/jumpcloud
[**JumpCloud**]: /docs/organization-and-accounts/organization-management/single-sign-on/jumpcloud
[Ping Identity]: /docs/organization-and-accounts/organization-management/single-sign-on/ping-identity-sso
[**Ping Identity**]: /docs/organization-and-accounts/organization-management/single-sign-on/ping-identity-sso
[multi-factor authentication (MFA)]: /docs/organization-and-accounts/account-management/multi-factor-authentication
[**Multi-factor Authentication (MFA)**]: /docs/organization-and-accounts/account-management/multi-factor-authentication
[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues

---

## BigQuery Connection Guide

In this guide, you'll learn how Google permissions work for Coalesce, how to set permissions in Google Cloud IAM, and how to authenticate in Coalesce. You can use OAuth (recommended) or a service account.

## Prerequisites

- A Google Cloud project with the **BigQuery API** enabled.
- You must be an admin user.

## OAuth Authentication

OAuth lets each user authenticate with their own Google identity instead of sharing a service account key file. Setup has two parts: a one-time Workspace configuration (admin), and per-user sign-in.

### Step 1: Create an OAuth Consent Screen

Follow these steps if you don't already have an app created.

1. Go to **APIs & Services** and click **OAuth consent screen**.
2. Go through the steps to create an App.
3. Select the Audience based on your organization policy. We suggest Internal for workspace organizations.
4. Go to **Clients** or click **Create OAuth client.**
5. Set Application type to **Web application**.
6. Under Authorized JavaScript origins add your domain `https://<your-coalesce-domain>`. For example, `https://my-org.app.coalescesoftware.io`.
7. Under Authorized redirect URIs add your domain and redirect. `https://<your-coalesce-domain>/oauthredirect`. For example, `https://my-org.app.coalescesoftware.io/oauthredirect`.
8. Copy the **Client ID** and **Client Secret**.

<DocCarousel slidesPerView={1}>
  <figure>
      
      <figcaption>Go to APIs & Services then OAuth consent screen</figcaption>
  </figure>

  <figure>
      
      <figcaption>Add the JavaScript origins and redirect URL</figcaption>
  </figure>

</DocCarousel>

### Step 2: Add OAuth To Coalesce

1. In Coalesce, go to Workspace **Settings.**
2. Open the **OAuth Settings** tab.
3. Toggle **Enable OAuth** on.
4. Click **Edit OAuth Credentials** and enter the **Client ID** and **Client Secret** from Google Cloud.
5. Save. A checkmark will confirm the credentials are stored.

    

6. In Coalesce, go to **Build Settings > Environments** and select your environment.
7. Open the **User Credentials** tab.
8. In the **Authentication Type** dropdown, select **OAuth**.
9. Click **Authenticate**. You'll be redirected to Google's consent screen.
10. Sign in with your Google account and grant BigQuery access.
11. You'll be redirected back to Coalesce. Your Google email will appear confirming the connection.
12. Click **Test Connection** to verify.
    

You've authenticated BigQuery in Coalesce. [Add some data][] to get started.

## Service Accounts

A service account uses a JSON key file instead of per-user OAuth. Use it when you want shared credentials for your Workspace, such as for automated jobs or when OAuth is not an option. You'll create a service account in Google Cloud, assign project and data set permissions, then upload the JSON key to Coalesce.

### Understanding Service Account Permissions

Service account access in BigQuery is controlled at two levels: project and data set. Project roles define what the service account can do across the project; data set roles define access to specific data sets.

### Project Level Permissions

Project permissions are roles assigned in Google Cloud IAM. These roles are inherited in data sets only when a service account is also granted permission to access that data set.

#### Data Set Level Permissions

Data set permissions control what the service account or principal can do with specific data sets. A hierarchy for common roles is **Data Viewer < Data Editor < Data Owner**.

| Role | Capabilities |
| --- | --- |
| **Data Viewer** | Can test a connection and view storage mappings, but can't add sources or build. This role isn't recommended. |
| **Data Editor** | Can do all things within your data transformation app. This role is recommended. |
| **Data Owner** | Can do all things within your data transformation app and gains extra permissions within BigQuery. |

#### Minimum Required Permissions

The minimum combination to use Coalesce with BigQuery:

| Level | Role |
| --- | --- |
| Project | BigQuery User |
| Data Set | BigQuery Data Editor |

### Set Up Project Level Permissions

1. Make sure you're in the project you want to add the service account to.
2. Go to **IAM & Admin > Service Accounts** in Google Cloud.

    

3. Click **Create service account**.

    

4. Fill out the required **Service account ID** and any other information required by your company's policy. Click **Create and Continue**.
5. Search for the role **BigQuery User**. Click **Continue**.

    

6. You can skip **Principals with access**. Click **Done**. Move on to setting data set level permissions.

### Set Up Data Set Level Permissions

1. Go to **BigQuery Studio**.
2. In the Explorer, find the data set you want to add to Coalesce.

    

3. In the data set, click **Share > Manage Permissions**.

    

4. Click **Add principal**.
5. Search for your service account name. It should look similar to `name@projectname.iam.gserviceaccount.com`.
6. Then assign roles as **BigQuery Data Editor**.

### Adding the Google Service Account to Coalesce

1. Go to **IAM & Admin > Service Accounts** in Google Cloud.
    

2. Click on the service account created for Coalesce.
3. Click **Keys**.
4. Then **Add key > Create new key**.

    

5. Download the JSON file.
6. In Coalesce in **Create Workspace** step or in **Workspace** settings, upload the JSON file from Google Cloud IAM. Then click **Test Connection**.

You've authenticated BigQuery in Coalesce. [Add some data][] to get started.

:::caution[Unable to Connect]

Make sure the permissions are set on the project and data set.

:::

## What's Next?

- [Add some data][] to get started

---

[Add some data]: /docs/setup-your-project/add-a-data-source

---

## Databricks Connection Guide

This guide will take you through connecting your Databricks account to Coalesce.

## Pre-Requisites

There are different guidelines for Databricks and Coalesce.

:::warning[Mixing Data Platforms]

A Project can only be associated with one data platform.

:::

### Databricks

* Your workspace must be using the [Unity catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspaces).
* Must be a [Databricks account admin](https://docs.databricks.com/aws/en/admin/#account-admins) create to OAuth applications.
* Databricks users must be given the permission to create a Personal Access Token (PAT) for Token authentication.

### Coalesce

* Must be a [Coalesce org admin](https://docs.coalesce.io/docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles/).
* For Coalesce, have a [Project](/docs/setup-your-project/create-your-project) with version control.
* Any users need to be added to Coalesce by [either adding them manually][] by the org admin or [setting up SSO][](Single-Sign-On) to Coalesce so they are added automatically.

## Get Your Databricks Account URL

You'll need your Databricks account URL for authentication. Your account URL is the URL after logging in. You only need the information in the brackets. No backslashes or HTTPS is needed.

`https://<abc-11111111-2222.cloud.databricks.com>/settings/workspace/`

## OAuth User-to-Machine

User-to-Machine is used for your daily work access. If you want to set up any automation, use Machine-to-Machine. You will need to create a App Connection and get the client ID and client secret along with the URL and path.

1. Go to the [Account console][]. If you don't have access, you'll need to reach out to your Databricks administrator.
2. Then go to **Settings > App Connections** and click **Add connection**.

    <figure>
      
      <figcaption>Databricks - Settings > App connections</figcaption>
    </figure>

3. Give the app a **Name**:
   1. **Redirect URL** is your Coalesce account URL at `oauthredirect`. This is usually the URL you use to log into Coalesce. For example, `https://<COALESCE_APP_DOMAIN>/oauthredirect`, `https://app.coalesce.io/oauthredirect`.

        <figure>
        
        <figcaption>Databricks - Add Connection</figcaption>
        </figure>

   2. Change the **Access Scopes** to All APIs.
   3. **Generate client secret** is checked.
   4. You can leave TTL as the default or change it.
4. Click **Add**.
5. A Client ID and Client Secret will pop-up. Save them for later.

### User-to-Machine Coalesce Configuration

You should have your Databricks client ID, client secret, and Databricks account URL before continuing.

1. In Coalesce, go to a [Project and Workspace][] that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.
2. Open the Build Settings, then Workspace, and click on the cog <CogSVG/> next to the Workspace name to open **Workspace Settings**.
3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or `https://`.

    <figure>
    
    <figcaption>Workspace Settings > User Credentials</figcaption>
    </figure>

4. Click on **User Credentials**.
5. Set Authentication Type to OAuth (User-to-Machine).

    <figure>
      
      <figcaption>Workspace Settings > User Credentials</figcaption>
    </figure>

6. Go to OAuth Settings and toggle **Enable OAuth**.
7. Click **Edit** and enter the Client ID and Client Secret from Databricks.

    <figure>
      
      <figcaption>Workspace Settings > Add Client ID and Client Secret</figcaption>
    </figure>

8. Click **Save**.
9. Go back to **User Credentials** and click **Authenticate**.
10. You'll be taken to the Build Page and see a testing, then success message.

<figure>
  
  <figcaption>Testing and Success Message</figcaption>
</figure>

## OAuth Machine-to-Machine

Machine-to-Machine is used for automation. You'll need to create a Service Principal.

1. On the Databricks Account Console, go to **User Management** and select **Service Principals**.

    <figure>
      
      <figcaption>Databricks User management</figcaption>
    </figure>

2. After giving the service principal a name, you'll grant the right permissions, then create the client ID and client secret.
   1. The service principal should have Workspace, Warehouse, and Catalog permissions.
3. Click on the service principal and go to the Permissions tab and click **Grant Access**. Add the service principal you just created as a User.

    <figure>
      
      <figcaption>Add the new service principal as user</figcaption>
    </figure>

4. Go to the Databricks **Catalog** and select the catalog the service principal should have access to.
5. Click **Grant**.

    <figure>
      
      <figcaption>Click Grant on the catalog</figcaption>
    </figure>

6. Choose the service principal and then select **Data Editor**.

    <figure>
      
      <figcaption>Grant the Data Editor Role</figcaption>
    </figure>

7. Then go to **SQL Warehouses**, and select the warehouse the service principal should have access to.

    <figure>
      
      <figcaption>SQL Warehouse Permissions</figcaption>
    </figure>

8. Click **Permissions**, and select the service principal from the list. Set it to **Can Use**.

    <figure>
      
      <figcaption>Permission Level is Can Use</figcaption>
    </figure>

    :::warning[Service Principal Missing From List]

    If your service principal is missing from the list, go to your user **Settings > Identity and access > Service principals**, and add it there too. Then go back to the SQL Warehouse step.

    :::

9. Go to the **Account console** in Databricks.
10. Select **User management > Service principals**.
11. Click the one you created, then click **Generate secret**.

    <figure>
      
      <figcaption>Generate Your Client ID and Client Secret</figcaption>
    </figure>

12. Choose how long the token should be valid for, then click **Generate**.
13. Save the Client ID and Client Secret for configuring in Coalesce.

### Machine-to-Machine Coalesce Configuration

You should have your Databricks client ID, client secret, your Databricks path, and Databricks account URL before continuing.

1. In Coalesce, go to a [Project and Workspace][] that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.
2. Open the Build Settings, then Workspace, and click on the cog <CogSVG/> next to the Workspace name to open **Workspace Settings**.
3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or `https://`.

    <figure>
    
    <figcaption>Workspace Settings > User Credentials</figcaption>
    </figure>

4. Click on **User Credentials**.
5. Set Authentication Type to **OAuth (Machine-to-MACHINE)**.
6. Add your **Databricks path** and click **Save**.
7. Click Edit and add enter the **Client ID** and **Client Secret** from Databricks.
8. **You must click Save first**.
9. Then click **Test Connection**.

    <figure>
      
      <figcaption>Workspace Settings</figcaption>
    </figure>

10. You'll be taken to the Build Page and see a testing, then success message.

    <figure>
      
      <figcaption>Testing and Success Message</figcaption>
    </figure>

## Token

:::info[OAuth]

We recommend using OAuth for authentication.

:::

Tokens or [Personal Access Tokens](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-workspace-users) (PAT) are used in Databricks to authenticate at the workspace level.

### Get Your Databricks Access Token

1. Log into your Databricks account.
2. Go to your username at the top and select **Settings**.
3. Under User, select **Developer**.
4. Next to Access Tokens, click **Manage**.
5. Click **Generate new token**.
6. Write a description for the token and choose the lifetime.
7. Set the scope to **BI Tools** and select **sql**.
8. Make sure to save the access token somewhere secure. It can’t be recovered or shown again.

<DocCarousel slidesPerView={1}>
  <figure>
    
    <figcaption>Manage access tokens and generate a new token</figcaption>
  </figure>

  <figure>
    
    <figcaption>Set scope to BI Tools and select sql</figcaption>
  </figure>
</DocCarousel>

:::info[ Databricks PAT]

Read the Databricks documentation on [Databricks personal access token authentication][] to learn more about managing your access tokens in Databricks.

:::

### Add Your Connection Details in Coalesce

You should have your Databricks access token, your Databricks path, and Databricks account URL before continuing.

1. In Coalesce, go to a [Project and Workspace][] that has been configured for Databricks. This means the repo attached is either empty or has only been used for Databricks previously.
2. Open the Build Settings, then Workspace, and click on the cog <CogSVG/> next to the Workspace name to open **Workspace Settings**.
3. Add your Databricks account URL. The account URL can’t contain a backslash, spaces, or `https://`.

    <figure>
    
    <figcaption>Workspace Settings</figcaption>
    </figure>

4. Click on **User Credentials**.
5. Set Authentication Type to Token (Cloud).
6. Token is your Databricks access token.
7. Path is the Databricks HTTP path.
8. Click **Test Connection**.

    <figure>
      
      <figcaption>Workspace Settings > User Credentials</figcaption>
    </figure>

:::tip

Congratulations, you’ve connected Databricks to Coalesce.

:::

---

## What's Next?

* [**Learn about Role-Based Access Control**](/docs/organization-and-accounts/organization-management/role-based-access-control/)
* [**Create a project and workspace**](/docs/setup-your-project/)

---

[Databricks personal access token authentication]: https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-workspace-users
[setting up SSO]: /docs/organization-and-accounts/organization-management/single-sign-on/
[either adding them manually]: /docs/organization-and-accounts/organization-management/users
[Account console]: https://accounts.cloud.databricks.com/settings/app-integrations
[Project and Workspace]: /docs/setup-your-project/

---

## Fabric Connection Guide

:::success[Learn More About Fabric]

If you’d like to explore how Fabric can support your data strategy, reach out to your Coalesce account manager or [contact our sales team](https://coalesce.io/request-demo/).

:::

## Before You Begin

### Fabric

- You must an Azure administrator or be able to grant admin permissions.
- Have a table in Fabric ready to add to Coalesce.

### Coalesce

- Must be a [Coalesce org admin](https://docs.coalesce.io/docs/organization-and-accounts/organization-management/role-based-access-control/rbac-permission-and-roles/).
- For Coalesce, have a [Project](https://www.notion.so/docs/setup-your-project/create-your-project) with version control.
- Any users need to be added to Coalesce by [either adding them manually](/docs/organization-and-accounts/organization-management/users) by the org admin or [setting up SSO](/docs/organization-and-accounts/organization-management/single-sign-on) to Coalesce so they are added automatically.

## Get Your Fabric SQL Connection String

1. Log into your Fabric app.
2. Find the workspace you want to connect to.
3. Select the three dots next to the warehouse name and then select **Copy SQL connection string.**
4. You’ll need this to connect Fabric and Coalesce.

## Create an Azure App Registration

You’ll create a Service Principal to use in Coalesce.

### Register a New App

1. Create a new app registration in Azure.
2. Give the app a name.
3. Set the supported account types based on your security preferences.
4. Leave the Redirect URI blank and click **Register**.

### Add Permissions

Go to **API Permissions** for the app you just created an add the following permissions:

### Azure SQL Database

- `app_impersonation` - Access Azure SQL DB and Data Warehouse. Set to application.
- `user_impersonation` - Access Azure SQL DB and Data Warehouse. Set to delegated.

### Create App Role

1. Go to **App roles**.
2. Click **Create app role**.
3. Configure the following options:
    1. Display Name
    2. Allowed member types: `Both`
    3. Value: `allowedMemberTypes`
    4. Description

### Certificates and Secrets

1. Go to **Certificates & secrets.**
2. Click **New client secret**.
3. Give it a name and set the expiration.
4. Copy the **Value and Secret ID**.

## Configure Fabric

1. Go to the settings (cog) <CogSVG/>.
2. Then go to **Governance and insights > Tenant Settings > Developer Settings.**
3. Check **Service principals can use Fabric APIs and Service principals can call Fabric APIs.**
4. Choose if you want it to apply to the entire organization or certain groups.

### Workspace Settings

1. Go to the Workspace and click **Manage Access**.
2. Start typing the name or ID of the service principal you created.
3. Set the role to **Contributor**.

## Connect to Coalesce

1. Create a new Project in Coalesce. Review the instructions for [Create a Project](https://docs.coalesce.io/docs/setup-your-project/create-your-project).
2. Give the Project a name and change the platform to **Fabric**.
3. If you have a Version control repository URL, you can add it. You can also skip it for now and ignore the warnings.
4. In the Project you just created, click **Create Workspace**.
5. Give Workspace a name and click **Create**.
6. Open the **Workspace Settings**.
7. Add the SQL connection string in the Fabric Connection Account. Remove the `.datawarehouse.fabric.microsoft.com.`

1. Click **User Credentials** and **click Edit.**
2. Enter the **Client ID** (Secret ID) and **Client Secret** (Value).
3. Click **Test Connection** and then **Save**.

---

## Snowflake Key Pair Authentication

Coalesce supports Snowflake's [key pair authentication][] for connecting **Development Workspaces** and **Environments** to Snowflake instances. Both encrypted and unencrypted private keys are supported. Encrypted keys have a corresponding passphrase that is required to use them, while unencrypted keys can be used directly. While keys are allowed to be encrypted with an empty passphrase by Snowflake, this is not supported in Coalesce and will result in an error.

We recommend using key pair authentication for automated actions.

## Prerequisites

- Administrative access to your Snowflake account.
- Ability to create and manage Snowflake users.
- Access to Coalesce Build Settings for Environments or Workspaces.
- Understanding of your organization's security policies for key management.

## Step 1: Get Your Snowflake Account Identifier

## Step 2: Generate Key Pair in Snowflake

1. Go through Snowflake’s [key pair authentication][] steps to generate your keys.
2. Make sure to generate a private key and save it for use in Coalesce.
3. Generate your public key using the private key created for Coalesce.
4. Assign the public key to your Snowflake user.

## Step 3: Authenticate in Coalesce

1. Navigate to **Build Settings** > **Environments or Workspaces**.
2. Select **Edit**, <CogSVG/>, on the Environment or Workspace that you want to connect to Snowflake using key pair authentication.
3. In **Edit Environment or Workspace** > **User Credentials**, select **Authentication Type** as **Key Pair**.
4. Enter your Snowflake **Username**, **Private Key**, **Private Key Passphrase (if applicable)**, **Role** and **Warehouse** into their respective fields and **Save**. Click **Test Connection** to ensure this works as expected.

:::info[Adding Your Private Key]

When entering your private key, make sure it's formatted properly. It must include the full private key including the lines BEGIN ENCRYPTED PRIVATE KEY and END ENCRYPTED PRIVATE KEY.

```text
-----BEGIN ENCRYPTED PRIVATE KEY-----
...
-----END ENCRYPTED PRIVATE KEY-----
```

:::

## Using Key Pair Authentication with Coalesce API

When using the Coalesce API to trigger Jobs programmatically, you can pass key pair credentials in your API requests.

### API Request Structure

```json
{
  "runDetails": {
    "jobID": "your-job-id",
    "environmentID": "your-environment-id"
  },
  "userCredentials": {
    "snowflakeAuthType": "KeyPair",
    "snowflakeUsername": "YOUR_SERVICE_ACCOUNT",
    "snowflakeKeyPairKey": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
    "snowflakeKeyPairPass": "your-passphrase",
    "snowflakeWarehouse": "YOUR_WAREHOUSE",
    "snowflakeRole": "YOUR_ROLE"
  }
}
```

### API Credentials vs Environment Credentials

The [Start Run API][start-run-api] requires `userCredentials` in every request. For key pair authentication, you must pass the same key pair credentials (username, private key, passphrase, role, warehouse) that are configured on the Environment in **Build Settings > Environments or Workspaces > User Credentials**.

**What this means for you:**

- **Include `userCredentials` in each request:** API integrations (PowerShell, Airflow, ADF, Snowflake Tasks, GitHub Actions, and similar) must include `userCredentials` in every Start Run request. For key pair, pass the same credentials configured on the Environment.
- **Credential rotation requires updates everywhere:** When you rotate keys or change passphrases, update credentials in Coalesce Build Settings and in all scripts, secrets, and CI/CD configurations that call the Start Run API.

#### Scheduled Jobs

For scheduled jobs in the Job Scheduler, the scheduler uses the credentials of the user who created or last modified the scheduled job, not Environment credentials. When you change authentication methods (for example, switching to key pair), scheduled jobs need to be manually updated.

#### Credential Rotation Checklist

When rotating Snowflake key pair credentials, update them in the Coalesce UI and in every API integration (PowerShell, Airflow, ADF, Snowflake Tasks, GitHub Actions, and similar). Otherwise, API-triggered Jobs will fail with authentication errors.

| Trigger | Where credentials come from |
| :------ | :--------------------------- |
| Coalesce UI (Deploy, Refresh) | **Build Settings > Environments or Workspaces > User Credentials** |
| Start Run API (`/scheduler/startRun`) | `userCredentials` in the request body (required) |
| Job Scheduler (scheduled jobs) | Credentials of the user who created or last modified the scheduled job |
| CLI | Depends on context; see [CLI setup][cli-setup] for key pair configuration |

### Important API Considerations

1. **Required fields:**
   - `snowflakeUsername` with KeyPair authentication.
   - `snowflakeKeyPairKey`: The PEM-encoded private key
   - `snowflakeAuthType` must be exactly "KeyPair" (case-sensitive).
   - `snowflakeKeyPairPass` is only required if the private key is encrypted.

2. **Key formatting for API:**
   - For JSON payloads, newlines within the key must be represented as `\n`.
   - Preserve the full key structure including headers and footers.

### Integration Examples

**PowerShell:**

```powershell
# Read and format private key
$privateKeyContent = (Get-Content "path/to/rsa_key.p8" -Raw).Replace("`r`n", "`n")

$body = @{
    runDetails = @{
        jobID = "your-job-id"
        environmentID = "your-env-id"
    }
    userCredentials = @{
        snowflakeAuthType = "KeyPair"
        snowflakeUsername = "your_service_account"
        snowflakeKeyPairKey = $privateKeyContent
        snowflakeKeyPairPass = $keyPassphrase
        snowflakeWarehouse = "your_warehouse"
        snowflakeRole = "your_role"
    }
} | ConvertTo-Json -Depth 3

$headers = @{
    "Authorization" = "Bearer $coalesceBearerToken"
    "Content-Type" = "application/json"
}

$response = Invoke-RestMethod -Uri "https://your-instance.coalescesoftware.io/scheduler/startRun" -Method Post -Headers $headers -Body $body
```

**Python (Apache Airflow):**

```python

from airflow.models import Variable

def trigger_coalesce_job():
    # Get private key from Airflow Variable (stored securely)
    private_key = Variable.get("SNOWFLAKE_PRIVATE_KEY")
    
    payload = {
        "runDetails": {
            "jobID": Variable.get("COALESCE_JOB_ID"),
            "environmentID": Variable.get("COALESCE_ENV_ID")
        },
        "userCredentials": {
            "snowflakeAuthType": "KeyPair",
            "snowflakeUsername": Variable.get("SNOWFLAKE_USERNAME"),
            "snowflakeKeyPairKey": private_key,
            "snowflakeWarehouse": Variable.get("SNOWFLAKE_WAREHOUSE"),
            "snowflakeRole": Variable.get("SNOWFLAKE_ROLE")
        }
    }
    
    headers = {
        "Authorization": f"Bearer {Variable.get('COALESCE_API_TOKEN')}",
        "Content-Type": "application/json"
    }
    
    response = requests.post(
        "https://your-instance.coalescesoftware.io/scheduler/startRun",
        json=payload,
        headers=headers
    )
    
    return response.json()
```

## Troubleshooting Common Issues

### JWT Token Invalid Errors

**Symptoms:**

- "JWT token is invalid" error messages.
- Authentication failures despite correct credentials.

**Solutions:**

1. **Verify key assignment in Snowflake:**

   ```sql
   DESC USER your_service_account;
   -- Check that RSA_PUBLIC_KEY_FP is populated with a fingerprint value
   ```

2. **Validate account identifier:** Ensure you're using the correct Snowflake account identifier format in your connection settings.

3. **Check private key format:**

   - Ensure the private key includes proper header and footer lines.
   - Verify there are no extra spaces, line breaks, or invalid characters.
   - The key should be continuous text between the header and footer.

### Private Key Parsing Failures

**Symptoms:**

- "Unable to parse private key" errors.
- Connection validation failures in Coalesce.

**Solutions:**

1. **Test key validity:**

   ```bash
   # Verify your private key is valid
   openssl rsa -in rsa_key.p8 -check
   ```

2. **Special character handling in passphrases:** If your passphrase contains special characters such as `;` or `#`, you may need to escape them:
   - Example: `hnw6a#n` should be entered as `hnw6a\#n`.
   - This is particularly important if you're storing the passphrase in configuration files.

3. **Format requirements:**
   - The private key must be in PEM format.
   - Ensure no extra whitespace before or after the header and footer lines.
   - Maintain original line breaks within the key content.

### Environment-Specific Authentication Issues

**Symptoms:**

- Authentication works in development but fails in production.
- Cached credential issues after switching authentication methods.

**Solutions:**

1. **Clear Environment cache:** After changing authentication methods, redeploy your Environments. This ensures all cached credentials are refreshed.

2. **Service account configuration:**
   - Use dedicated service accounts for automated processes
   - Verify the service account has the necessary Snowflake roles assigned:

   ```sql
   -- Check service account permissions
   SHOW GRANTS TO USER your_service_account;
   ```

3. **Consistent authentication:** Ensure all Environments use the same authentication method. Avoid mixing password and key pair authentication for the same user.

### Connection Test Failures

**Symptoms:**

- **Test Connection** button fails in Coalesce UI.
- Error messages about invalid credentials or permissions.

**Solutions:**

1. **Verify all required fields:**
   - Username must match the Snowflake user with the assigned public key
   - Private key must be complete with headers and footers
   - Passphrase must be provided if the key is encrypted
   - Role must be a valid role assigned to the user
   - Warehouse must exist and be accessible to the user

2. **Check Snowflake user status:**

   ```sql
   -- Ensure user is not locked or disabled
   SHOW USERS LIKE 'your_service_account';
   ```

3. **Validate role and warehouse access:**

   ```sql
   -- Verify the user can access the specified warehouse
   USE ROLE your_role;
   USE WAREHOUSE your_warehouse;
   ```

### Authentication Policy Conflicts

**Symptoms:**

- `Authentication attempt rejected by the current authentication policy` error.
- Key pair works in some accounts or for some users but not others.

If you see this error, Snowflake's account-level or user-level authentication policies are blocking your key pair login. Snowflake documentation states that key pair and OAuth are exempt from MFA policies, but custom authentication policies can still restrict which methods are allowed.

**What causes this:** Account-level or user-level authentication policies in Snowflake can restrict login to specific methods (for example, password only, PAT only, or SSO only). A policy that allows only PAT (Personal Access Token) authentication will block key pair users until the policy is changed or a separate policy allows key pair for the affected users.

**How to resolve it:**

1. **Identify the policy:** Ask your Snowflake SECURITYADMIN which authentication policy is assigned to your account or to your service account user. Policies can be set at the account level (default for all users) or at the user level.

2. **Request a user-level policy for service accounts:** For Coalesce service accounts that use key pair authentication, work with SECURITYADMIN to create or assign a user-level authentication policy that explicitly allows key pair authentication. This keeps account-wide policies (for example, MFA for interactive users) in place while permitting key pair for automation.

3. **Avoid PAT-only policies for key pair users:** If an account or user has a policy that allows only PAT authentication, key pair will fail. Either revert or adjust the policy to include key pair, or assign a different policy to users who need key pair for Coalesce.

:::tip[Coordinate With Your Snowflake Admin]

Authentication policies are managed in Snowflake, not in Coalesce. Document which Coalesce service accounts use key pair auth so your SECURITYADMIN can assign appropriate user-level policies.

:::

## Best Practices

### Service Accounts

- Use dedicated Snowflake service accounts for automation.
- Apply the principle of least privilege when assigning Snowflake roles.
- Monitor service account usage and access patterns.
- Document which applications and processes use each service account.

### Migration From Password Authentication

If you're migrating from password-based authentication to key pair authentication:

1. **Preparation phase:**
   - Generate and test key pairs in Development Workspaces.
   - Update any automation scripts or API integrations.
   - Coordinate with your security team on key management procedures.
   - Document the migration plan and rollback procedures.

2. **Implementation phase:**
   - Deploy key pair authentication to Development Workspaces first.
   - Test all Job executions and API integrations.
   - Gradually migrate Production Environments.
   - Maintain password authentication as a fallback during transition.

3. **Validation phase:**
   - Monitor Job execution success rates.
   - Verify all automated processes function correctly.
   - Confirm all API integrations work as expected.
   - Once stable, remove password-based authentication configurations.

[key pair authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[start-run-api]: https://docs.coalesce.io/docs/api/runs/startrun
[cli-setup]: https://docs.coalesce.io/docs/coa/version-732-and-below/coa-setup

---

## Snowflake OAuth

Coalesce supports OAuth authentication with Snowflake. When Snowflake OAuth is enabled, users can authorize their Development credentials using Single Sign-On (SSO) via Snowflake rather than submitting a username and password to Coalesce directly.

OAuth is configured per Environment and Workspace, this allows for the flexibility of different Snowflake accounts per environment and workspace. If your organization uses the same Snowflake account and OAuth configuration for multiple environments, you will need to manually copy your configuration to each environment respectively. Workspaces can be easily [duplicated][] including their settings.

:::warning[Permissions Requirement]
Only a Snowflake `ACCOUNTADMIN` role or a role with the global `CREATE INTEGRATION` privilege can create security integrations.
:::

## Step 1: Add Your Snowflake Account Identifier

### Step 2: Create a Security Integration in Snowflake

In Snowflake, execute a query to create a security integration to get the CLIENT ID and CLIENT Secret. Please see the complete documentation for this in Snowflake's [Create Security Integration](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration.html) documentation. You can find a sample `CREATE OR REPLACE` security integration query below. Make sure to update your domain.

Replace `<COALESCE_APP_DOMAIN>` with your Coalesce app domain. For example, `app.coalescesoftware.io`.

Run the first part of the code, Create Security Integration. Update the `OAUTH_REDIRECT_URI` to your app URL. Make sure it runs successfully. Then run the second part of the code to get the `OAUTH_CLIENT_ID` and `OAUTH_CLIENT_SECRET`.

  <Tabs>
  <TabItem value="create-security-integration" label="Create Security Integration - Run First" default>

```sql title="Create Security Integration"
-- Create Security Integration 
CREATE OR REPLACE SECURITY INTEGRATION "COALESCE_OAUTH" 
ENABLED = TRUE 
TYPE = OAUTH 
OAUTH_CLIENT = CUSTOM 
OAUTH_CLIENT_TYPE = 'CONFIDENTIAL' 
OAUTH_REDIRECT_URI = 'https://<COALESCE_APP_DOMAIN>/oauthredirect' 
OAUTH_ISSUE_REFRESH_TOKENS = TRUE 
OAUTH_REFRESH_TOKEN_VALIDITY = 7776000; 
```

  </TabItem>
  <TabItem value="fetch-client-id" label="Fetch ClientID and ClientSecret - Run Second">

```sql title="Fetch ClientID and ClientSecret"
    -- Fetch ClientID and ClientSecret 
    WITH INTEGRATION_SECRETS AS (
        SELECT parse_json(system$show_oauth_client_secrets('COALESCE_OAUTH')) AS SECRETS
    )
    SELECT
        SECRETS:"OAUTH_CLIENT_ID"::STRING AS CLIENT_ID,
        SECRETS:"OAUTH_CLIENT_SECRET"::STRING AS CLIENT_SECRET
    FROM
        INTEGRATION_SECRETS;
```

  </TabItem>

  </Tabs>

### Step 3: Enter the Client ID and Client Secret

As a Coalesce Organization Admin:

1. Edit your Workspace or Environment by selecting the cog icon, <CogSVG />, next to the Workspace or Environment name.
2. Go to OAuth Settings, click **Enable OAuth**, and click **Edit**.
3. Enter the **Client ID** and **Client Secret**. Click Save and then Save again.

If you need to get the values again, you can use the following:

```sql
WITH INTEGRATION_SECRETS AS (
  SELECT parse_json(system$show_oauth_client_secrets('COALESCE_OAUTH')) AS SECRETS
) 
SELECT
  SECRETS:"OAUTH_CLIENT_ID"::STRING     AS CLIENT_ID,
  SECRETS:"OAUTH_CLIENT_SECRET"::STRING AS CLIENT_SECRET
FROM 
  INTEGRATION_SECRETS;
```

### Step 4: Login with Snowflake OAuth

1. Go to the Authentication Type, set Authentication Type to **Snowflake OAuth**.
2. Enter desired **Role** then select **Authenticate** and follow the directions in the Snowflake OAuth popup. Only certain roles are able to authenticate, review the `BLOCKED_ROLES_LIST` in Snowflake OAuth.
3. Once successfully authenticated, you may input a desired **Warehouse**, **Test Connection**, and **Save**.

After OAuth has been configured for the desired Coalesce Environment or Workspace, you need to log in using Snowflake OAuth.

:::warning[ACCOUNTADMIN Role and OAuth]
Snowflake does not allow login using OAuth while specifying a role of ACCOUNTADMIN for security reasons.
:::

## Security Integration Parameters

For more configuration options, see [Snowflake's Create Security Integration](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration.html) documentation.

| Parameter | Description |
| --- | --- |
| **ENABLED (Required)** | **Coalesce requires this field to be set to `TRUE`** to enable Coalesce's Snowflake OAuth support. Enabled specifies whether to initiate operation of the integration or suspend it. Note: Although not required in Snowflake documentation, `ENABLED` defaults to `FALSE`. |
| **TYPE (Required)** | **Coalesce requires this field to be set to `OAUTH`** which uses Snowflake OAuth rather than External OAuth. |
| **OAUTH\_CLIENT (Required)** | **Coalesce requires this field to be set to`CUSTOM`**. |
| **OAUTH\_CLIENT\_TYPE (Required)** | **Coalesce requires this field to be set to `CONFIDENTIAL`**. Client type specifies the type of client being registered. Confidential clients can store a secret. They run in a protected area where end users cannot access them. |
| **OAUTH\_REDIRECT\_URI (Required)** | **Coalesce requires this field to be set to `<https://app.coalescesoftware.io/oauthredirect`\>**. Specifies the client URI. After a user is authenticated, the web browser is redirected to this URI. |
| **OAUTH\_ISSUE\_REFRESH\_TOKENS** | Boolean that specifies whether to allow the client to exchange a refresh token for an access token when the current access token has expired. If set to `FALSE`, a refresh token is not issued regardless of the integer value set in `OAUTH_REFRESH_TOKEN_VALIDITY`. User consent is revoked, and the user must confirm authorization again. Default: `TRUE` |
| **OAUTH\_REFRESH\_TOKEN\_VALIDITY** | Integer that specifies how long refresh tokens should be valid (in seconds). This can be used to expire the refresh token periodically. Note that `OAUTH_ISSUE_REFRESH_TOKENS` must be set to `TRUE`. Use a smaller value to force users to re-authenticate with Snowflake more frequently. Default: `7776000` (90 days) |

## OAuth and Snowflake Role

The role will be fixed for each OAuth connection established. If you'd like to change role, you'll need to disconnect, change role, and then reconnect.

## SSO OAuth

Once a user has authorized Coalesce with Snowflake using their identity provider, Snowflake will return a Refresh Token to Coalesce. Coalesce is then able to exchange this refresh token for an Access Token which can then be used to open a Snowflake connection and execute queries in Coalesce on behalf of users.

## Disabling Snowflake OAuth

To disable Snowflake OAuth, set the `ENABLED` parameter to `FALSE`. More details in Snowflake's [ALTER SECURITY INTEGRATION](https://docs.snowflake.com/en/sql-reference/sql/alter-security-integration).

:::warning[The OAuth Toggle]
Disabling OAuth using the Enable OAuth toggle will only prevent new OAuth connections from being created; existing connections will not be affected.
:::

```sql title="Example Snowflake Query to Disable Integration"
alter integration COALESCE_OAUTH set enabled = false
```

## Removing Snowflake OAuth

To remove the integration you can drop it using the following command, review Snowflake's, [Drop Integration](https://docs.snowflake.com/en/sql-reference/sql/drop-integration). Then you need to delete the configuration settings in Coalesce.

```sql title="Example Snowflake Query to Remove Integration"
drop integration if exists COALESCE_OAUTH;
```

## MFA Accounts

If you have MFA enabled on your **Snowflake** account, you should use OAuth authentication.

## Troubleshooting

**Invalid consent request**: When authenticating with Snowflake, OAuth successfully redirects you to the Snowflake login page, but you receive an Invalid consent request error, your Snowflake user may not have access to the Snowflake role configured when authorizing OAuth. Double-check that you have access to that role and if the role name has been correctly entered in as Snowflake is case-sensitive.

---

## What's Next?

* [Troubleshooting Common SSO Errors](/docs/faq/general-setup-faq-troubleshooting/common-sso-errors)

---

[duplicated]: /docs/get-started/coalesce-fundamentals/workspaces

---

## Setting Up a Snowflake Service Account for Coalesce

We recommend using a service account with key pair authentication in Snowflake to connect to your non-development Environments in Coalesce. This avoids linking Production or other shared Environments to an individual's account, which helps ensure your data pipeline's continuity and stability.

This guide walks you through the best practices for creating this service account.

## Create an RSA Public Key

We recommend authenticating the service account with key pair authentication. Authenticating with a username and password will be deprecated, and OAuth requires manual re-authorization that can interrupt your Production data pipelines. The following steps assume you've configured the service account for key pair authentication.

Follow the steps in the [Snowflake documentation][] for key pair authentication to generate your private and public keys. You'll need the public key in the next section.

## Provision the Service Account in Snowflake

Once you have the public key, you can assign it to the service account. The SQL code below creates a service account user and role. Feel free to edit the script to follow your organization's naming conventions.

```sql
-- Use a role with sufficient privileges to create roles and users (e.g., USERADMIN or SECURITYADMIN)
USE ROLE ACCOUNTADMIN; 

-- Define variables for easier customization
SET coalesce_role_name = 'COALESCE_SERVICE_ROLE';
SET coalesce_user_name = 'COALESCE_SERVICE_USER';
SET coalesce_warehouse_name = 'COALESCE_WH'; -- Replace with your chosen warehouse

-- Create the Coalesce role
CREATE ROLE IF NOT EXISTS IDENTIFIER($coalesce_role_name)
COMMENT = 'Dedicated role for Coalesce service account';

-- Create the Coalesce user with the public key
-- Replace '<YOUR_PUBLIC_KEY_STRING_HERE>' with the actual public key from rsa_key.pub (without headers/footers)
CREATE USER IF NOT EXISTS IDENTIFIER($coalesce_user_name)
RSA_PUBLIC_KEY = '<YOUR_PUBLIC_KEY_STRING_HERE>' 
DEFAULT_ROLE = IDENTIFIER($coalesce_role_name)
DEFAULT_WAREHOUSE = IDENTIFIER($coalesce_warehouse_name)
COMMENT = 'Service account user for Coalesce ETL/ELT operations';

-- Grant the COALESCE_ROLE to the COALESCE_USER
GRANT ROLE IDENTIFIER($coalesce_role_name) TO USER IDENTIFIER($coalesce_user_name);

-- Grant the COALESCE_ROLE to the SYSADMIN role (or another administrative role) 
-- This allows administrators to manage and troubleshoot the Coalesce setup.
GRANT ROLE IDENTIFIER($coalesce_role_name) TO ROLE SYSADMIN;
```

## Assign Permissions to the Service Account Role

After you create and assign the service account and role, you need to grant permissions. The role requires read permissions for all your source Storage Locations (the databases and schemas in Snowflake where your raw data is). It also needs read and write permissions for the target Storage Locations (the databases and schemas where you will write new objects).

```sql
-- Define variables for easier customization 
SET coalesce_role_name = 'COALESCE_SERVICE_ROLE'; 
SET coalesce_user_name = 'COALESCE_SERVICE_USER'; 
SET coalesce_warehouse_name = 'COALESCE_WH'; -- Replace with your chosen warehouse 
SET source_db_1 = 'RAW_DATA_DB'; -- Replace with your source database name 
SET source_schema_1 = 'SALES'; -- Replace with your source schema name 
SET dest_db_1 = 'ANALYTICS_DB'; -- Replace with your destination database name SET dest_schema_1 = 'REPORTING'; -- Replace with your destination schema name
-- ***Add additional Source/Target DBs/Schema variables as needed*** --

-- *** Step 2: Grant USAGE on Virtual Warehouse ***
USE ROLE ACCOUNTADMIN; -- Or a role that owns/manages warehouses

GRANT USAGE ON WAREHOUSE IDENTIFIER($coalesce_warehouse_name) TO ROLE IDENTIFIER($coalesce_role_name);

-- *** Step 3: Read Access to Source Databases/Schemas.  ***
-- Example for one source database/schema. Repeat for all relevant sources.
USE ROLE SYSADMIN; -- Or the owner role of the source DB

-- Source Database 1 (Repeat for all Sources)
GRANT USAGE ON DATABASE IDENTIFIER($source_db_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT USAGE ON SCHEMA IDENTIFIER($source_db_1).IDENTIFIER($source_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- Grant SELECT on existing tables and views
GRANT SELECT ON ALL TABLES IN SCHEMA IDENTIFIER($source_db_1).IDENTIFIER($source_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT SELECT ON ALL VIEWS IN SCHEMA IDENTIFIER($source_db_1).IDENTIFIER($source_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- Grant SELECT on FUTURE tables and views (CRUCIAL for new objects)
GRANT SELECT ON FUTURE TABLES IN SCHEMA IDENTIFIER($source_db_1).IDENTIFIER($source_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT SELECT ON FUTURE VIEWS IN SCHEMA IDENTIFIER($source_db_1).IDENTIFIER($source_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- *** Step 4: Read/Write Access to Destination Databases/Schemas ***
-- Example for one destination database/schema. Repeat for all relevant destinations.
USE ROLE SYSADMIN; -- Or the owner role of the destination DB

-- Destination Database 1 (Repeat for all Target DBs/Schemas).
GRANT USAGE ON DATABASE IDENTIFIER($dest_db_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT USAGE ON SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- Grants for creating objects
GRANT CREATE TABLE ON SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT CREATE VIEW ON SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT CREATE STAGE ON SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- DML on existing objects
GRANT SELECT, INSERT, UPDATE, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT SELECT ON ALL VIEWS IN SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- DML on FUTURE objects
GRANT SELECT, INSERT, UPDATE, DELETE, TRUNCATE ON FUTURE TABLES IN SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);
GRANT SELECT ON FUTURE VIEWS IN SCHEMA IDENTIFIER($dest_db_1).IDENTIFIER($dest_schema_1) TO ROLE IDENTIFIER($coalesce_role_name);

-- *** Step 5: Optional Global Privileges (Highly Recommended for Monitoring) ***
USE ROLE ACCOUNTADMIN; -- Global privileges require ACCOUNTADMIN

GRANT MONITOR EXECUTION ON ACCOUNT TO ROLE IDENTIFIER($coalesce_role_name);
GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE IDENTIFIER($coalesce_role_name);
```

## Configure the Service Account in Coalesce Environments

Once you have the service account username and private key, enter them into the relevant Environments in Coalesce. For detailed instructions, refer to our documentation on how to [configure Environment settings][].

## Optional: Key Pair Rotation

Although key pair keys don't expire, we recommend rotating them periodically for security. Follow [Snowflake's key rotation guide][] to update your keys according to your organization's policies.

[Snowflake documentation]: https://docs.snowflake.com/en/user-guide/key-pair-auth#configuring-key-pair-authentication
[configure Environment settings]: https://docs.coalesce.io/docs/setup-your-project/connection-guides/snowflake/snowflake-key-pair-authentication
[Snowflake's key rotation guide]: https://docs.snowflake.com/en/user-guide/key-pair-auth#configuring-key-pair-rotation

---

## Snowflake Username and Password

:::danger[Snowflake MFA Requirements]

Snowflake is moving towards MFA for all password based sign-ins. We recommend using OAuth for user login and Key pair for automation accounts. For the most recent information, see Snowflake's blog post: [Snowflake Will Block Single-Factor Password Authentication by November 2025][].

:::

Basic Auth is the fastest way to connect your target database platform to Coalesce, and the credential can be stored either in your local browser's storage (not recommended) or in Coalesce's cloud secrets vault.

## Get Your Snowflake Account URL

## Add Basic Auth to Snowflake

1. Navigate to **Build Settings > Environments or Workspaces**.
2. Select Edit on the environment or workspace that you wish to connect to Snowflake using Basic Auth.
3. In **Edit Environment or Workspace > User Credentials**, select **Authentication Type** as Username and Password (Cloud) or Username and Password (Browser Storage).
4. Enter Username and Password into their respective fields and Save.

[Snowflake Will Block Single-Factor Password Authentication by November 2025]: https://www.snowflake.com/en/blog/blocking-single-factor-password-authentification/

---

## Snowflake(Connection-guides)

## Placeholder

---

## Step 3: Create a Workspace

Before creating a Workspace, you should make sure you have a [Project][] created with version control. You can choose from username and password or OAuth.

A [Workspace][] is a sandbox environment where you can complete the development of your data, refine, and perform preliminary validation before merging them into the codebase. When creating a Workspace, you'll add your Storage Locations, Git URL, and connect to your data platform.

<BestPractices title="Workspace Best Practices">
- Create a Workspace strategy. Some approached you can take are:
  - **Use one Workspace per branch**. Each time you create a new branch, create a new Workspace and delete old ones.
  - **Use one Workspace per user**. Each user works in their own Workspace. They manage their branches from the Workspace.
  - **Create a separate Workspace for each feature branch**. This ensures that developers work within distinct locations, preventing conflicts and streamlining the development process.
- Decide how you want to connect your Workspace to a Data Platform. Review below.
- When feature development in a specific Workspace is complete, it should be merged into the main branch and checked out by the Main Workspace when preparing for deployment into higher Environments.
- It is best practice to deploy to higher (Production, UAT, Testing) Environments from the main Git branch in the Main Workspace.
</BestPractices>

:::info
Before you begin, make sure you have administrator access to your chosen platform and the necessary permissions to create authentication credentials.

Just need a quick reference for authentication? Look at the [Connection guides][] for your data platform.
:::

## Choose Your Platform

Select the platform you want to connect to:

<Tabs groupId="platform" queryString="platform">

<TabItem value="bigquery" label="BigQuery (Private Preview)">

An admin or a user with the ability to assign permissions is needed for BigQuery.

**Set Up Project Level Permissions.**

1. Make sure you're in the project you want to add the service account to.
2. Go to **IAM & Admin > Service Accounts** in Google Cloud.

    

3. Click **Create service account**.

    

4. Fill out the required **Service account ID** and any other information required by your company's policy. Click **Create and Continue**.
5. Search for the role **BigQuery User**. Click **Continue**.

    

6. You can skip **Principals with access**. Click **Done**. Move on to setting data set level permissions.

**Set Up Dataset Level Permissions.**

1. Go to **BigQuery Studio**.
2. In the Explorer, find the data set you want to add to Coalesce.

    

3. In the data set, click **Share > Manage Permissions**.

    

4. Click **Add principal**.
5. Search for your service account name. It should look similar to `name@projectname.iam.gserviceaccount.com`.
6. Then assign roles as **BigQuery Data Editor**.

    

7. Go to **IAM & Admin > Service Accounts** in Google Cloud.
    

8. Click on the service account created for Coalesce.
9. Click **Keys**.
10. Then **Add key > Create new key**.

    

11. Download the JSON file.

**Create Your Workspace.**

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).
5. Click **Next**.

    

6. **Add your BigQuery Credentials**
7. Upload the JSON file from Google Cloud IAM. Then click **Test Connection**.

    

8. Click **Next**.
9. **Set Up Version Control**
10. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
11. Then give the new branch a name.
12. Click **Next**.

    

13. **Storage Location and Storage Mappings**
14. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

15. Click **Create Workspace**.

</TabItem>

<TabItem value="databricks" label="Databricks">

Before starting with Databricks authentication, make sure the following is in place:

* Your workspace must be using the [Unity catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspaces).
* Must be a [Databricks account admin](https://docs.databricks.com/aws/en/admin/#account-admins) create to OAuth applications.
* Databricks users must be given the permission to create a Personal Access Token (PAT) for Token authentication.

<Tabs groupId="databricks-auth" queryString="db-auth">
<TabItem value="token" label="Token Authentication">

:::info[OAuth]

We recommend using OAuth for authentication.

:::

Tokens or [Personal Access Tokens](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-workspace-users) (PAT) are used in Databricks to authenticate at the workspace level.

In this section, you get your access token from Databricks and then add it to Coalesce as part of creating a Workspace.

**Get Your Databricks Token.**

1. Log into your Databricks account.
2. Go to your username at the top and select **Settings**.
3. Under User, select **Developer**.
4. Next to Access Tokens, click **Manage**.
5. Click **Generate new token**.
6. Write a description for the token and choose the lifetime.
7. Make sure to save the access token somewhere secure. It can’t be recovered or shown again.

<figure>
  
  <figcaption>Databricks Access Token Page</figcaption>
</figure>

:::info[ Databricks PAT]

Read the Databricks documentation on [Databricks personal access token authentication][] to learn more about managing your access tokens in Databricks.

:::

**Create Your Workspace.**

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).
5. Click **Next**.

    

6. **Connect Databricks to this Workspace**
7. Enter your Databricks Account URL.
8. Click **Next**.

    

9. **Add your Databricks Credentials**,
10. Enter the token and then click **Retrieve Warehouses**. Select the one you want to use.
11. Click **Next**.

    

12. **Set Up Version Control**
13. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
14. Then give the new branch a name.
15. Click **Next**.

    

16. **Storage Location and Storage Mappings**
17. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click **Create Workspace**.

</TabItem>
<TabItem value="oauth-m2m" label="OAuth Machine-to-Machine (M2M)">

Machine-to-Machine is used for automation. You'll need to create a Service Principal.

1. On the Databricks Account Console, go to **User Management** and select **Service Principals**.

    <figure>
      
      <figcaption>Databricks User management</figcaption>
    </figure>

2. After giving the service principal a name, you'll grant the right permissions, then create the client ID and client secret.
   1. The service principal should have Workspace, Warehouse, and Catalog permissions.
3. Click on the service principal and go to the Permissions tab and click **Grant Access**. Add the service principal you just created as a User.

    <figure>
      
      <figcaption>Add the new service principal as user</figcaption>
    </figure>

4. Go to the Databricks **Catalog** and select the catalog the service principal should have access to.
5. Click **Grant**.

    <figure>
      
      <figcaption>Click Grant on the catalog</figcaption>
    </figure>

6. Choose the service principal and then select **Data Editor**.

    <figure>
      
      <figcaption>Grant the Data Editor Role</figcaption>
    </figure>

7. Then go to **SQL Warehouses**, and select the warehouse the service principal should have access to.

    <figure>
      
      <figcaption>SQL Warehouse Permissions</figcaption>
    </figure>

8. Click **Permissions**, and select the service principal from the list. Set it to **Can Use**.

    <figure>
      
      <figcaption>Permission Level is Can Use</figcaption>
    </figure>

    :::warning[Service Principal Missing From List]

    If your service principal is missing from the list, go to your user **Settings > Identity and access > Service principals**, and add it there too. Then go back to the SQL Warehouse step.

    :::

9. Go to the **Account console** in Databricks.
10. Select **User management > Service principals**.
11. Click the one you created, then click **Generate secret**.

    <figure>
      
      <figcaption>Generate Your Client ID and Client Secret</figcaption>
    </figure>

12. Choose how long the token should be valid for, then click **Generate**.
13. Save the Client ID and Client Secret for configuring in Coalesce.

**Create Your Workspace.**

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).
5. Click **Next**.

    

6. **Connect Databricks to this Workspace**
7. Enter your Databricks Account URL.
8. Click **Next**.

    

9. **Add your Databricks Credentials**.
10. Change the **Authentication Type** to **Machine-to-Machine**.
11. Enter the **Client ID** and **Client Secret**, then click **Retrieve Warehouses**. Select the one you want to use.
12. Click **Next**.

    

13. **Set Up Version Control**
14. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
15. Then give the new branch a name.
16. Click **Next**.

    

17. **Storage Location and Storage Mappings**
18. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

19. Click **Create Workspace**.

</TabItem>

<TabItem value="oauth-u2m" label="OAuth User-to-Machine (U2M)">

User-to-Machine is used for your daily work access. If you want to set up any automation, use Machine-to-Machine. You will need to create a App Connection and get the client ID and client secret along with the URL and path.

1. Go to the [Account console][]. If you don't have access, you'll need to reach out to your Databricks administrator.
2. Then go to **Settings > App Connections** and click **Add connection**.

    <figure>
      
      <figcaption>Databricks - Settings > App connections</figcaption>
    </figure>

3. Give the app a **Name**:
   1. **Redirect URL** is your Coalesce account URL at `oauthredirect`. This is usually the URL you use to log into Coalesce. For example, `https://<COALESCE_APP_DOMAIN>/oauthredirect`, `https://app.coalesce.io/oauthredirect`.

        <figure>
        
        <figcaption>Databricks - Add Connection</figcaption>
        </figure>

   2. Change the **Access Scopes** to All APIs.
   3. **Generate client secret** is checked.
   4. You can leave TTL as the default or change it.
4. Click **Add**.
5. A Client ID and Client Secret will pop-up. Save them for later.

**Create Your Workspace.**

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).
5. Click **Next**.

    

6. **Connect Databricks to this Workspace**
7. Enter your Databricks Account URL.
8. Toggle **Databricks OAuth (User-to-Machine)** on. This will open a window where you can enter the Client ID and Secret.
9. Coalesce will verify the connect and then you can click **Next** to move to the next step.

    

10. **Add your Databricks Credentials**
11. Select the **SQL Warehouse** you want to use, then click **Next**.

    

12. **Set Up Version Control**
13. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
14. Then give the new branch a name.
15. Click **Next**.

    

16. **Storage Location and Storage Mappings**
17. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click **Create Workspace**.

</TabItem>

</Tabs>

</TabItem>
<TabItem value="snowflake" label="Snowflake">

Before you begin, you'll need your Snowflake account URL.

---

<Tabs groupId="snowflake-auth" queryString="sf-auth">
<TabItem value="sf-oauth" label="OAuth">

:::warning[Permissions Requirement]
Only a Snowflake `ACCOUNTADMIN` role or a role with the global `CREATE INTEGRATION` privilege can create security integrations.
:::

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).

    
  
5. **Connect Snowflake to this Workspace**
6. Enter your Snowflake Account URL.
7. Then enable **Snowflake OAuth**.
8. A popup will include the code needed to create the Client ID and Client Secret. First run `Create Security Integration`, then run `Fetch ClientID and ClientSecret`.

    <figure>
      
      <figcaption>The OAUTH_REDIRECT_URI will match your Coalesce app domain.</figcaption>
    </figure>

9. The next screen will ask you to enter your Role (`SYSADMIN`) and Warehouse if they are required. Click **Authenticate**.
10. You'll be asked to sign in to your Snowflake instance.
11. Coalesce will confirm the connection, then click **Next**.

    

12. **Set Up Version Control**
13. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
14. Then give the new branch a name.
15. Click **Next**.

    

16. **Storage Location and Storage Mappings**
17. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click **Create Workspace**.

</TabItem>

<TabItem value="key-pair" label="Key Pair Authentication">

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).

    

5. **Connect Snowflake to this Workspace**
6. Enter your Snowflake Account URL.
7. Click **Next**.

    

8. **Add your Snowflake Credentials**
9. Choose **Key Pair** as the Authentication Type.
   1. Go through Snowflake’s [key pair authentication][] steps to generate your keys.
   2. Make sure to generate a private key and save it for use in Coalesce.
   3. Generate your public key using the private key created for Coalesce.
   4. Assign the public key to your Snowflake user.
10. Add your Private Key and Private Key Passphrase.
11. Click **Test Connection**.

    :::info[Adding Your Private Key]

    When entering your private key, make sure it's formatted properly. It must include the full private key including the lines BEGIN ENCRYPTED PRIVATE KEY and END ENCRYPTED PRIVATE KEY.

    ```text
    -----BEGIN ENCRYPTED PRIVATE KEY-----
    ...
    -----END ENCRYPTED PRIVATE KEY-----
    ```

    :::

12. After authentication is successful, click **Next**.

    

13. **Set Up Version Control**
14. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
15. Then give the new branch a name.
16. Click **Next**.

    

17. **Storage Location and Storage Mappings**
18. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

19. Click **Create Workspace**.

</TabItem>

<TabItem value="sf-username-password" label="Username and Password (Not recommended)">

1. Select the Project you want to create the Workspace in.
2. Click **Create Workspace**.
3. **Workspace Details**
4. Give your Workspace a Name and Description(optional).

    
  
5. **Connect Snowflake to this Workspace**
6. Enter your Snowflake Account URL.
7. Click **Next**.

    

8. **Add your Snowflake Credentials**
9. Enter your Snowflake username and password.
10. Enter the Role and Warehouse.
11. After authentication is successful, click **Next**.

    

12. **Set Up Version Control**
13. Workspaces let you work on a branch. This is the version control repo you created in [Setup Version Control][] and added to your [Project][]. You will need to select a branch and commit to make a new branch. For example, if you want to create a branch off main, select main, then select the commit in main to create your branch from.
14. Then give the new branch a name.
15. Click **Next**.

    

16. **Storage Location and Storage Mappings**
17. Add [Storage Locations and Storage Mappings][] to your Workspace. If you have any existing Storage Locations and Storage Mappings from your repo, they will be listed here. You can delete the defaults and add new Storage Locations.

    

18. Click **Create Workspace**.

</TabItem>

</Tabs>

</TabItem>
</Tabs>

---

## What's Next?

Review the Coalesce Fundamentals to learn more about:

* [**Projects**][]
* [**Workspaces**][]
* [**Add Your Data**][]

---

[Project]: /docs/setup-your-project/create-your-project
[Workspace]: /docs/get-started/coalesce-fundamentals/workspaces
[**Projects**]: /docs/setup-your-project/create-your-project
[**Workspaces**]: /docs/get-started/coalesce-fundamentals/workspaces
[Storage Locations and Storage Mappings]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[**Add Your Data**]:/docs/setup-your-project/add-a-data-source
[Connection Guides]: /docs/category/connection-guides
[Setup Version Control]: /docs/setup-your-project/setup-version-control
[key pair authentication]: https://docs.snowflake.com/en/user-guide/key-pair-auth
[Account console]: https://accounts.cloud.databricks.com/settings/app-integrations
[Databricks personal access token authentication]: https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-workspace-users

---

## Step 7: Create Your Environments

This guide goes over configuring [Environments][] for deploy and refresh. Environments let you push your changes to different locations such as production, testing, and QA.

Each Environment can have it's own Storage Location, Storage Mappings, and connection settings.

<BestPractices title="Environment Best Practices">
- Each environment should have its own Database and Schema. As a feature flows from a main development Workspace to Test, QA, and finally to the Prod environment, the storage mappings should point to the locations where you want to create the objects in question in your pipeline.
- A good idea is to create environments for DEV, QA, and Production matching the Storage Mappings.
- You'll have to connect each environment to your [data platform][].
</BestPractices>

## Create Your Environment and Authenticate

1. In your Workspace, go to **Build Settings > Environments**.

    

2. Give the Environment a descriptive name.
3. You'll need to authenticate your Environment, even if another Workspace or Environment has been authenticated.

    1. On the **Setting** page, enter your account URL.

    

4. Decide how you want to authenticate your Environment. You can use:
   1. Username and Password
   2. OAuth
   3. Key Pair Authentication

5. Review the authentication method for your [data platform][] and choose the best method for you. For Snowflake non-development Environments, follow [Setting Up a Snowflake Service Account for Coalesce][] and configure key pair authentication.
6. Go to **User Credentials** and enter your credentials. The steps below use username and password as an example. For Snowflake QA, staging, and production Environments, enter the service account username and private key instead. Optionally enter the role and warehouse. Click **Save**.

## Add Storage Mappings

The Storage Mapping is the database and schema in your data provider you want to deploy to.

1. While in **Environment Settings**, go to **Storage Mappings**.
2. Select the database and schema.

### Override Mapping Values

To input database and schema objects manually, instead of choosing them from the dropdown menu, activate the **Override Mapping Values** option. This step may be necessary when mapping to a database or schema for which you lack access permissions.

## Parameters

[Parameters][] act as read-only environment variables. Parameters are defined as a JSON blob set by the user that can be accessed in the metadata during template rendering.

1. Click on **Parameters** and add your Parameters in JSON format.
2. Make sure to click **Save** to save your new parameters.

---

## What's Next?

Review the Coalesce Fundamentals to learn more about:

- [**Parameters**][]
- [**Environments**][]
- [**Storage Location and Storage Mappings**][]
- [**Learn about Deploy and Refresh**][]

---

[data platform]: /docs/data-platforms
[**Environments**]: /docs/get-started/coalesce-fundamentals/environments
[Environments]: /docs/get-started/coalesce-fundamentals/environments
[Parameters]: /docs/deploy-and-refresh/parameters
[**Parameters**]: /docs/deploy-and-refresh/parameters
[**Storage Location and Storage Mappings**]: /docs/get-started/coalesce-fundamentals/storage-locations-and-storage-mappings
[**Learn about Deploy and Refresh**]: /docs/get-started/coalesce-fundamentals/understanding-deploy-and-refresh
[Setting Up a Snowflake Service Account for Coalesce]: /docs/setup-your-project/connection-guides/snowflake/snowflake-service-accounts

---

## Step 2: Create a Project

This guide goes over the steps to create a new Project in Coalesce.

<BestPractices title="Project Best Practices">
* Each Project should only use one Git Repo.
* A [Project](/docs/get-started/coalesce-fundamentals/projects) is similar to a folder on your computer that helps organize your work. Projects can be organized by purpose, department, or business area.
* Each user will need their own provider account and create a personal access token for Coalesce.
* Each user will need to be in the organization's account.
* It's recommended to create a new repository for Coalesce.
</BestPractices>

## Create a New Project

---

## What's Next?

* [**Coalesce Git Requirements**][] - Review the requirements to using version control in Coalesce.
* [**Set Up Version Control**][] - Learn how to set up version control.
* [**Projects**][] - Learn more about projects in Coalesce fundamentals.

---

[**Set Up Version Control**]: /docs/setup-your-project/setup-version-control
[**Coalesce Git Requirements**]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements
[**Projects**]: /docs/get-started/coalesce-fundamentals/projects

---

## Setup and Configuration

For a single checklist that walks you from setup through deployment, see the [Transform Onboarding Guide][]. Before you begin, confirm you meet the [prerequisites][] (cloud warehouse access, Git repo, SQL basics, admin access). Then work through the setup requirements and step-by-step guides below.

<DocCardList />

---

[Transform Onboarding Guide]: /docs/get-started/transform-onboarding-guide
[prerequisites]: /docs/get-started/transform-onboarding-guide#prerequisites-checklist

---

## Administrative Tasks

There are several setup tasks that may need to be completed by an administrator for your Coalesce account to be configured to work seamlessly within your larger data and technology infrastructure.

## Authentication

Coalesce supports several authentication methods which can be mixed and matched according to your requirements and use cases. Review [Authentication Methods][].

### Snowflake

- **Connecting to Snowflake:** If your organization uses a firewall or network policies in Snowflake to limit connectivity, you will have to allow connectivity from the Coalesce Services to Snowflake in order for your users to be able to connect to your Snowflake account within Coalesce. Review [Allow Inbound Traffic from Coalesce][].
- **Snowflake Authentication**: Coalesce authenticates users with the warehouse using Snowflake OAuth. The `ACCOUNTADMIN` role is required to set up this security integration, which is a highly elevated role in Snowflake. Please view our page on [Snowflake OAuth][] for more details. Alternatively, password-based authentication to the warehouse is supported.

### Databricks

- **Connecting to Databricks:** If your organization uses a firewall or network policies in Databricks to limit connectivity, you will have to allow connectivity from the Coalesce Services to connect their accounts within Coalesce. Review [Allow Inbound Traffic from Coalesce][].
- **Databricks Authentication**: Coalesce authenticates users with the warehouse using Token, Machine to Machine, or User to Machine. Take a look at our [Databricks Connection Guide][]. You must be a Databricks account admin.

### Single Sign-on To Coalesce

Optionally, Coalesce can authenticate users to the platform using Single Sign-On. Please view our page on [Single Sign-On][] for more details. Alternatively, password-based authentication to Coalesce is supported.

## Git Integration

Git integration is a core component of the Coalesce platform and must be configured, with access provided to each of your Coalesce developers, to deploy your data projects to non-development environments. There are a few tasks you should complete as an administrator. Review [Git Integration][].

- **Create and provision access to your Git repositories for Coalesce**. This task will need to be completed by a Git administrator in your organization. Each Project in Coalesce should have its own repository, and the repository should not be shared with any work outside of Coalesce.
- **Allow network connectivity between Coalesce and your Git provider and repository.** If your organization uses network policies to limit traffic to and from your Git provider or specific repositories on your Git provider, a Git administrator in your organization will need to modify your network policies to allow connectivity between Coalesce and your Git provider. The list of IPs that need to be allowed are in Network Connectivity.
- **Point each Coalesce Project to the Git repository created for it**. Each Coalesce Project must have a Git repository configured for full Project functionality. The Project repository is designated in the [Project settings][].
- Each developer will need read and write permissions for each Git repository they are working in.

## Command Line Interface

To use Coalesce's command line interface (`coa`) to complete deployments or refreshes, it will need to be installed on the machine or service where it will be used.

- Review [CLI Overview][].

## Outbound Connectivity

If your organization limits outbound traffic to the public internet, you will need to allow outbound HTTPS connectivity on your network to several domains in order to use the Coalesce App, API, or CLI.

- Review the [Allow Outbound Traffic to Coalesce][]

## Private Link

These requirements assume the traditional Coalesce connectivity approach is being leveraged. If you are using Coalesce with AWS PrivateLink or Azure Private Link, these requirements may not be required. Interested in learning more about our AWS PrivateLink or Azure Private Link capabilities? [Contact Us][] for more information.

## Browser

We currently only support Google Chrome.

[Authentication Methods]: /docs/organization-and-accounts/organization-management/
[Single Sign-On]: /docs/organization-and-accounts/organization-management/single-sign-on/
[Git Integration]: /docs/git-integration/set-up-your-git-integration
[Project settings]: /docs/setup-your-project/create-your-project
[CLI Overview]: /docs/coa/
[Allow Inbound Traffic from Coalesce]: /docs/setup-your-project/setup-requirements/network-requirements
[Snowflake OAuth]: /docs/setup-your-project/connection-guides/snowflake/snowflake-oauth
[Allow Outbound Traffic to Coalesce]: /docs/setup-your-project/setup-requirements/network-requirements
[Contact Us]: mailto:support@coalesce.io
[Databricks Connection Guide]:/docs/setup-your-project/connection-guides/databricks

---

## Coalesce Git Requirements

Coalesce requires Git to be used for storage of your code used within the Coalesce application. Most organizations will utilize a major publicly available SaaS provider for their needs. Coalesce actively supports four leading SaaS providers:

- [GitHub][]
- [Bitbucket][]
- [GitLab][]
- [Azure DevOps Repos][]

If IP whitelisting is already in use by your organization, please add the Coalesce IP addresses to allow our application access:

- [GitHub - Adding an Allowed IP Address][]
- [Bitbucket - Allowlisting IP Addresses][]
- [GitLab - IP Address Restrictions][]
- [Azure DevOps - Application Access Policies][]

If your SaaS provider is not listed above, or if you host your own Git instance within your network, you still may be able to use it within Coalesce if it meets the necessary requirements:

- Accessible over the internet
  - Access can be limited to the [static IP addresses][] for your Coalesce cloud/region  
  - Accessible over HTTPS / TCP port 443. (SSH access is not supported)
- Supports authentication using a personal access token (PAT)

If your self-hosted Git instance is not already publicly available, or if IP whitelisting is in use, your security or network teams should be consulted on how to approach this requirement as steps to do so will be unique based on your Git implementation, network topology and security posture.

A list of our IP addresses for the cloud/region of Coalesce in use by your organization can be found on our [Network Requirements][] page.  

[GitHub]: https://github.com/
[Bitbucket]: https://bitbucket.org/
[GitLab]: https://gitlab.com/
[Azure DevOps Repos]: https://dev.azure.com/
[Github - Adding an Allowed IP Address]:
https://docs.github.com/en/enterprise-cloud@latest/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization#adding-an-allowed-ip-address
[Bitbucket - Allowlisting IP Addresses]: https://support.atlassian.com/bitbucket-cloud/docs/control-access-to-private-content-in-a-workspace/#Allowlisting-IP-addresses
[Gitlab - IP Address Restrictions]: https://docs.gitlab.com/ee/administration/reporting/ip_addr_restrictions.html
[Azure DevOps - Application Access Policies]: https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/change-application-access-policies
[static IP addresses]: https://docs.coalesce.io/docs/setup-your-project/setup-requirements/network-requirements/
[Network Requirements]:https://docs.coalesce.io/docs/setup-your-project/setup-requirements/network-requirements/

---

## Network Requirements

<!-- markdownlint-disable -->
<!-- vale off -->

:::info

You need to allow inbound and outbound traffic from Coalesce.

:::

## Allow Inbound Traffic from Coalesce

When using Coalesce, we will connect to your Data Platform from the following IP addresses. Be sure to allow traffic from all IPs in the respective Coalesce region by locating your domain (URL) below.

### Network Table

<NetworkTable spreadsheetId="1qzoNv9CQw4EiU6jKvPBHCTzGLoocF9Z84u0qeExI2_w" />

## Data Platform Setup

<Tabs>
  <TabItem value="databricks" label="Databricks" default>

    When connecting Coalesce to Databricks, both inbound and outbound access must be configured so Coalesce can authenticate and exchange data securely.

    Databricks allows you to manage connectivity through several policy layers. The following apply when using Coalesce:

    * **Serverless Egress Policies** - Allow outbound access from [Databricks serverless workloads][].
    * **IP Access Lists** - Optionally configure [workspace-level IP access lists][] to explicitly allow connections from Coalesce. This adds another layer of ingress protection.
    * **Private Connectivity (Optional)** - If your Databricks workspace uses PrivateLink or VNet injection, ensure Coalesce IPs and domains are reachable through your private endpoint configuration.

  </TabItem>
  <TabItem value="Snowflake" label="Snowflake">

    In Snowflake, IP address policies can be assigned at the Account, Security Integration, and User levels, with a defined precedence. If a policy is set at the Account level, it applies to all users by default. However, a User-specific policy will override the Account policy, applying only to the designated user. Authentication methods like Username/Password and OAuth are subject to User-defined policies, while Key Pair authentication is user-specific, as it involves setting a public key directly for a Snowflake user. Review [Snowflake Network policy precedence][].

    Snowflake recommends using [Network Rules][] to update network information.

    :::warning[Updating Network Policies]

    Running `ALTER ACCOUNT SET NETWORK_POLICY = ‘<COALESCE_NETWORK_POLICY>` will overwrite any existing network policies, which can lead to other users losing Snowflake access.

    If there are current network policies, you should add the Coalesce IPs in the ALLOWED_IP_LIST to the existing policy, instead of replacing the entire policy.

    :::
  </TabItem>
</Tabs>

---

## Allow Outbound Traffic to Coalesce

It's required to allow outbound HTTPS connectivity on your network to the following domains in order to connect to Coalesce GUI, API, and/or CLI.

```txt
https://firestore.googleapis.com
https://firebasestorage.googleapis.com/
https://identitytoolkit.googleapis.com/
https://securetoken.googleapis.com/
https://storage.coalescesoftware.io/
https://app.coalescesoftware.io
https://*.app.coalescesoftware.io
https://app.eu.coalescesoftware.io/
https://*.app.eu.coalescesoftware.io/
https://app.australia-southeast1.gcp.coalescesoftware.io/
https://*.app.australia-southeast1.gcp.coalescesoftware.io/
https://app.us-east-1.aws.coalescesoftware.io/
https://*.app.us-east-1.aws.coalescesoftware.io/
https://app.us-west-2.aws.coalescesoftware.io/
https://*.app.us-west-2.aws.coalescesoftware.io/
```

| Name | Description|
|---|---|
| `https://firestore.googleapis.com` | Database that holds all metadata.|
| `https://firebasestorage.googleapis.com` | Send deployment metadata from the client about deployments running in the cloud environment.|
|  `https://identitytoolkit.googleapis.com` | Authentication mechanism containing all users, providing the user with a JWT for OAuth.|
| `https://securetoken.googleapis.com` |Allows Coalesce to exchange the Access Token to an OAuth JWT.|
| `https://storage.coalescesoftware.io` | CDN hosting for all static assets such as images and JavaScript.|
| `https://app.coalescesoftware.io` | The web application. |
| `https://*.app.coalescesoftware.io` |An alias for the web application to allow SSO. Such as redirecting to a specific organization.|

:::info[In-Network Command-Line Interface]

The Coalesce Command-Line Interface may be used to deploy and run Jobs in an Environment. This omits the requirement to allow list Coalesce IP addresses. These IPs are still used in the Coalesce app.

:::

---

## What's Next?

* If users still see a blank or spinning screen after login, see [Troubleshooting Browser Login Issues][]. 

---

[inbound]: /docs/setup-your-project/setup-requirements/network-requirements
[outbound]: /docs/setup-your-project/setup-requirements/network-requirements
[Network Rules]: https://docs.snowflake.com/en/user-guide/network-rules
[Snowflake Network policy precedence]: https://docs.snowflake.com/en/user-guide/network-policies#network-policy-precedence
[Troubleshooting Browser Login Issues]: /docs/faq/general-setup-faq-troubleshooting/browser-login-issues
[workspace-level IP access lists]: https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list-workspace
[Databricks serverless workloads]: https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies

---

## Private Connectivity Services

Coalesce supports private connectivity services on AWS, Azure, and Google Cloud Platform.

:::info[Coalesce does not offer private connectivity services]

These are services that Coalesce supports for you to use with your own cloud accounts.

:::

## What Is AWS PrivateLink?

AWS PrivateLink is a networking service provided by Amazon Web Services (AWS) that allows you to privately connect your AWS services and VPC endpoint services to other services within the AWS network. This connection is made without exposing your traffic to the public internet, which enhances security by reducing the exposure to threats such as brute force and DDoS attacks.

## What Is Azure PrivateLink?

Azure PrivateLink is a networking service offered by Microsoft Azure that enables private connectivity from Azure Virtual Networks (VNet) to Azure services, customer-owned services, and services provided by Microsoft partners. This service ensures that data transmitted between your Azure services and Virtual Networks never traverses the public internet.

## What Is GCP Private Service Connect?

Google Cloud Platform's Private Service Connect (PSC) is a networking service that enables private connectivity between your VPC networks and Google services or third-party services. It allows you to access services using internal IP addresses rather than public IPs, keeping all traffic within Google's network backbone. Private Service Connect eliminates the need for internet traversal, improving security and potentially reducing latency while simplifying network management.

---

## System Requirements

:::warning[Desktops Can Vary]

- You might need more or less RAM, CPU, etc.
- Running other software could change the requirements.
- You might need to reconfigure your Virtual Machine if you experience performance issues.

:::

## Desktop System Requirements

### Minimum System Requirements

If you are running 100 Nodes with 50 Columns.

- **Browser support**: Chrome 110+ (3D acceleration/support turned off in Chrome)
- **OS**:
  - Windows Server 2016, 2019, 2022
  - Windows 11
  - MacOS Catalina 10.15+
  - 64-bit Ubuntu 18.04+
  - Debian 10+
  - openSUSE 15.5+
  - Fedora 38+
- **RAM**: 4 GB RAM
- **CPU**:
  - i5 5th Generation+ (2 Cores (4 Threads), 2.20GHz)  
  - Any Apple Silicon preferred
- **HDD/SSD**: 150 GB+ HDD (Windows and Chrome overhead plus memory swap space)

### Recommended System Requirements​

If you have are running high volume data pipelines such as 3000 Nodes with 100 Columns.

- Browser support: Chrome 120+ (If the hardware is not capable of 3D acceleration in Chrome, 3D acceleration should be turned off)
- **OS**:
  - Windows Server 2016, 2019, 2022
  - Windows 11
  - MacOS Catalina 10.15+
  - 64-bit Ubuntu 22.04+
  - Debian 12+
  - openSUSE 15.5+
  - Fedora 39+
- **RAM**: 16 GB RAM
- **CPU**:
  - i5 8th Generation+ (6 Cores (12 Threads), 4GHz)
  - Any Apple Silicon preferred
- **HDD/SSD**: 150 GB+ SSD (Windows and Chrome overhead plus memory swap space)

## CLI System Requirements
<!-- vale off -->

<!-- vale on -->

---

## User Tasks

As a Coalesce user, there are a few tasks you’ll need to complete before starting.

## Generate Your Fine-Grained Access Token

Connectivity between Coalesce and Git is authenticated using a fine-grained access token. Each individual developer needs to generate and use an individual access token associated with their individual account. See [Set Up Version Control (Git)][] for instructions on your Git provider.

## Configure your Git Account In the User Settings

For each Git Provider or fine-grained access token you want to use Coalesce, you will need to configure a Git account in your [User Menu][] in the Coalesce UI.

## Associate your Git Account to Coalesce Projects

When you begin working with a Project in Coalesce, you will be prompted to designate which of your configured Git Accounts should be associated with the Project. Select the account associated with the access token that provides you with write access to the Git repository that has been provisioned for the Project.

## Browser

We currently only support Google Chrome.

[Set Up Version Control (Git)]: /docs/setup-your-project/setup-version-control
[User Menu]: /docs/organization-and-accounts/user-menu-overview/

---

## Step: 1 Setup Version Control

When setting up a project in Coalesce, you'll need your repository URL and access token. This guide will walk you through:

- Creating a repo and getting your URL
- Creating your access token
- Adding your version control account to Coalesce by creating a Project.

:::info[Quick Instructions]

If you have experience with version control and want a quick overview, take a look at [Set Up Version Control][].

:::

## Before You Begin

- Review [Coalesce Git Requirements][] before starting.

Coalesce supports the following Git providers:

- [GitHub][]
- [Azure DevOps][]
- [Bitbucket][]
- [GitLab][]

:::info[Video Guides]

Each section includes a video walkthrough of all the steps.

:::

## GitHub

GitHub is a web-based platform that provides version control and collaborative software development services using Git. It allows developers to store, manage, and track changes to their code while facilitating collaboration through features like pull requests, issue tracking, and project management tools.

### Create Your GitHub Repo

1. Go to [https://github.com/][].
2. Click the green **New** button, to create a new repository. It can be found on the homepage or on your profile page under Repositories.

    

3. On the **Create a new repository page**, choose the Owner, give the repo a name. The name will become part of your URL.
4. Optionally add a description.
5. Choose if it should be Public or Private. If you're an GitHub Enterprise customer, you'll have the option for Internal.
6. Click Create Repository. You shouldn't add a README, use a template, or add a `.gitignore`.
7. Once created, you'll see your Git URL. For example `https://github.com/Coalesce-Software-Inc/data-transformations.git`. Select the HTTPS version to use in Coalesce.

:::warning[`.gitignore`]

Don't add a `.gitignore` to your repo when creating it. It should be completely empty of all files. This is so Coalesce can initiate the repo correctly. After the repo is initiated, you can create a `.gitignore`.

These file paths are protected paths and files and should NOT be in the `.gitignore` and should not be used for anything else.

- data.yml
- locations.yml
- nodes/
- subgraphs/
- packages/
- macros/
- nodeTypes/
- environments/
- jobs/

:::

<Wistia id="9f3mpa4vkq" subtitle="How to create a new repo in GitHub."/>

### Get Your GitHub Access Token

1. Click on your GitHub profile and go to **Settings > Developer Settings**.
2. Click **Personal access tokens > Fine-grained tokens**.

    

3. Give the token a name and description.
4. Set the following based on your company policy.
   1. [Resource owner][]
   2. Expiration date
   3. Repository access should be either **All repositories** or **Only select repositories**.
5. Open **Permissions > Repository permissions**.
6. Set **Contents** to **Read and write**. GitHub will also automatically grant Metadata, which is required.

    

7. Save your access token somewhere secure. You won't be able to recover it.

<Wistia id="brwokqiaik" subtitle="How to create your access token in GitHub"/>

### Add Your GitHub Token and Repo to Coalesce

Review the steps in [Add your URL and Token to Coalesce][].

## Azure DevOps

Azure DevOps Git is Microsoft's enterprise-grade Git repository service that provides source code management with branch policies, code reviews, and integrated CI/CD pipelines.

### Create Your Azure DevOps Repo and Token

In this step, you'll create your repo and get your URL.

1. Log in to your organization, for example, `https://dev.azure.com/<Your_Organization>`.
2. If needed, create a new Project. Give it a name and description.

    

3. Then click on **Repos**.
4. You'll see your **HTTPS** repo URL. Save this for use in Coalesce.
5. You can generate your Git Credentials on this page, but make sure to update the permissions as listed starting in **Step 7**.
6. Scroll down to Initialize main branch with a Read or `gitignore` and **uncheck the box**.

     tjperry07 > Coalesce Demo > Repos > Files.' className="mdImages" width="65%"/>

7. Go to **User settings > Personal access tokens**.
8. If you have an existing token, click **Edit** on the token you want to use in Coalesce. If there are no tokens, click **New Token**.
9. Set the expiration date based on company policy.
10. Change the Scopes to Custom defined.
11. Select the following:
    1. Work Items: Read, write, & manage
    2. Code: Read, write, & manage
    3. Packaging: Read, write, & manage
12. Click Create or Save.

:::warning[`.gitignore`]

Don't add a `.gitignore` to your repo when creating it. It should be completely empty of all files. This is so Coalesce can initiate the repo correctly. After the repo is initiated, you can create a `.gitignore`.

These file paths are protected paths and files and should NOT be in the `.gitignore` and should not be used for anything else.

- data.yml
- locations.yml
- nodes/
- subgraphs/
- packages/
- macros/
- nodeTypes/
- environments/
- jobs/

:::

<Wistia id="3v20u5p0ls" subtitle="How to create a project and add create your PAT."/>

### Add Your Azure Token and Repo to Coalesce

Review the steps in [Add your URL and Token to Coalesce][].

## Bitbucket

Bitbucket is Atlassian's Git-based code hosting and collaboration platform. It offers both cloud and self-hosted options with features for code review and CI/CD pipelines.

### Create Your Bitbucket Repo and URL

1. From the **Workspace Overview**, click **Create** and select **Repository**.

    

2. On the **Create a new repository** page, fill out the repo details.
   1. **Include a README?** is No.
   2. **Include .gitignore?** is No.

    

3. On the Repo page, you can find the URL in the **HTTPS** box or under **Step 2: Connect your existing repository to Bitbucket**.

    

<Wistia id="ljmcynn3oy" subtitle="Create your repo and get your repo URL"/>

:::warning[`.gitignore`]

Don't add a `.gitignore` to your repo when creating it. It should be completely empty of all files. This is so Coalesce can initiate the repo correctly. After the repo is initiated, you can create a `.gitignore`.

These file paths are protected paths and files and should NOT be in the `.gitignore` and should not be used for anything else.

- data.yml
- locations.yml
- nodes/
- subgraphs/
- packages/
- macros/
- nodeTypes/
- environments/
- jobs/

:::

### Get Your Bitbucket API Token

Bitbucket requires an API Token.

:::danger[Bitbucket Cloud Removing App Passwords]

As of **September 9, 2025** Bitbucket no longer allows the creating of app passwords. New integrations will require an API token. On June 9, 2026 all existing app passwords will stop working.

Learn more in [Bitbucket Cloud enters phase two of app password deprecation](https://www.atlassian.com/blog/bitbucket/bitbucket-cloud-enters-phase-2-of-app-password-deprecation)

:::

<Tabs>
  <TabItem value="Bitbucket-repo-scope" label="Repository Scoped Access" default>

These access tokens are tied only to a [specific repository](https://support.atlassian.com/bitbucket-cloud/docs/create-a-repository-access-token/), not to a user account. They are created from the repository settings.

1. Go to **Repository Settings**.
2. Under **Security**, select **Access tokens**.
3. Click **Create Access Token**.
4. Choose the scopes: `Repositories Read and Write`.
5. Give it a name and set the expiration date according to your workspaces policy.
6. Review the information is correct and click **Create token**.
7. Copy the token since it won't be shown again.

   <DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>Repository Settings</figcaption>
    </figure>

    <figure>
    
    <figcaption>Security > Access tokens</figcaption>
    </figure>
    </DocCarousel>

    <Wistia id="qhib1wqfyo" subtitle="Create your repo access token"/>

  </TabItem>
  <TabItem value="bitbucket-user-scoped" label="User Scoped Access">
   These are tokens tied to a user account. They act like a personal access token that can be used to authenticate API calls or Git operations where allowed.

    1. Go to Settings and select **Atlassian account settings**.
    2. Select **Security**.
    3. Scroll to **API tokens**.
    4. Click **Create and manage API tokens**.
    5. Click **Create API token with scopes**.
    6. Select **Bitbucket** as the app.
    7. In the search box type **repository**. Select:
       1. `read:repository:bitbucket`
       2. `write:repository:bitbucket`
    8. Review the information is correct and click **Create token**.
    9. Copy the token since it won't be shown again.

   <DocCarousel slidesPerView={1}>
    <figure>
    
    <figcaption>Atlassian account settings</figcaption>
    </figure>

    <figure>
    
    <figcaption>Security > Create and manage API tokens</figcaption>
    </figure>
    </DocCarousel>

    <Wistia id="y7tcl7kjsc" subtitle="Create your API token in Bitbucket"/>
  </TabItem>

</Tabs>

---

### Add Your Bitbucket API Token and Repo to Coalesce

Review the steps in [Add your URL and Token to Coalesce][].

## GitLab

GitLab is an open-core DevOps platform that provides a complete software development lifecycle tool with integrated Git repository management, CI/CD, monitoring, and security features.

### Create Your GitLab Repo and URL

1. From the Projects page, click **New Project**.
2. Create a **Blank project**.
3. On the **Create blank project** page, make sure to uncheck **Initialize repository with a README**.
4. Click **Create project**.

    

5. On the project page, scroll down to **Add files**, and change the tab to **HTTPS**. Copy your repo URL.

    

:::warning[`.gitignore`]

Don't add a `.gitignore` to your repo when creating it. It should be completely empty of all files. This is so Coalesce can initiate the repo correctly. After the repo is initiated, you can create a `.gitignore`.

These file paths are protected paths and files and should NOT be in the `.gitignore` and should not be used for anything else.

- data.yml
- locations.yml
- nodes/
- subgraphs/
- packages/
- macros/
- nodeTypes/
- environments/
- jobs/

:::

<Wistia id="prjvcn9j6w" subtitle="Create your repo and get your URL in GitLab"/>

### Get Your GitLab Access Token

1. Click your avatar and select **Preferences**.
2. Select **Access tokens**.
3. Under **Personal access tokens**, click **Add new token**.

    

4. Give the token a name.
5. Select the scopes:
   1. `read_repository`
   2. `write_repository`

    

6. **Create personal access token**. Make sure to save this some where secure.

<Wistia id="640a1irnwe" subtitle="How to get your GitLab access token."/>

### Add Your GitLab Token and Repo to Coalesce

Review the steps in [Add your URL and Token to Coalesce][].

## Add Your Token and Repo To Coalesce

:::info[Bitbucket Repo Access]

Make sure to set the username as `x-token-auth` when adding a version control account in Coalesce for Bitbucket repo access generated tokens only.

:::

These instructions show you how to add your token and repo URL to your Project in Coalesce. The steps are the same for each provider.

After you have your access token, you need to add it to Coalesce and connect your Git account. You can do this as part of creating a project or by adding it your User Settings. In this example, we'll add it while creating a project.

1. Go to the Project page. If you are on the [Build page](/docs/build-your-pipeline/the-build-interface/) , click the **back arrow**.
2. Click the **plus sign**(+) next to Projects.

   

3. Enter the Project name and description. Click **Next**.

   

4. Enter your **Version control repository URL**.

   

5. On the next page, Click **Add New Account**.
   1. Enter an account nickname. This will displayed in the interface.
   2. Enter your username. If you are using Bitbucket AND generated a repo access token, set `x-token-auth` as the username.
   3. Enter your token. This will be either the GitLab token, Git token, Azure token, or Bitbucket Token
   4. Enter the **Author Name**, which identifies the committer. This doesn't have to match your version control account.
   5. Enter the **Author Email**, which identifies the committer email. This doesn't have to match your version control account.
   6. Click **Add**.

    

6. Select the Git account you just created in the drop down, then click **Test Account**.

   

7. Then click **Finish** to create your Project.

<Wistia id="zw9ah0t2ve" subtitle="How to create a project and add your version control information."/>

:::tip

**You can move to [Step 3: Create a Workspace][] since you've already created a Project in this step.**

:::

## What's Next

- Make sure each user is added to the repo.
- Each user needs to create an access token and update their Git Account in Coalesce. They can follow the instructions for each provider on creating a token. They can go to **User Settings > Version Control Accounts**.

[https://github.com/]:https://github.com/
[Resource owner]: https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/setting-a-personal-access-token-policy-for-your-organization
[Set Up Version Control]: /docs/git-integration/set-up-your-git-integration
[Add your URL and Token to Coalesce]:/docs/setup-your-project/setup-version-control#add-your-token-and-repo-to-coalesce
[Coalesce Git Requirements]: /docs/setup-your-project/setup-requirements/coalesce-git-requirements
[Step 3: Create a Workspace]: /docs/setup-your-project/create-a-workspace
[GitHub]: /docs/setup-your-project/setup-version-control#github
[Azure DevOps]: /docs/setup-your-project/setup-version-control#azure-devops
[Bitbucket]: /docs/setup-your-project/setup-version-control#bitbucket
[GitLab]: /docs/setup-your-project/setup-version-control#gitlab