Base Node Types

‹ View all packages

Package ID: @coalesce/databricks/base-node-types

Supported Platform:

Latest Version: 1.0.0 (June 06, 2025)

Overview

Coalesce base Node Types to construct facts and dimensions.

Installation

1. Copy the Package ID: @coalesce/databricks/base-node-types
2. In Coalesce, open the Workspace where you wish to install the package.
3. Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
4. Paste the Package ID, and proceed with the installation process.

Description

Coalesce Databricks Base Node Types Package

The Coalesce Base Node Types Package includes:

• Work
• Persistent Stage
• Dimension
• Fact
• View
• Code

---

Work

The Coalesce work node is a versatile node that allows you to develop and deploy a Work table/view in Databricks.

A Work node serves as an intermediary object and is commonly employed to store raw data before undergoing the crucial phases of transformation and loading into the main tables of the data warehouse.

This pivotal step ensures that the raw data is processed and structured effectively.

Work Node Configuration

The Work node type has two configuration groups:

• Node Properties
• Options

Work Node Properties

Property	Description
Storage Location	Storage Location where the WORK will be created
Node Type	Name of template used to create node objects
Description	A description of the node's purpose
Deploy Enabled	If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment

Work Options

You can create the node as:

• Table
• View

Work Options Table

Property	Description
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination - INSERT: Individual insert for each source False: Single source node or multiple sources combined using a join
Truncate Before	Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data
Enable tests	Toggle: True/False Determines if tests are enabled
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible
Order By	Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible
Pre-SQL	SQL to execute before data insert operation
Post-SQL	SQL to execute after data insert operation

Work Options View

Setting	Description
Override Create SQL	Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible

Work Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

📘 Specify Group by and Order by Clauses

You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

Work Deployment

Work Initial Deployment

When deployed for the first time into an environment the Work node of materialization type table will execute the below stage:

Stage	Description
Create Work Table	This will execute a CREATE OR REPLACE statement and create a table in the target environment
Create Work View	This will execute a CREATE OR REPLACE statement and create a view in the target environment

Work Redeployment

After the WORK node with materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the WORK Table or recreating the WORK table.

Altering the Work Tables

A few types of column or table changes will result in an ALTER statement to modify the Work Table in the target environment, whether these changes are made individually or all together:

• Changing table names
• Dropping existing columns
• Altering column data types
• Adding new columns

The following stages are executed:

Stage	Description
Clone Table	Creates an internal table
Rename Table| Alter Column | Delete Column | Add Column | Edit table description	Alter table statement is executed to perform the alter operation
Swap Cloned Table	Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost
Delete Table	Drops the internal table

Recreating the Work Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

• Join clause
• Adding transformation
• Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

Stage	Description
Create Table	Creates a new table
Replace Table	Replaces an existing table

Recreating the Work Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

Stage	Description
Delete View	Removes existing view
Create View	Creates new view with updated definition

Work Undeployment

If a Work Node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkTable in the target environment will be dropped.

This is executed in two stages:

Stage	Description
Delete Table	Coalesce Internal table is dropped
Delete Table	Target table in Databricks is dropped

If a Work Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the WorkView in the target environment will be dropped.

The stage executed:

Stage	Description
Delete View	Drops the existing Work view from target environment

---

Persistent Stage

The Coalesce Persistent Stage Nodes element, serving as an intermediary object, is frequently utilized to maintain data persistence across multiple execution cycles.

It plays a crucial role in tracking the historical changes of columns linked to business keys.

This functionality is particularly beneficial when the objective is to retain raw data for prolonged durations.

Persistent Stage Node Configuration

The Persistent node type has two configuration groups:

• Node Properties
• Options

Persistent Stage Node Properties

Property	Description
Storage Location	Storage Location where the WORK will be created
Node Type	Name of template used to create node objects
Description	A description of the node's purpose
Deploy Enabled	If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment

Persistent Stage Options

Option	Description
Create As	Table is the only option at this time
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Business key	Required column for both Type 1 and Type 2
Change tracking	Required column for Type 2
Truncate Before	Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data
Enable tests	Toggle: True/False Determines if tests are enabled
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible
Order By	Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible
Pre-SQL	SQL to execute before data insert operation
Post-SQL	SQL to execute after data insert operation

Persistent Stage Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

📘 Specify Group by and Order by Clauses

You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

Persistent Stage Deployment

Persistent Stage Initial Deployment

When deployed for the first time into an environment the Persistent node will execute the below stage:

Stage	Description
Create Persistent Table	This will execute a CREATE OR REPLACE statement and create a table in the target environment

Persistent Stage Redeployment

After the Persistent node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Persistent Table or recreating the Persistent table.

Altering the Persistent Tables

A few types of column or table changes will result in an ALTER statement to modify the Persistent Table in the target environment, whether these changes are made individually or all together:

• Changing table names
• Dropping existing columns
• Altering column data types
• Adding new columns

The following stages are executed:

Stage	Description
Clone Table	Creates an internal table
Rename Table| Alter Column | Delete Column | Add Column | Edit table description	Alter table statement is executed to perform the alter operation
Swap Cloned Table	Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost
Delete Table	Drops the internal table

Recreating the Persistent Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

• Join clause
• Adding transformation
• Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

Stage	Description
Create Table	Creates a new table
Replace Table	Replaces an existing table

Persistent Stage Undeployment

If a Persistent Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Persistent Table in the target environment will be dropped.

This is executed in two stages:

Stage	Description
Delete Table	Drops table
Drop Table or View	Removes the table

---

Dimension

The Coalesce Dimension UDN is a versatile node that allows you to develop and deploy a Dimension table in Snowflake.

A dimension table or dimension entity is a table or entity in a star, Snowflake, or starflake schema that stores details about the facts. Dimension tables describe the different aspects of a business process.

Dimension Node Configuration

The Dimension node type has two configuration groups:

• Node Properties
• Options

Dimension Node Properties

Property	Description
Storage Location	Storage Location where the WORK will be created
Node Type	Name of template used to create node objects
Description	A description of the node's purpose
Deploy Enabled	If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment

Dimension Options

Dimension Table Options

Options	Description
Create As	Table or View
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Business key	Required column for both Type 1 and Type 2 Dimensions
Change tracking	Required column for Type 2 Dimension
Truncate Before	Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data
Enable tests	Toggle: True/False Determines if tests are enabled
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible
Order By	Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible
Pre-SQL	SQL to execute before data insert operation
Post-SQL	SQL to execute after data insert operation

Dimension View Options

Options	Description
Override Create SQL	Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Business key	Required column for both Type 1 and Type 2 Dimensions
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible

Dimension Joins

Join conditions and other clauses can be specified in the join space next to mapping of columns in the UI.

📘 Specify Group by and Order by Clauses

You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

Dimension Deployment

Dimension Initial Deployment

When deployed for the first time into an environment the Dimension node of materialization type table will execute the Create Dimension Table stage.

Stage	Description
Create Dimension Table	This will execute a CREATE OR REPLACE statement and create a table in the target environment
Create Dimension View	This will execute a CREATE OR REPLACE statement and create a view in the target environment

Dimension Redeployment

After the Dimension node of materialization type table has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Dimension Table or recreating the Dimension table.

Altering the Dimension Tables

A few types of column or table changes will result in an ALTER statement to modify the Dimension Table in the target environment, whether these changes are made individually or all together:

• Changing table names
• Dropping existing columns
• Altering column data types
• Adding new columns

The following stages are executed:

Stage	Description
Clone Table	Creates an internal table
Rename Table| Alter Column | Delete Column | Add Column | Edit table description	Alter table statement is executed to perform the alter operation
Swap Cloned Table	Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost
Delete Table	Drops the internal table

Recreating the Dimension Tables

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

• Join clause
• Adding transformation
• Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

Stage	Description
Create Table	Creates a new table
Replace Table	Replaces an existing table

Recreating the Dimension Views

Any of the following changes to views will result in deleting and recreating the Dimension view.

• View defintion
• Adding table description
• Renaming view results

Dimension Undeployment

If a Dimension Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension Table in the target environment will be dropped.

This is executed in two stages:

Stage	Description
Delete Table	Coalesce Internal table is dropped
Delete Table	Target table in Databricks is dropped

If a Dimension Node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Dimension View in the target environment will be dropped.

Stage	Description
Delete View	Drops the existing Dimension view from target environment.

Fact

The Coalesce Fact UDN is a versatile node that allows you to develop and deploy a Fact table in Databricks.

A fact table or a fact entity is a table or entity in a star or Databricks schema that stores measures that measure the business, such as sales, cost of goods, or profit. Fact tables and entities aggregate measures, or the numerical data of a business.

Fact Node Configuration

The Fact node has two configuration groups:

• Node Properties
• Options

Fact Node Properties

Properties	Description
Storage Location	Storage Location where the WORK will be created
Node Type	Name of template used to create node objects
Description	A description of the node's purpose
Deploy Enabled	If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment

Fact Options

Options	Description
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Business key	Required column for Fact table creation
Truncate Before	Toggle: True/False This determines whether a table will be overwritten each time a task executes. True: Uses INSERT OVERWRITE False: Uses INSERT to append data
Enable tests	Toggle: True/False Determines if tests are enabled
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible
Order By	Toggle: True/False True: Sort column and sort order drop down are visible and are required to form order by clause False: Sort column and sort order drop down are invisible
Pre-SQL	SQL to execute before data insert operation
Post-SQL	SQL to execute after data insert operation

Fact View

Setting	Description
Override Create SQL	Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible

Fact Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the UI.

📘 Specify Group by and Order by Clauses

You should specify group by and order by clauses in this space if you are not opting for the group by all and order by provided in OPTIONS config.

Fact Deployment

Fact Initial Deployment

When deployed for the first time into an environment the Fact node of materialization type table will execute the Create Fact Table stage.

Stage	Description
Create Fact Table	This will execute a CREATE OR REPLACE statement and create a table in the target environment
Create Fact View	This will execute a CREATE OR REPLACE statement and create a view in the target environment

Fact Redeployment

After the Fact node has been deployed for the first time into a target environment, subsequent deployments may result in either altering the Fact Table or recreating the Fact table.

Altering the Fact Tables

A few types of column or table changes will result in an ALTER statement to modify the Fact Table in the target environment, whether these changes are made individually or all together:

• Changing table names
• Dropping existing columns
• Altering column data types
• Adding new columns

The following stages are executed:

Stage	Description
Clone Table	Creates an internal table
Rename Table| Alter Column | Delete Column | Add Column | Edit table description	Alter table statement is executed to perform the alter operation
Swap Cloned Table	Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost
Delete Table	Drops the internal table

Recreating the Fact Tables

If any changes like change in join clause, adding transformations, change in business key column, change in configs like adding distinct, group by or orderby, the Fact Table will be recreated by running a CREATE OR REPLACE statement.

If any of the following change are detected, then the table will be recreated using a CREATE or REPLACE.

• Join clause
• Adding transformation
• Change in business key
• Changes in configuration like adding distinct, group by, or order by

One of the following stages are executed:

Stage	Description
Create Table	Creates a new table
Replace Table	Replaces an existing table

Recreating the Fact Views

The subsequent deployment of Work node of materialization type view with changes in view definition, adding table description or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

Stage	Description
Delete View	Removes existing view
Create View	Creates new view with updated definition

Fact Undeployment

If a Fact Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the Fact Table in the target environment will be dropped.

This is executed in two stages:

Stage	Description
Delete Table	Coalesce Internal table is dropped
Delete Table	Target table in Databricks is dropped

View

The Coalesce View UDN is a versatile node that allows you to develop and deploy a View in Databricks.

A view allows the result of a query to be accessed as if it were a table. Views serve a variety of purposes, including combining, segregating, and protecting data.

View Node Configuration

The View node type has two configuration groups:

• Node Properties
• Options

View Node Properties

Properties	Description
Storage Location	Storage Location where the WORK will be created
Node Type	Name of template used to create node objects
Description	A description of the node's purpose
Deploy Enabled	If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment

View Options

Options	Description
Override Create SQL	Toggle: True/False True: Customized Create SQL specified in the Create SQL space is executed. All other options are invisible False: Create view SQL based on options chosen are framed and executed
Multi Source	Toggle: True/False Implementation of SQL UNIONs True: Combine multiple sources in a single node True Options: - UNION: Combines with duplicate elimination - UNION ALL: Combines without duplicate elimination False: Single source node or multiple sources combined using a join
Distinct	Toggle: True/False True: Group by All is invisible. DISTINCT data is chosen for processing False: Group by All is visible
Group by All	Toggle: True/False True: DISTINCT is invisible. Data is grouped by all columns for processing False: DISTINCT is visible
Secure	Toggle: True/False True: A secured view is created False: A normal view is created

View Joins

Join conditions and other clauses like where, qualify can be specified in the join space next to mapping of columns in the Coalesce app.

📘 Specify Group by Clauses

Best Practice is to specify group by clauses in this space if you are not opting for the group by all provided in OPTIONS config.

View Deployment

View Initial Deployment

When deployed for the first time into an environment the View node will execute the Create View stage.

Stage	Description
Create View	This will execute a CREATE OR REPLACE statement and create a View in the target environment

View Redeployment

The subsequent deployment of View node with changes in view definition, adding table description, adding secure option or renaming view results in deleting the existing view and recreating the view.

The following stages are executed:

Stage	Description
Delete View	Removes existing view
Create View	Creates new view with updated definition

View Undeployment

If a View Node is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the View in the target environment will be dropped.

This is executed in the below stage:

Stage	Description
Delete View	Removes the view from the environment

Code

Work Code

• Node definition
• Create Template
• Run Template

Persistent Stage Code

• Node definition

• Create Template

• Run Template

  Dimenison Code

• Node definition

• Create Template

• Run Template

Fact Code

• Node definition
• Create Template
• Run Template

View Code

• Node definition
• Create Template
• Run Template

Macros

Versions

Available versions of the package.

Version #	Release Date	Notes
1.0.0	June 06, 2025	Initial version of Databricks Base node types