Incremental Nodes
@coalesce/databricks/incremental-nodes 
Overview
Incremental node types
Installation
- Copy the Package ID:
@coalesce/databricks/incremental-nodes - In Coalesce, open the Workspace where you wish to install the package.
- Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
- Paste the Package ID, and proceed with the installation process.
Description
Databricks-Incremental-nodes
The Coalesce Incremental Package includes:
Incremental Load
The Coalesce Incremental load node is a versatile node that allows you to develop and deploy a Stage table/view in Databricks where we can perform incremental load in comparison with a persistent table added on top of it.
Incremental Load Node Configuration
Incremental Load Node Properties
| Property | Description |
|---|---|
| Storage Location | Storage Location where the Incremental node will be created. |
| Node Type | Name of template used to create node objects. |
| Description | A description of the node's purpose. |
| Deploy Enabled | - If TRUE the node will be deployed / redeployed when changes are detected. - If FALSE the node will not be deployed or will be dropped during redeployment. |
Incremental Load Options
| Options | Description |
|---|---|
| Create As | Provides option to choose materialization type as table or view. |
| Filter data based on Persistent table | - True - provides option to perform incremental load. - False - a normal initial load of data from source is done. |
| Persistent table location(required) | The Coalesce storage location. |
| Persistent table name(required) | The table name of the persistent table. |
| Incremental load column(date) | A date column based on which incremental data is loaded. |
Incremental Load Example Workflow
- Add a source node.
- Add the Incremental UDN.
- Leave the 'Filter data based on Persistent Table' option set to False.
- Create the node.
- Add the Persistent table to the view.
- Create and Run the node.
- Go to the Incremental UDN and change the 'Filter data based on Persistent Table' option to true.
- Use the pattern based option to match the persistent table (alter the definition of the UDN, if necessary), or add the table name manually in the last config item.
- Remove the existing (basic) join and use the 'Copy To Editor' to add the new join, including sub-select.
- Re-run the Incremental UDN.
Incremental Load Deployment
Incremental Load Initial Deployment
When deployed for the first time into an environment the Incremental load node of materialization type table will execute the Create State Table.
| Stage | Description |
|---|---|
| Create Stage Table | This will execute a CREATE OR REPLACE statement and create a table in the target environment. When deployed for the first time into an environment the Work node of materialization type view will execute the Create Stage View. |
| Create Stage View | This will execute a CREATE OR REPLACE statement and create a view in the target environment. |
Incremental Load Redeployment
After the Incremental Load has been deployed for the first time into a target environment, subsequent deployments with column level changes or table level changes may result in altering the target Table.
Incremental Load Altering the Stage Tables
There are few column or table changes if made in isolation or all-together will result in an ALTER statement to modify the Work Table in the target environment.
- Changing the table name
- Dropping an existing column
- Altering Column data type
- Adding a new column
The following stages are executed:
| Stage | Description |
|---|---|
| Clone Table | Creates an internal table. |
| Rename Table | Alter Column | Delete Column | Add Column | Edit table description | Alter table statement is executed to perform the alter operation. |
| Swap cloned Table | Upon successful completion of all updates, the clone replaces the main table ensuring that no data is lost. |
| Delete Table | Drops the internal table. |
Incremental Load Recreating the Stage Views
The subsequent deployment of Incremental load node of materialization type view with changes in view definition,adding table description or renaming view results in deleting the existing view and recreating the view.
The following stages are executed:
| Stage | Description |
|---|---|
| Delete View | Delete the view |
| Create View | Create a new view |
Incremental Load Undeployment
If a Incremental load node of materialization type table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the stage table in the target environment will be dropped.
This is executed in two stages:
| Stage | Description |
|---|---|
| Delete Table | Coalesce Internal table is dropped. |
| Delete Table | Target table in Snowflake is dropped. |
If a Incremental load node of materialization type view is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher level environment then the StageView in the target environment will be dropped.
The stage executed:
| Stage | Description |
|---|---|
| Delete View | Drops the existing stage view from target environment. |
Code
Incremental Load Code
Versions
Available versions of the package.
| Version # | Release Date | Notes |
|---|---|---|
| 1.0.0 | November 14, 2025 | First release of Databricks Incremental node type |