Delta Live Table
- Latest Version:
- 1.0.1
- Supported Platform:
- Release Date:
- June 16, 2025
Overview
Delta Live Tables (DLT) in Databricks is a declarative framework for building and managing reliable, scalable, and maintainable data pipelines
Installation
- In Coalesce, open the Workspace where you wish to install the package.
- Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
- Paste the Package ID, and proceed with the installation process.
Description
Delta Live Tables
Introduction
The Coalesce DLT node type allows you to create a streaming table,a type of Delta Live Table.
Delta Live Tables is a declarative framework for building reliable, maintainable, and testable data processing pipelines. Delta Live Tables manages how your data is transformed based on queries you define for each processing step. You can also enforce data quality with Delta Live Tables expectations, which allow you to define expected data quality and specify how to handle records that fail those expectations.
Databricks recommends using streaming tables to ingest data using Databricks SQL. A streaming table is a table registered to Unity Catalog with extra support for streaming or incremental data processing. A Delta Live Tables pipeline is automatically created for each streaming table.
Key points
1.Loading file from external location/databricks managed volume/external volume is currently supported.
2.Databricks by default creates tables with lowercase.Hence,it is better to keep table names in lowercase.https://docs.databricks.com/en/sql/language-manual/sql-ref-names.html
3.The node which loads from a file creates a streaming table.For further processing,Re-Sync the columns in the mapping grid using Re-Sync columns button. The streaming table can be re-created with the Columns inferred using Include Columns Inferred option.
4.The streaming tables can be recreated and refreshed if there is a need to drop the inferred columns or add transformations to columns inferred in previous step.The structure of the streaming table is refreshed only on enabling the 'Refresh Stream' Option
DLT Node Configuration
The DLT has two configuration groups:
DLT Node Properties
There are four configs within the Node Properties group.
| Property | Description |
|---|---|
| Storage Location | Storage Location where the Materialized View will be created |
| Node Type | Name of template used to create node objects |
| Description | A description of the node's purpose |
| Deploy Enabled | If TRUE the node will be deployed / redeployed when changes are detected If FALSE the node will not be deployed or will be dropped during redeployment |
General Options
| Option | Description |
|---|---|
| Create As | Choose materialization type-by default it is STREAMING TABLE |
| Schedule refresh | True / False toggle - True: Schedule Option will be visible - False: Schedule Option will be disabled |
DLT options
| Option | Description |
|---|---|
| Type of Delta Live Table | - Streaming Table - Materialized View |
| Read Files | True / False toggle - True: Allows loading from an external cloud location into a streaming table. - False: Allows loading from a table source. |
| Include Columns Inferred | Enables recreating the streaming table with added transformations after syncing columns. |
| File Location | External cloud location of the file. |
| File Format | - CSV - JSON |
| Delimiter | The delimiter used in case of a CSV file. |
| Header | Specifies if a header is needed for CSV file parsing. |
| Table Properties | Defines table properties like quality. |
| Refresh Stream | Enables full refresh of the table if any changes in structure occur. |
| Partition By | Specifies the columns used for partitioning the streaming table. |
| Table Constraints | - Primary Key - Foreign Key |
| Other Constraints | - Column Name - Expectation Expression - On Violation Action |
Schedule Options
Schedule Options is available only when Schedule Refresh toggle is True
| Option | Description |
|---|---|
| Task Schedule | Options in Task Schedule - Periodic Schedule - CRON |
| Schedule refesh-time period | Available when Task Schedule is Set to Periodic Schedule Options in Schedule refesh-time period -Every Hours -Every Days -Every Weeks |
| Specific interval of periodic refresh(integer value) | Available when Task Schedule is Set to Periodic Schedule |
| CRON string | Available when Task Schedule is Set to CRON |
| CRON TIME ZONE | Available when Task Schedule is Set to CRON |
DLT Deployment
Initial Deployment
When deployed for the first time into an environment DLT node will execute three stages:
| Stage | Description |
|---|---|
| Create Streaming table | This stage will execute a CREATE OR REFRESH statement and create a Streaming table in the target environment |
Redeployment
After the Streaming table has deployed for the first time into a target environment, subsequent deployments may result in either altering the Streaming table or recreating the Streaming table.
If a Streaming table is to be altered this will run the following stage:
Altering the Streaming table
The following config changes trigger ALTER statements:
- Add schedule
- Alter schedule
- Drop schedule
These execute the two stages:
| Stage | Description |
|---|---|
| Alter Streaming table | Executes ALTER to modify parameters |
Recreating the Streaming table
If anything changes other than the configuration options specified above then the Streaming table will be recreated by running a CREATE OR REPLACE statement.
Undeployment
If a Streaming table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the Streaming table in the target environment will be dropped.
This is executed as a single stage:
| Stage | Description |
|---|---|
| Drop Streaming table | Removes the Streaming table |
Code
Versions
Available versions of the package.
| Version # | Release Date | Notes |
|---|---|---|
| 1.0.1 | June 16, 2025 | |
| 1.0.0 | June 06, 2025 | |