Skip to main content

Delta Live Table

  • streaming table,LDP

Overview

Lake Declarative Pipeline (LDP) in Databricks is a declarative framework for building and managing reliable, scalable, and maintainable data pipelines

Installation

  • Copy the Package ID  @coalesce/databricks/delta-live-table
  • In Coalesce, open the Workspace where you wish to install the package.
  • Go to the Build Setting of the Workspace, tab Packages, and click the Install button on the top right of the page.
  • Paste the Package ID, and proceed with the installation process.

Description

Lakeflow Declarative Pipelines

Introduction

The Coalesce LDP node type allows you to create a streaming table. A streaming table is a form of Unity Catalog managed table that is also a streaming target for Lakeflow Declarative Pipelines.

Lakeflow Declarative Pipelines is a declarative framework for developing and running batch and streaming data pipelines in SQL and Python. Lakeflow Declarative Pipelines runs on the performance-optimized Databricks Runtime (DBR), and the Lakeflow Declarative Pipelines flows API uses the same DataFrame API as Apache Spark and Structured Streaming. Common use cases for Lakeflow Declarative Pipelines include incremental data ingestion from sources such as cloud storage (including Amazon S3, Azure ADLS Gen2, and Google Cloud Storage) and message buses (such as Apache Kafka, Amazon Kinesis, Google Pub/Sub, Azure EventHub, and Apache Pulsar), incremental batch and streaming transformations with stateless and stateful operators, and real-time stream processing between transactional stores like message buses and databases.

A streaming table is a Delta table with additional support for streaming or incremental data processing. A streaming table can be targeted by one or more flows in an ETL pipeline.

Key points

1.Databricks by default creates tables with lowercase.Hence,it is better to keep table names in lowercase.https://docs.databricks.com/en/sql/language-manual/sql-ref-names.html

2.The node which loads from a file creates a streaming table.For further processing,Re-Sync the columns in the mapping grid using Re-Sync columns button. The streaming table can be re-created with the Columns inferred using Include Columns Inferred option.

dlt-resync

3.The streaming tables can be recreated and refreshed if there is a need to drop the inferred columns or add transformations to columns inferred in previous step.The structure of the streaming table is refreshed only on enabling the 'Refresh Stream' Option

LDP Node Configuration

The LDP has two configuration groups:

LDP Node Properties

There are four configs within the Node Properties group.

PropertyDescription
Storage LocationStorage Location where the Materialized View will be created
Node TypeName of template used to create node objects
DescriptionA description of the node's purpose
Deploy EnabledIf TRUE the node will be deployed / redeployed when changes are detected
If FALSE the node will not be deployed or will be dropped during redeployment

General Options

OptionDescription
Create AsChoose materialization type-by default it is STREAMING TABLE
Schedule refreshTrue / False toggle
- True: Schedule Option will be visible
- False: Schedule Option will be disabled

LDP options

OptionDescription
Type of Lakeflow Declarative Pipeline- Streaming Table
Read FilesTrue / False toggle
- True: Allows loading from an external cloud location into a streaming table.
- False: Allows loading from a table source.
Include Columns InferredEnables recreating the streaming table with added transformations after syncing columns.
File LocationExternal cloud location of the file.
File Format- CSV
- JSON
DelimiterThe delimiter used in case of a CSV file.
HeaderSpecifies if a header is needed for CSV file parsing.
Table PropertiesDefines table properties like quality.
Refresh StreamEnables full refresh of the table if any changes in structure occur.
Partition BySpecifies the columns used for partitioning the streaming table.
Table Constraints- Primary Key
- Foreign Key
Other Constraints- Column Name
- Expectation Expression
- On Violation Action

Schedule Options

Schedule Options is available only when Schedule Refresh toggle is True

OptionDescription
Task ScheduleOptions in Task Schedule
- Periodic Schedule
- CRON
Schedule refesh-time periodAvailable when Task Schedule is Set to Periodic Schedule
Options in Schedule refesh-time period
-Every Hours
-Every Days
-Every Weeks
Specific interval of periodic refresh(integer value)Available when Task Schedule is Set to Periodic Schedule
CRON stringAvailable when Task Schedule is Set to CRON
CRON TIME ZONEAvailable when Task Schedule is Set to CRON

LDP Deployment

Initial Deployment

When deployed for the first time into an environment DLT node will execute three stages:

StageDescription
Create Streaming tableThis stage will execute a CREATE OR REFRESH statement and create a Streaming table in the target environment

Redeployment

After the Streaming table has deployed for the first time into a target environment, subsequent deployments may result in either altering the Streaming table or recreating the Streaming table.

If a Streaming table is to be altered this will run the following stage:

Altering the Streaming table

The following config changes trigger ALTER statements:

  1. Add schedule
  2. Alter schedule
  3. Drop schedule

These execute the two stages:

StageDescription
Alter Streaming tableExecutes ALTER to modify parameters

Recreating the Streaming table

If anything changes other than the configuration options specified above then the Streaming table will be recreated by running a CREATE OR REPLACE statement.

Undeployment

If a Streaming table is deleted from a Workspace, that Workspace is committed to Git and that commit deployed to a higher-level environment then the Streaming table in the target environment will be dropped.

This is executed as a single stage:

StageDescription
Drop Streaming tableRemoves the Streaming table

Code

Versions

Available versions of the package.

Version #Release DateNotes
1.0.2July 30, 2025
 Node name changed to LDP 
1.0.1June 16, 2025
 Scheduling option added 
1.0.0June 06, 2025
 Initial version of Delta Live Tables Databricks