databricks Platform · Updated May 15, 2026

Spark Declarative Pipelines on Lakeflow Sinks

The sink API for Spark Declarative Pipelines is now generally available, enabling seamless data integration.

What it does

The Spark Declarative Pipelines sink API allows users to write transformed data from pipelines to various external targets, including Delta tables, Apache Kafka topics, and Azure Event Hubs. This functionality supports both Unity Catalog managed and external tables, as well as custom Python data sources. Users can create sinks using the `create_sink()` function and stream data using append flows.

Why it matters

This update enhances data integration capabilities, allowing organizations to efficiently stream processed data to multiple external systems. By leveraging the sink API, businesses can improve their data workflows, reduce latency in data processing, and enable real-time analytics, ultimately driving better decision-making and operational efficiency.

Key capabilities

Multiple Sink Types

Supports Delta tables, Apache Kafka, Azure Event Hubs, and custom Python data sources.

Append Flow Processing

Allows streaming of transformed data records to sinks using append flows.

Custom Sink Creation

Enables users to create custom sinks using Python for tailored data integration.

Unity Catalog Integration

Facilitates writing to Unity Catalog managed and external tables.

Prerequisites

Python API is required to use the sink functionality.

A pipeline must be set up to process streaming event data.

Recommended use cases

Streaming data from event sources to Delta tables for analytics.Integrating real-time data with Apache Kafka for event-driven architectures.Sending processed data to Azure Event Hubs for further processing.

Limitations · remember

!Only the Python API is supported; SQL is not available.

!Only streaming queries can be used; batch queries are not supported.

!Pipeline expectations are not supported.

!Full refresh updates do not clean up previously computed results in sinks.

Architecture

Data Ingestion → Data Transformation → Sink Creation → Data Streaming

Get started

Create Sink

Use the `create_sink()` function to define your sink.

Define Append Flow

Set up an append flow to specify how data will be written to the sink.

Configure Sink Options

Provide necessary options for the sink, such as table names or connection details.

Stream Data

Begin streaming processed records to the sink using the defined flow.

Key takeaway

The general availability of the Spark Declarative Pipelines sink API significantly enhances data integration capabilities for real-time analytics.

2026-05-15-spark-declarative-pipelines-on-lakeflow-sinks-are-now-generally-available Generated by the Databricks Updates Agent · browse all updates