databricks Platform · Updated Jun 9, 2026

Update Flow API for Spark Declarative Pipelines

Now generally available for enhanced data processing.

What it does

The `@dp.update_flow` decorator allows users to create update flows that write to sinks in update output mode, emitting only the rows that change in each batch. This feature supports stateful aggregations without requiring a watermark, making it distinct from append flows. Update flows are specifically designed to target sinks, although they do not support Delta tables as targets.

Why it matters

This update enhances data processing efficiency by enabling real-time updates and reducing unnecessary data writes, which can lead to cost savings and improved performance. Organizations can leverage this capability for more responsive data pipelines, particularly in scenarios requiring stateful processing.

Key capabilities

Stateful Aggregations

Supports stateful aggregations without the need for watermarks.

Selective Data Emission

Emits only changed rows in each batch, optimizing data flow.

Custom Sink Targeting

Allows targeting of various sinks for data output, excluding Delta tables.

Flexible Configuration

Offers optional parameters for flow name, comments, and Spark configurations.

Recommended use cases

Real-time data processing for event counts.Streaming analytics with stateful aggregations.Efficient data updates in data lakes.

Limitations · remember

!Delta tables are not supported as targets for update flows.

Architecture

Data Ingestion → Streaming Processing → Update Flow API → Data Sink

Get started

Create Sink

Use `dp.create_sink` to define the target sink for the update flow.

Define Update Flow

Apply the `@dp.update_flow` decorator to a function that returns a streaming DataFrame.

Configure Parameters

Set optional parameters like flow name and Spark configurations as needed.

Implement Logic

Write the logic for data processing within the decorated function.

Run the Flow

Execute the flow to start processing and emitting updates.

Key takeaway

The update flow API enables efficient real-time updates in Spark pipelines, enhancing data processing capabilities.

2026-06-09-658bd9e1 Generated by the Databricks Updates Agent · browse all updates