databricks Platform · Updated Dec 9, 2025

ForEachBatch for Spark Declarative Pipelines

Now available in public preview for processing streams as micro-batches.

Public Preview Open docs ↗release note ↗

What it does

The ForEachBatch sink allows users to process streams in Spark Declarative Pipelines as a series of micro-batches using Python. It enables the application of custom logic to each micro-batch, such as merging data into external tables or writing to unsupported destinations. The sink supports Unity Catalog features and requires users to manage downstream data resets during full refreshes.

Why it matters

This update enhances the flexibility and capability of data pipelines, allowing organizations to handle complex data transformations and integrations with external systems. It benefits data engineers and analysts by enabling more sophisticated data processing workflows, ultimately improving data accuracy and operational efficiency.

Key capabilities

Custom Logic Processing

Apply arbitrary actions to each micro-batch using Python code.

Full Refresh Management

Manage checkpoints and data resets during full refresh scenarios.

Unity Catalog Integration

Utilize Unity Catalog features for reading and writing data.

Event Logging

Track the creation and usage of each ForEachBatch sink in the event log.

Recommended use cases

Merging or upserting into Delta Lake tables.Writing to multiple or unsupported destinations.Applying advanced transformations using Python libraries.

Limitations · remember

!No automatic data cleanup for ForEachBatch outputs.

!Metrics may be incomplete or unusual due to the flexibility of ForEachBatch.

!Requires serialization of user-defined functions for Databricks Connect compatibility.

!Custom logic must avoid using non-serializable resources.

Architecture

Lakeflow Pipelines → ForEachBatch Sink → Unity Catalog

In short

You can now use ForEachBatch to process streaming data in micro-batches with custom Python logic.

Read the full documentation ↗

Key takeaway

ForEachBatch enhances Spark Declarative Pipelines by enabling flexible micro-batch processing with custom logic.

2025-12-09-a8173e71 Generated by the Databricks Updates Agent · browse all updates