databricks Platform · Updated Sep 10, 2025

Spark Declarative Pipelines on Lakeflow

Now supports stream progress metrics in Public Preview.

Public Preview Open docs ↗release note ↗

What it does

The update enables users to query the event log for metrics related to the progress of streaming pipelines. This includes tracking pipeline updates, monitoring data quality, and understanding data lineage through detailed event logs. Users can access this information via the pipeline monitoring UI, REST API, or direct SQL queries. The event log captures various metrics, including the number of output rows and data quality checks.

Why it matters

This enhancement allows data engineers and platform leaders to gain real-time insights into the performance and health of their streaming data pipelines. By monitoring progress metrics, organizations can improve operational efficiency, reduce downtime, and ensure data quality, ultimately leading to better decision-making and resource allocation.

Key capabilities

Event Log Querying

Users can directly query the event log to retrieve detailed metrics about pipeline progress.

Custom Event Actions

Define custom actions triggered by specific events logged in the pipeline.

Data Quality Monitoring

Metrics on data quality, such as passed and failed records, are now available for analysis.

Streaming Query Support

Views created over event logs support streaming queries, enhancing real-time data processing.

Prerequisites

Pipelines must be configured with Unity Catalog.

Users need sufficient privileges to access the event log.

Recommended use cases

Monitoring real-time data pipeline performance.Debugging issues with incremental refresh in materialized views.Tracking data quality metrics for compliance and governance.

Limitations · remember

!Event log metrics may not be available for all data sets.

!Deleting the event log can cause pipeline failures in future runs.

Architecture

Lakeflow → Event Log → Unity Catalog → Streaming Queries

Get started

Access Event Log

Use the pipeline ID to query the event log with the event_log function.

Create Event Log View

Create a view over the event log table for easier querying.

Define Event Hooks

Set up event hooks to trigger actions based on logged events.

Key takeaway

The new support for stream progress metrics enhances monitoring capabilities for Spark Declarative Pipelines on Lakeflow.

2025-09-10-fd48ee38 Generated by the Databricks Updates Agent · browse all updates