databricks Platform · Updated Nov 25, 2025

Spark Declarative Pipelines on Lakeflow

Stream progress metrics are now generally available for enhanced monitoring.

What it does

The Spark Declarative Pipelines on Lakeflow now allows users to query the event log for metrics related to the progress of streaming data pipelines. This includes detailed information about pipeline updates, flow progress, data quality metrics, and overall pipeline state. Users can access this data through the pipeline monitoring UI, the Pipelines REST API, or by directly querying the event log using SQL syntax.

Why it matters

This update enhances the ability to monitor and troubleshoot streaming data pipelines, leading to improved operational efficiency and reduced downtime. By providing detailed metrics, data engineers and platform leaders can ensure better data quality and faster response times to issues, ultimately supporting more reliable data-driven decision-making.

Key capabilities

Event Log Access

Users can query the event log directly to retrieve detailed metrics about pipeline progress.

Custom Event Hooks

Define custom actions to trigger alerts or notifications based on logged events.

Data Quality Metrics

Access metrics related to data quality, including records that passed or failed expectations.

Streaming Queries Support

Use structured streaming to query views defined on top of event log tables.

Prerequisites

Pipelines must be configured with Unity Catalog.

Users must have sufficient privileges to query the event log.

Recommended use cases

Monitoring the status and progress of streaming data pipelines.Debugging issues related to incremental refresh in materialized views.Tracking data quality metrics to ensure compliance with data standards.

Limitations · remember

!Event log metrics may not be available for all data sets.

!Deleting the event log may cause pipeline failures in future runs.

Architecture

Lakeflow → Event Log → Pipelines REST API → Monitoring UI

Get started

Access Event Log

Use the pipeline ID to query the event log using SQL syntax.

Define Event Hooks

Set up custom actions to trigger alerts based on event log entries.

Create Views

Create views over the event log table for easier querying.

Monitor Pipeline Updates

Query the event log to track updates and their statuses.

Key takeaway

The general availability of stream progress metrics enhances monitoring capabilities for Spark Declarative Pipelines on Lakeflow.

2025-11-25-69edeeb5 Generated by the Databricks Updates Agent · browse all updates