databricks Platform · Updated Apr 28, 2026

Spark Declarative Pipelines on Lakeflow

Update history retention extended to 60 days.

What it does

The update history for Spark Declarative Pipelines on Lakeflow is now retained for 60 days, allowing users to view and access updates in the pipeline UI and through the Pipelines REST API. This change enhances the ability to track changes and manage pipeline versions over a longer period. Users can now more effectively monitor and audit their data processing workflows.

Why it matters

This extended retention period improves governance and operational efficiency by allowing data engineers and platform leaders to better track changes and troubleshoot issues over a longer timeframe. It supports compliance and auditing requirements, ultimately leading to enhanced data reliability and trustworthiness.

Key capabilities

Pipeline Management

Create, edit, delete, and start pipelines using the API.

Task Orchestration

Manage task orchestration and cluster management automatically.

Data Quality Enforcement

Define expectations for data quality and handle records that fail those expectations.

Monitoring and Error Handling

Monitor pipeline execution and manage error handling effectively.

Architecture

Lakeflow → Spark Declarative Pipelines → Pipelines REST API → Data Processing Workflows

In short

You can now see updates to Spark Declarative Pipelines for 60 days instead of 30.

Read the full documentation ↗

Key takeaway

The retention period for update history in Spark Declarative Pipelines has been extended to enhance tracking and governance.

2026-04-28-spark-declarative-pipelines-on-lakeflow-updates-are-retained-for-60-days Generated by the Databricks Updates Agent · browse all updates