databricks Platform · Updated Sep 2, 2025

Spark Declarative Pipelines on Lakeflow

Migration feature rolled back to Public Preview due to issues.

Public Preview Open docs ↗release note ↗

What it does

The migration feature for Spark Declarative Pipelines allows users to transition from the legacy publishing mode, which is limited to a single catalog and schema, to the default publishing mode that supports multiple catalogs and schemas. This migration affects only the metadata of the pipeline and does not involve any data movement. The legacy publishing mode is deprecated, and users are encouraged to migrate to the default mode for enhanced functionality.

Why it matters

Rolling back the migration feature to Public Preview allows Databricks to address issues before full deployment, ensuring that users can reliably publish to multiple catalogs and schemas without disruptions. This impacts data engineers and platform teams by providing a more flexible and powerful pipeline management system, ultimately improving productivity and reducing risks associated with data governance.

Key capabilities

Multiple Catalogs Support

The default publishing mode enables pipelines to publish to multiple catalogs and schemas.

Simplified Syntax

The new mode includes a simplified syntax for working with tables and views.

Metadata Only Migration

Migration affects only the metadata of the pipeline, ensuring no datasets are altered.

Backward Compatibility

The default publishing mode is generally backward compatible with the legacy mode.

Prerequisites

Databricks CLI version v0.230.0 or above is required.

Recommended use cases

Data engineers migrating existing pipelines to leverage new publishing capabilities.Teams needing to manage multiple catalogs and schemas efficiently.Organizations looking to modernize their data pipeline architecture.

Limitations · remember

!Migration is not supported using Declarative Automation Bundles.

!Once migrated, pipelines cannot revert to the legacy publishing mode.

!Materialized views and streaming tables cannot be moved across schemas after creation.

Architecture

Lakeflow → Spark Declarative Pipelines → Default Publishing Mode

Get started

Access Pipeline Settings

Click 'Jobs & Pipelines' from the left sidebar in your workspace.

Select Pipeline

Click the name of the pipeline you want to migrate.

Pause Updates

Pause updates and ensure any currently running pipeline stops.

Set Configuration

Add the configuration 'pipelines.enableDPMForExistingPipeline' and set it to 'true'.

Start Manual Update

Start a manual update and let it complete to enable the default publishing mode.

Key takeaway

The migration feature for Spark Declarative Pipelines has been rolled back to Public Preview to address issues, ensuring a stable transition to enhanced publishing capabilities.

2025-09-02-2f2b8730 Generated by the Databricks Updates Agent · browse all updates