databricks Platform · Updated Jun 23, 2025

Spark Declarative Pipelines Migration

Transition from legacy to default publishing mode for enhanced flexibility.

What it does

The migration from legacy publishing mode to the default publishing mode allows Spark Declarative Pipelines on Lakeflow to publish to multiple catalogs and schemas, enhancing flexibility in data management. This update simplifies the syntax for working with tables and views within the pipeline and is intended to replace the deprecated legacy mode. Migration affects only the metadata of the pipeline and does not involve any data movement or modification.

Why it matters

This migration is crucial for organizations looking to optimize their data workflows, as it enables more complex data architectures and reduces the limitations imposed by the legacy mode. By allowing multiple catalogs and schemas, businesses can improve data accessibility and governance, ultimately enhancing productivity and reducing operational risks.

Key capabilities

Multiple Catalogs Support

The default publishing mode allows a single pipeline to write to multiple catalogs and schemas.

Simplified Syntax

The new mode includes a simplified syntax for working with tables and views.

Metadata Migration Only

Migration affects only the metadata and does not read, move, or write to datasets.

Backward Compatibility

The default publishing mode is generally backward compatible with the legacy mode.

Prerequisites

Databricks CLI version v0.230.0 or above is required.

Recommended use cases

Organizations needing to manage complex data workflows across multiple schemas.Teams looking to enhance data governance and accessibility.Data engineers migrating existing pipelines to modern standards.

Limitations · remember

!Migration is not supported using Declarative Automation Bundles.

!Once migrated, pipelines cannot revert to the legacy publishing mode.

!Materialized views and streaming tables cannot be moved across schemas after creation.

Architecture

Lakeflow Pipelines → Default Publishing Mode → Data Management Layer

Get started

Access Pipelines

Click **Jobs & Pipelines** from the left sidebar in your workspace.

Select Pipeline

Click the name of the pipeline you want to migrate.

Pause Updates

Pause updates and ensure any running pipeline stops.

Configure Migration

Add the configuration `pipelines.enableDPMForExistingPipeline` set to `true`.

Start Update

Start a manual update and let it complete.

Key takeaway

Migrating to the default publishing mode enhances flexibility and supports complex data architectures.

2025-06-23-f9303d1e Generated by the Databricks Updates Agent · browse all updates