databricks Platform · Updated May 26, 2026

Spark Declarative Pipelines

Pipeline parameters are now in Beta, enhancing flexibility and reusability.

What it does

Pipeline parameters allow users to define key-value pairs at the pipeline level, enabling the reuse of the same pipeline source code across different environments or datasets. Users can declare default parameter values in pipeline settings and override them at runtime, referencing these parameters in SQL source code using named parameter syntax. This feature supports SQL source code only, while Python source code can utilize the Configuration field for parameterization.

Why it matters

This update significantly enhances the flexibility and efficiency of data pipelines, allowing organizations to streamline their data processing workflows across various environments without duplicating code. By enabling parameterization, teams can reduce operational overhead, improve productivity, and ensure consistent data transformations, which is crucial for maintaining data integrity and governance.

Key capabilities

Parameter Definition

Define default parameter values in pipeline settings for reuse across different runs.

Runtime Overrides

Override parameter values at runtime for specific updates without changing saved defaults.

SQL Integration

Reference parameters directly in SQL source code using named parameter syntax.

Job Parameter Pushdown

Automatically push job parameters down to pipeline tasks for dynamic execution.

Prerequisites

Workspace admin access to control feature visibility.

Familiarity with SQL and pipeline configuration.

Recommended use cases

Running the same transformations on different datasets (e.g., dev and prod).Dynamic ingestion from various source paths based on runtime parameters.Creating materialized views with flexible date ranges.

Limitations · remember

!Named parameters are only supported in SQL source code, not in Python.

!Pipelines can only run one update at a time, limiting concurrency.

!Date filters in SQL can trigger full refreshes if not managed properly.

Architecture

Lakeflow → Pipeline Configuration → SQL Source Code → Job Parameter Management

Get started

Access Pipeline Settings

Navigate to Jobs and Pipelines in the sidebar and select your pipeline.

Edit Parameters

In the pipeline settings sidebar, find the Parameters section and click Edit.

Define Key-Value Pairs

Add Key and Value entries for your parameters and click Save.

Reference in SQL

Use the defined parameters in your SQL source code with the named parameter syntax.

Override at Runtime

Use the Run with different settings option to override parameters for a specific update.

Key takeaway

The introduction of pipeline parameters in Beta enhances the flexibility and reusability of Spark Declarative Pipelines, enabling more efficient data processing workflows.

2026-05-26-spark-declarative-pipelines-on-lakeflow-parameters-are-now-in-beta Generated by the Databricks Updates Agent · browse all updates