The Lakeflow Spark Declarative Pipelines documentation is reorganized: all conceptual topics are now grouped under Concepts.
The Lakeflow Spark Declarative Pipelines documentation is reorganized: all conceptual topics are now grouped under Concepts. Lakeflow Spark Declarative Pipelines (SDP) is a declarative framework for building batch and streaming data pipelines in SQL and Python. Its core concepts are pipelines, flows, streaming tables, materialized views, and sinks, which work together to process data with automatic orchestration and incremental updates.
Lakeflow SDP extends and is interoperable with Apache Spark Declarative Pipelines, while running on the performance optimized Databricks Runtime, and the Lakeflow Spark Declarative Pipelines flows API uses the same DataFrame API as Apache Spark and Structured Streaming.
The Lakeflow Spark Declarative Pipelines documentation is reorganized: all conceptual topics are now grouped under Concepts. Lakeflow Spark Declarative Pipelines (SDP) is a declarative framework for building batch and streaming data pipelines in SQL and…
Read the full documentation ↗