The evolution of DLT into a robust framework for data pipelines.
Lakeflow Spark Declarative Pipelines (SDP) is a framework designed for creating both batch and streaming data pipelines using SQL and Python. It extends Apache Spark Declarative Pipelines and operates on the optimized Databricks Runtime, enabling efficient data ingestion from various sources like cloud storage and message buses. Users can develop, configure, and monitor pipelines, facilitating incremental transformations and data flows.
This update enhances data engineering capabilities by providing a more powerful and flexible framework for building data pipelines, which can lead to improved performance and reduced operational complexity. Organizations can benefit from streamlined data ingestion and transformation processes, ultimately driving better insights and decision-making.
The DLT product has been rebranded as Spark Declarative Pipelines on Lakeflow, offering enhanced capabilities for data pipelines.
Read the full documentation ↗