The @dp.update flow decorator in Lakeflow Spark Declarative Pipelines lets you add an update flow that writes to a sink in update output mode, emitting only the rows that change in each batch.
The @dp.update flow decorator in Lakeflow Spark Declarative Pipelines lets you add an update flow that writes to a sink in update output mode, emitting only the rows that change in each batch. Unlike append flows, update flows support stateful aggregations without requiring a watermark. Use the @dp.update flow decorator to create an update flow.
Reduces operational overhead and unlocks new capabilities for data teams on Databricks.
The @dp.update flow decorator in Lakeflow Spark Declarative Pipelines lets you add an update flow that writes to a sink in update output mode, emitting only the rows that change in each batch. Unlike append flows, update flows support stateful aggregations…
Read the full documentation ↗