Moving tables between Spark Declarative Pipelines is now generally available.
This update allows users to move streaming tables and materialized views between Spark Declarative Pipelines in Lakeflow, enabling the destination pipeline to manage the table instead of the source. It supports various scenarios such as splitting large pipelines, merging multiple pipelines, and changing refresh frequencies. The feature requires specific Databricks Runtime versions and shared metastores between workspaces for cross-workspace moves.
This capability enhances flexibility in data pipeline management, allowing organizations to optimize their ETL processes by reorganizing tables without the need for extensive rewrites. It can lead to improved performance, reduced operational complexity, and better resource utilization, ultimately driving productivity and cost efficiency.