databricks Platform · Updated Aug 25, 2025

Automatic Liquid Clustering

Enhance query performance with intelligent clustering.

What it does

Automatic liquid clustering allows users to optimize query performance by using the `CLUSTER BY AUTO` syntax in Spark Declarative Pipelines on Lakeflow. This feature enables Databricks to intelligently select clustering keys based on the data characteristics, improving the efficiency of data retrieval. It is particularly beneficial for streaming tables, which support incremental data processing and can be refreshed on demand or via scheduling.

Why it matters

This update significantly enhances query performance, reducing latency and improving resource utilization, which can lead to cost savings. Data engineers and analysts will benefit from faster query responses, enabling more efficient data exploration and analysis. The automatic selection of clustering keys also simplifies the data management process, allowing teams to focus on insights rather than manual optimization.

Key capabilities

Automatic Clustering

Databricks automatically selects the best clustering keys to optimize performance.

Streaming Table Support

Enhances streaming tables with automatic clustering for better incremental data processing.

Improved Query Performance

Reduces query latency and improves resource efficiency through intelligent clustering.

Simplified Data Management

Eliminates the need for manual clustering key selection, streamlining data operations.

Architecture

Lakeflow → Spark Declarative Pipelines → Streaming Tables → Automatic Liquid Clustering

In short

You can now automatically optimize data clustering in Spark Declarative Pipelines to improve query performance.

Read the full documentation ↗

Key takeaway

Automatic liquid clustering optimizes query performance in Spark Declarative Pipelines, enhancing data processing efficiency.

2025-08-25-eb903d43 Generated by the Databricks Updates Agent · browse all updates