databricks Platform · Updated Jul 14, 2026

Lakeflow Connect

Query-based connectors now default for compliance workspaces.

What it does

Query-based connectors in Lakeflow Connect enable data ingestion from various databases by directly querying the source using a cursor column, eliminating the need for change data capture (CDC) configurations. This method supports databases like Oracle, SQL Server, and PostgreSQL, and allows for ingestion without an ingestion gateway or staging volume. The connectors operate on a scheduled basis, retrieving only the latest changes since the last run, and they can utilize Unity Catalog connections and Lakehouse Federation for broader source compatibility.

Why it matters

This update enhances data ingestion efficiency and flexibility for organizations using compliance security profiles, allowing them to easily integrate data from multiple sources without complex configurations. It reduces operational overhead by removing the need for CDC setups and staging areas, which can lead to cost savings and improved performance. Businesses can benefit from faster data availability and simplified data management processes.

Key capabilities

Direct Database Querying

Ingest data directly from databases using a cursor column without requiring CDC configurations.

Scheduled Ingestion

Run ingestion pipelines on a defined schedule, capturing only the latest changes.

Broad Source Compatibility

Support for multiple databases including Oracle, SQL Server, and PostgreSQL, as well as Lakehouse Federation sources.

Flexible History Tracking

Support for various history tracking modes, including SCD Type 1 and Type 2.

Prerequisites

Workspace must have the compliance security profile enabled.

Compute environment must allow network connectivity to the source database.

Recommended use cases

Ingesting data from multiple databases into a unified data lake.Tracking changes in source data without complex CDC setups.Implementing slowly changing dimensions in data warehousing.

Limitations · remember

!Query performance may be slower compared to CDC connectors.

!Queries run directly on source tables, potentially increasing load on the source database.

Architecture

Lakeflow Connect → Unity Catalog → Source Database → Streaming Tables

In short

Query-based connectors will be available by default for workspaces with compliance security profiles, simplifying data ingestion from various databases.

Read the full documentation ↗

Key takeaway

Query-based connectors streamline data ingestion for compliance-focused workspaces, enhancing efficiency and reducing complexity.

2026-07-14-31f6c6e5 Generated by the Databricks Updates Agent · browse all updates