databricks Platform · Updated Jul 2, 2026

Lakeflow Connect Row Filtering

Enhance data ingestion efficiency with targeted row filtering.

What it does

Row filtering in Lakeflow Connect allows users to ingest only the rows that meet specified conditions, similar to a SQL `WHERE` clause. This filtering occurs on the source side of the connector, ensuring that only relevant data is transferred, staged, or stored. It applies during both the initial load and subsequent incremental updates, maintaining consistency over time and reducing unnecessary data movement.

Why it matters

This feature significantly improves performance and reduces costs by minimizing data duplication and storage needs. By allowing organizations to ingest only the necessary data, it enhances productivity and streamlines data management processes, making it easier to maintain clean and relevant datasets.

Key capabilities

Conditional Row Ingestion

Define boolean expressions to filter rows based on specific conditions.

Incremental Updates

Filters are applied consistently during both initial loads and incremental updates.

Source-side Filtering

Filtering occurs before data transfer, reducing unnecessary data movement.

Support for Multiple Connectors

Compatible with SaaS connectors like Google Analytics, Salesforce, and ServiceNow.

Prerequisites

A Lakeflow Connect ingestion pipeline on a supported connector.

Unity Catalog enabled for the destination catalog and schema.

Permission to edit the ingestion pipeline definition.

Recommended use cases

Ingest only rows for a specific region, business unit, or active-record flag.Exclude test or archived records from a SaaS system.Reduce cost and duplication for very large source tables.

Limitations · remember

!Filters can only reference columns available on the source object; joins across objects are not supported.

!Changing a filter after the initial load may require a full refresh of the affected object.

!Non-deterministic source functions are not supported inside a filter.

Architecture

Lakeflow Connect → Source Connectors → Data Transfer → Data Storage → Unity Catalog

Get started

Open Ingestion Pipeline

Access your Lakeflow Connect ingestion pipeline.

Select Source Object

Choose the source object you want to filter.

Add Row Filter

Define a row filter condition using the object's columns.

Validate Filter

Check the filter against a sample of source rows.

Run Pipeline

Execute the pipeline to apply the filter to the initial load and incremental updates.

Key takeaway

Lakeflow Connect's row filtering feature enables efficient data ingestion by allowing users to selectively ingest only the necessary rows, improving performance and reducing costs.

2026-07-02-lakeflow-connect-row-filtering-is-now-generally-available Generated by the Databricks Updates Agent · browse all updates