Query based connectors in Lakeflow Connect are now generally available.
Query based connectors in Lakeflow Connect are now generally available. Query based connectors ingest data from databases by querying the source directly using a cursor column, without requiring change data capture (CDC) configuration or an ingestion gateway. Supported sources include Oracle, Teradata, SQL Server, MySQL, MariaDB, and PostgreSQL, as well as all Lakehouse Federation data sources using foreign catalog ingestion.
The trade off is that query performance can be slower and queries run directly on source tables, which can place more load on the source database compared to CDC connectors that query the binlog.
Query based connectors in Lakeflow Connect are now generally available. Query based connectors ingest data from databases by querying the source directly using a cursor column, without requiring change data capture (CDC) configuration or an ingestion gateway.
Read the full documentation ↗