databricks Platform · Updated Jul 1, 2026

MySQL Integrated CDC Pipeline

Simplified change data capture for MySQL in Databricks.

What it does

The integrated CDC pipeline for MySQL allows users to ingest change data directly into Databricks using a single pipeline, which combines both extraction and application stages. This approach eliminates the need for a separate ingestion gateway, simplifying the setup process. The pipeline supports scheduled execution and auto full refresh, enhancing data management capabilities.

Why it matters

This update streamlines the data ingestion process from MySQL, reducing operational complexity and setup time. By consolidating the pipeline architecture, organizations can achieve faster data updates and better resource utilization, ultimately leading to improved decision-making and analytics capabilities.

Key capabilities

Unified Pipeline Architecture

Runs both extraction and application in a single pipeline, eliminating the need for a separate ingestion gateway.

Scheduled Execution

Supports triggered execution, allowing users to schedule data ingestion at regular intervals.

Auto Full Refresh

Automatically supports full refreshes, a feature not available in the standard gateway-based architecture.

Exactly-Once Semantics

Ensures data integrity with exactly-once processing during ingestion.

Prerequisites

Workspace must be enabled for Unity Catalog.

User must have necessary privileges on the metastore and target catalog/schema.

MySQL source setup must be completed.

Recommended use cases

Organizations needing simplified MySQL data ingestion.Data teams requiring scheduled updates for analytics.Use cases involving auto full refresh for data accuracy.

Limitations · remember

!Continuous execution is not supported; only scheduled execution is available.

!Serverless compute is not supported; classic compute is required.

Architecture

Unity Catalog Connection → Integrated CDC Pipeline → Data Staging Volume → Destination Streaming Tables

Get started

Create Unity Catalog Connection

Establish a connection to MySQL in Unity Catalog.

Define Pipeline Resource

Create the integrated CDC pipeline resource using YAML or API.

Schedule Pipeline Execution

Set up a Lakeflow Jobs task to trigger the pipeline on a schedule.

Deploy Pipeline

Use Databricks CLI to deploy and run the pipeline.

Monitor Pipeline Updates

Check for successful execution and manage any queued updates.

Key takeaway

The MySQL integrated CDC pipeline simplifies data ingestion, enabling efficient and reliable change data capture directly into Databricks.

2026-07-01-mysql-integrated-cdc-pipeline-is-now-in-beta Generated by the Databricks Updates Agent · browse all updates