Lakeflow Spark Declarative Pipelines now supports querying the event log for metrics about the progress of a stream.
Lakeflow Spark Declarative Pipelines now supports querying the event log for metrics about the progress of a stream. The pipeline event log contains all information related to a pipeline, including audit logs, data quality checks, pipeline progress, and data lineage. You can use the event log to track, understand, and monitor the state of your data pipelines.
SQL WITH latest update AS ( SELECT origin.pipeline id, origin.update id AS latest update id FROM event log raw AS origin WHERE origin.event type = 'create update' ORDER BY timestamp DESC LIMIT 1 remove if you want to get all of the update ids ), parsed planning AS ( SELECT origin.pipeline name, origin.pipeline id, origin.flow name, lu.latest update id, from json( details:planning information, 'struct< technique information: array
Lakeflow Spark Declarative Pipelines now supports querying the event log for metrics about the progress of a stream. The pipeline event log contains all information related to a pipeline, including audit logs, data quality checks, pipeline progress, and data…
Read the full documentation ↗