Auto Loader now reports batchSizeNumFiles, batchSizeNumBytes, and file processing states (numFilesProcessed, numFilesSkippedCorrupted, numFilesSkippedMissing, numFilesUnknownState) as metrics.
Auto Loader now reports batchSizeNumFiles, batchSizeNumBytes, and file processing states (numFilesProcessed, numFilesSkippedCorrupted, numFilesSkippedMissing, numFilesUnknownState) as metrics. BY NAME now supports the REPLACE WHERE clause. The configuration property spark.sql.xml.legacyXMLParser.enabled has been renamed to spark.sql.legacy.useLegacyXMLParser.
Reduces operational overhead and unlocks new capabilities for data teams on Databricks.
Auto Loader now reports batchSizeNumFiles, batchSizeNumBytes, and file processing states (numFilesProcessed, numFilesSkippedCorrupted, numFilesSkippedMissing, numFilesUnknownState) as metrics. BY NAME now supports the REPLACE WHERE clause.
Read the full documentation ↗