Auto Loader now supports automatic type widening with the new addNewColumnsWithTypeWidening schema evolution mode.
Auto Loader now supports automatic type widening with the new addNewColumnsWithTypeWidening schema evolution mode. This feature automatically widens data types (such as int to long or float to double) without requiring data rewrite or user intervention. Type widening is available for all file formats with schema evolution support in Auto Loader including text formats (JSON, CSV, XML) and binary formats (Avro, Parquet).
It also reduces pipeline maintenance by automatically handling complex schema changes.
Auto Loader now supports automatic type widening with the new addNewColumnsWithTypeWidening schema evolution mode. This feature automatically widens data types (such as int to long or float to double) without requiring data rewrite or user intervention.
Read the full documentation ↗