Repository navigation
Native CSV scan for V1 FileSourceScanExec #7430
Description
What is the problem the feature request solves?
The native CSV scan (spark.comet.scan.csv.v2.enabled) covers DataSource V2 CSVScan only. Most CSV reads plan V1 FileSourceScanExec with CSVFileFormat, which Spark reads.
Describe the potential solution
Route V1 CSV file-source scans to the native CSV reader, with the same fallback rules as the V2 path (corrupt-record column, inferSchema, multi-character delimiters, non-UTC timestamp parsing). Then decide whether native CSV can become default.
Additional context
Related: #7243, which changes how native CSV scan tasks get their file groups.
Activity