You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
JSON sources (FileSourceScanExec with JsonFileFormat, and V2 JsonScan) are read by Spark. spark.comet.convert.json.enabled can convert Spark's rows to Arrow, but parsing still runs on the JVM.
Describe the potential solution
Add a native JSON scan that matches Spark's JSON reader:
user-supplied schema
PERMISSIVE, DROPMALFORMED and FAILFAST modes, and columnNameOfCorruptRecord
timestamp and date formats with the session timezone
multi-line JSON and compression codecs
fall back for the options it can't match
Additional context
Native from_json already exists (opt-in). It may share parsing code with the scan.
Activity