Skip to content

Native JSON file scan #7429

Open
Open

Description

@parthchandra

What is the problem the feature request solves?

JSON sources (FileSourceScanExec with JsonFileFormat, and V2 JsonScan) are read by Spark. spark.comet.convert.json.enabled can convert Spark's rows to Arrow, but parsing still runs on the JVM.

Describe the potential solution

Add a native JSON scan that matches Spark's JSON reader:

  • user-supplied schema
  • PERMISSIVE, DROPMALFORMED and FAILFAST modes, and columnNameOfCorruptRecord
  • timestamp and date formats with the session timezone
  • multi-line JSON and compression codecs
  • fall back for the options it can't match

Additional context

Native from_json already exists (opt-in). It may share parsing code with the scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions