[EPIC] Variant on DataFusion #26517

Is your feature request related to a problem or challenge?

OpenSearch is adding a Variant field type to its Parquet-backed indices. The query engine is DataFusion, the writer and Variant kernels are arrow-rs, and the SQL functions are datafusion-contrib/datafusion-variant. Files mix shredded and unshredded layouts within one table, and the same table is read by Spark.

This epic lists the core hooks that work still needs, following the split in #16116: generic extension points in core, Variant-specific implementations in the crate. It sits under #16116 and #21308. Items already tracked elsewhere are linked rather than restated; the new issues are the ones marked new.

Spelling the type in SQL (section 6) already has issues filed on DF:

• Implement DFExtensionType for Arrow's Canonical Extension Types #21144
• Support Mapping SQL Types to the Canonical Extension Types #21626
• Support pluggable cast resolution to support casting to and from extension types #20755

They are linked here so the epic covers the whole core surface Variant needs.

Describe the solution you'd like

1. Read only the leaves a UDF needs

variant_get(v, '$.x') reads and decodes the whole Variant group because the Parquet planner only trusts get_field. Two PRs address this: #25013 (two hooks on ScalarUDFImpl) and draft #26282 (NestedInputAccess capability, also used by get_field, with row group and page pruning on the declared leaves). Our requirement for whichever shape lands: a function must be able to require more than one leaf for one argument (the typed leaf and the value fallback leaf), and pruning must keep any container that holds a fallback row. #26282 does both.

• Review feat: expose UDF field access and input requirements for Parquet pruning #25013 / feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282 against shredded and unshredded test data and help converge on one shape
• Row group and page pruning on a declared leaf, with the fallback guard: Support row group pruning for struct field predicates #20871. feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282 includes it; if the work is split, this is the successor to feat: expose UDF field access and input requirements for Parquet pruning #25013
• datafusion-variant: implement the capability for variant_get once it lands

2. Keep the extension tag through planning

A Variant is a struct whose field metadata carries ARROW:extension:name = arrow.parquet.variant. Two core paths drop it, after which variant_get sees a plain struct and rejects the column.

• new Parquet schema inference clears all field metadata before Schema::try_merge when skip_metadata = true (the default, config.rs#L1343), so any ListingTable over more than one file loses the tag. The drop is clear_metadata in datasource-parquet/src/file_format.rs: every top-level Field is rebuilt with_metadata(Metadata::new()), which is exactly where the Variant tag lives. It is called from ParquetFormat::infer_schema. Reproduced on 54. Proposed fix: keep the ARROW:extension:* keys recursively and drop the rest; the extension keys are the type's identity, not writer annotation.
• CASE / COALESCE build the output Field from the DataType alone: CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860 (reported on geoarrow.wkb, same mechanism). Drop sites:
  • COALESCE: CoalesceFunc::return_field_from_args takes data_type() and is_nullable() from the argument fields and returns Field::new(name, return_type, nullable); the argument fields' metadata is never read.
  • CASE, logical plan: ExprSchemable::to_field puts Expr::Case in the catch-all arm that returns Field::new(name, get_type()?, nullable()?) with no metadata.
  • CASE, physical plan: CaseExpr (case.rs#L1259) does not override return_field, so it inherits the PhysicalExpr::return_field default, which is again Field::new(name, data_type()?, nullable()?).

3. Tables that mix shredded and unshredded files

When one file stores v as Struct<metadata, value> and another as Struct<metadata, value, typed_value: Struct<x: Int64>>, the merged table schema has typed_value, and the default adapter null-fills it for the unshredded file. The fill never touches the files; it is an in-memory rewrite of the read expression, done once per file at open, in DefaultPhysicalExprAdapter (physical-expr-adapter/src/schema_rewriter.rs):

• Reading v whole: rewrite_column sees the file field differs from the table field and wraps the column in a CastExpr from the file struct to the table struct; the arrow struct cast adds typed_value as an all-null child at batch time.
• Reading v['typed_value'] from a file that lacks it: try_rewrite_struct_field_access replaces the access with a null Literal; try_narrow_struct_cast does the same when the access goes through an adapter-inserted cast.
• The batch path applies the same rewrite to every RecordBatch: BatchAdapter::adapt_batch.

For an ordinary struct that is correct. For Variant the shredding spec makes value = object, typed_value = null an invalid row, so a compliant reader returns NULL for every field of every row of that file, silently. Reproduced: a two-file table returned 0 of 1024 unshredded rows. Files shredded on different keys fail differently, at open, with target field is non-nullable but missing from source, raised from the same place (rewrite_column).

The constraint we would like core to hold: a file is read in its own layout and reconstructed, never null-filled into a layout it does not have. Two ways to satisfy it: per-file required_fields answers (#26282 evaluates them against each file's schema at open, but states that "adapter-inserted struct casts are handled as before", so it narrows the read without removing the cast; whether variant_get then sees the file struct or the cast struct is to be confirmed), or a hook for a registered extension type to supply its own per-file adaptation, which satisfies the constraint by construction. The Variant adapter itself would live in the crate. Named earlier on #16116 (2025-07-15) and #22908 (2026-09-08).

• new Per-file adaptation for extension types, so an unshredded file is never null-filled into the shredded layout: per-file required_fields answers (feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282) or an extension-type adapter hook, with the Variant adapter in the crate

4. Statistics the optimizer can use

ColumnStatistics is flat, so a nested leaf (shredded or not) and any UDF output have no planner statistics. Two designs are open for the UDF half and one has to be picked; #26282 adds a container-level output_statistics for pruning, which is a third data point.

• Nested and extension-type leaves in Statistics: Nested types and extension types are not supported in Statistics / ColumnStatistics #20707
• UDF output statistics: ScalarUDFImpl::output_statistics (ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307) or ExpressionAnalyzer (Add ExpressionAnalyzer for pluggable expression-level statistics estimation #21122). We implement whichever is chosen; the crate half (typed-leaf min/max, fallback-leaf null count) is the same code either way.

5. Variant value semantics

GROUP BY, DISTINCT, joins and ORDER BY on a Variant column are undefined today. Spark rejects all of them at analysis time: OrderUtils.isOrderable returns false for VariantType, which fails =/< and join keys in BinaryComparison.checkInputDataTypes (DATATYPE_MISMATCH.INVALID_ORDERING_TYPE), ORDER BY in CheckAnalysis (EXPRESSION_TYPE_IS_NOT_ORDERABLE), GROUP BY (GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE), and set operations in CheckAnalysis, where the comment reads "remove this type check once we support Variant ordering". DuckDB 1.5 allows them but does not document the rule (Variant docs cover storage, variant_typeof, extraction and Parquet only); probed on 1.5: =, GROUP BY, DISTINCT and ORDER BY each apply a different rule (equal numbers of different width compare equal but group apart; ORDER BY is by inner type tag, so 7.0 sorts after 42).

• new, after design discussion A canonical-bytes normalisation kernel in arrow-rs so equal values have equal bytes, and a variant_normalize function in the crate on top of it, so hashing operators work without each inventing a rule. For ORDER BY our lean is to refuse at plan time as Spark does. Issue to follow once the SQL semantics are written up.

6. Spelling VARIANT in SQL

The generic machinery is merged: ExtensionTypeRegistry and DFExtensionType (#20312, DataFusion 54), TypePlanner (#20676, 54), and metadata-preserving CAST (#23169, 55.1.0 via backport #24875). What is missing is the Variant entries. Without them a Variant prints as raw struct bytes in datafusion-cli, and CREATE TABLE t (v VARIANT) or CAST(x AS VARIANT) cannot be planned. Spark 4.2 and DuckDB 1.5 both spell the type VARIANT in CREATE TABLE and CAST, so this is what lets a query written for them run unchanged.

• Register arrow.parquet.variant as a DFExtensionType: storage_type() fixes the struct shape, create_array_formatter() gives JSON display in place of raw struct bytes (as DFUuid does), and the registry entry is the identity the other hooks key on instead of re-parsing ARROW:extension:name from field metadata. The one unticked box in Implement DFExtensionType for Arrow's Canonical Extension Types #21144. Small; the natural first PR
• Map the SQL name VARIANT to that type via TypePlanner: Support Mapping SQL Types to the Canonical Extension Types #21626, same shape as UUID and JSON. convert_data_type_to_field asks TypePlanner::plan_type_field first; the Variant planner returns a FieldRef whose DataType is the Variant struct and whose metadata carries ARROW:extension:name = arrow.parquet.variant, taken from the DFExtensionType above. There is no DataType::Variant; the type is struct plus tag. The crate-side registration goes with it
• Pluggable cast resolution to and from extension types (CAST(x AS VARIANT), v::INT): Support pluggable cast resolution to support casting to and from extension types #20755, after the registration lands and a maintainer design. Not on our critical path

Adjacent, in arrow-rs

• The Parquet writer omits specification_version from the VARIANT logical type (LogicalType::variant(None) in logical_type_for_struct), although the Parquet spec requires the version in the declaration. Spark 4.1 only accepts version 1 and rejects the file with PARQUET_TYPE_NOT_RECOGNIZED; arrow-rs itself reads the file fine, so the gap only shows cross-engine. One-line fix: [Variant] Parquet writer omits specification_version on the VARIANT logical type arrow-rs#11917

Describe alternatives you've considered

Keeping all of this inside OpenSearch (a fork of the Parquet reader and a private variant_get). Rejected: every hook above is generic to extension types, and the mixed-layout and statistics problems are hit by any Variant reader on DataFusion, not just ours.

Additional context

Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires.

Open questions for maintainers:

1. Are items 2 and 3 accepted as core concerns (neither mentions Variant), with the Variant adapter shipped from the crate?
2. For item 1, which shape: feat: expose UDF field access and input requirements for Parquet pruning #25013, feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282, or a merge of the two?
3. For item 4, ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307 or Add ExpressionAnalyzer for pluggable expression-level statistics estimation #21122?