Repository navigation
[Format] Add an Arrow Canonical Extension Type for Parquet Variant #46908
Description
Describe the enhancement requested
Parquet has added a new type for semi-structured data called Variant which is defined here:
- Variant encoding spec: https://github.com/apache/parquet-format/blob/main/VariantEncoding.md
- Variant shredding spec: https://github.com/apache/parquet-format/blob/main/VariantShredding.md
As it is common for engines to read data from Parquet into Arrow for in memory processing it is useful to have support for Variant in Arrow. @CurtHagenlocher proposes adding native Variant support in the Arrow format itself here:
An alternate approach is to add a Canonical Extension Type
@zeroshade wrote up a proposal
- Mailing List Discussion: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
- Google Document: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
And implemented an implementation in Go
This ticket tracks the idea of adding Variant as an official extension type
See also @neilechao 's PR to add variant read support to parquet
Component(s)
Format
Reacted by nicpetit
Activity
alamb commentedon Jun 25, 2025
Most recent update from @zeroshade on the mailing list is:
[-]Add a Canonical Variant Extension Type[/-][+][Format] Add a Canonical Variant Extension Type[/+][-][Format] Add a Canonical Variant Extension Type[/-][+][Format] Add an Arrow Canonical Extension Type for Parquet Variant[/+]GH-46908: [Docs][Format] Add variant extension type docs (#47456)
ianmcook commentedon Sep 17, 2025
Closed by #47456
apacheGH-46908: [Docs][Format] Add variant extension type docs (apach…
docs(variant): link VariantArray doc to official Parquet Variant exte…
apacheGH-46908: [Docs][Format] Add variant extension type docs (apach…
docs(variant): link VariantArray doc to official Parquet Variant exte…