Repository navigation
Open Variant Type for semi-structured data #10987
Description
I've been starting to experiment with implementing the Open Variant Type 1 in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake 2 and Iceberg 3. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type 4.
I'm working on creating an extension similar to datafusion-functions-json. If we could create a new repo datafusion-functions-variant, I'd be happy to develop that in the open.
Footnotes
Reacted by Samuel Colvin, Adrian Garcia Badaracco, Jay Zhan, advancedxy, Trent Hauck, Weny Xu, Piotr Findeisen and ruslandoga
Metadata
Metadata
Assignees
Labels
No labels
Activity
alamb commentedon Jun 18, 2024
@samuelcolvin and @adriangb and I were speaking about datafusion-functions-json just today. They may be interested too
I have created https://github.com/datafusion-contrib/datafusion-functions-variant and added you as an admin 🚀
samuelcolvin commentedon Jun 18, 2024
Interesting, this is exactly what we need.
alamb commentedon Apr 28, 2025
FWIW the variant type has been added to the parquet spec, and is now much closer to standardization and we are beginning work on the support at the parquet layer. See epic
ianmcook commentedon May 20, 2025
There's a discussion happening on the Arrow dev mailing list about adding Variant as a canonical extension type in the Arrow spec. Input from the DataFusion developer community would be much appreciated.