You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Preserve field metadata across expressions in logical plans #6886
Is your feature request related to a problem or challenge?
I'm working on a project that requires using Arrow extension types. Because of the way extension types are encoded in arrow-rs (see apache/arrow-rs#4472) I have to keep track of type information at the field/schema level. This works at execution time, but I've run into an issue trying to extract metadata from a field during the planning phase as field metadata isn't being propagated.
This seems to originate in the ExprSchemable::to_field trait, where when a field is constructed from an expression it only copies the datatype and nullability from the input schema.
Describe the solution you'd like
A reasonable approach to me would be to add a metadata method to the ExprSchemable trait that would forward field metadata for type-preserving expressions (e.g. column, alias, etc.) and return empty metadata for others. ExprSchemable::to_field could then be updated to pass the metadata to the DFField constructor.
Describe alternatives you've considered
The proposed solution would enable metadata propagation through expressions. I haven't gone through every logical plan node so it's possible I'm missing other places where metadata would be erased that would also need to be changed.
Activity