Skip to content

Preserve field metadata across expressions in logical plans #6886

Closed

Description

@sydduckworth

Is your feature request related to a problem or challenge?

I'm working on a project that requires using Arrow extension types. Because of the way extension types are encoded in arrow-rs (see apache/arrow-rs#4472) I have to keep track of type information at the field/schema level. This works at execution time, but I've run into an issue trying to extract metadata from a field during the planning phase as field metadata isn't being propagated.

This seems to originate in the ExprSchemable::to_field trait, where when a field is constructed from an expression it only copies the datatype and nullability from the input schema.

Describe the solution you'd like

A reasonable approach to me would be to add a metadata method to the ExprSchemable trait that would forward field metadata for type-preserving expressions (e.g. column, alias, etc.) and return empty metadata for others.
ExprSchemable::to_field could then be updated to pass the metadata to the DFField constructor.

Describe alternatives you've considered

The proposed solution would enable metadata propagation through expressions. I haven't gone through every logical plan node so it's possible I'm missing other places where metadata would be erased that would also need to be changed.

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions