Skip to content

Missing field metadata for projections during physical planning #23529

Closed

Description

@linhr

Describe the bug

DataFusion seems to drop field metadata for projections during physical planning. This may cause schema mismatch when the physical schema is later validated against the logical schema (e.g., when execution.skip_physical_aggregate_schema_check is false).

To Reproduce

Here is a simple test case to show this behavior.

#[cfg(test)]
mod tests {
    use std::collections::HashMap;
    use std::sync::Arc;

    use datafusion::arrow::datatypes::{DataType, Field, Schema};
    use datafusion::common::ToDFSchema;
    use datafusion::common::metadata::FieldMetadata;
    use datafusion::functions_aggregate::min_max::max_udaf;
    use datafusion::logical_expr::expr::AggregateFunction;
    use datafusion::logical_expr::{EmptyRelation, Expr, LogicalPlan};
    use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner};
    use datafusion::prelude::{SessionContext, col};
    use datafusion_expr::logical_plan::{Aggregate, Projection};

    #[tokio::test]
    async fn test_aggregate_planning_fails_on_projection_with_field_metadata() {
        let schema = Schema::new(vec![Field::new("value", DataType::Utf8, false)]);
        let input = LogicalPlan::EmptyRelation(EmptyRelation {
            produce_one_row: false,
            schema: Arc::new(schema.to_dfschema().unwrap()),
        });
        let metadata = FieldMetadata::from(HashMap::from([("foo".to_string(), "bar".to_string())]));
        let projection = LogicalPlan::Projection(
            Projection::try_new(
                vec![col("value").alias_with_metadata("value", Some(metadata))],
                Arc::new(input),
            )
            .unwrap(),
        );
        let aggregate = LogicalPlan::Aggregate(
            Aggregate::try_new(
                Arc::new(projection),
                vec![],
                vec![Expr::AggregateFunction(AggregateFunction::new_udf(
                    max_udaf(),
                    vec![col("value")],
                    false,
                    None,
                    vec![],
                    None,
                ))],
            )
            .unwrap(),
        );

        DefaultPhysicalPlanner::default()
            .create_physical_plan(&aggregate, &SessionContext::new().state())
            .await
            .unwrap();
    }
}

The test would fail with the following error:

called `Result::unwrap()` on an `Err` value: Internal("Physical input schema should be the same as the one converted from logical input schema. Differences: \n\t- field metadata at index 0 [value]: (physical) {} vs (logical) {\"foo\": \"bar\"}")

Expected behavior

DefaultPhysicalPlanner should handle field metadata in the logical plan, possibly inside create_project_physical_exec_with_props.

Additional context

No response

Activity

  1. added theissue type on Jul 13, 2026
  2. mkleen commented on Jul 14, 2026

    @mkleen
    Contributor

    take

  3. mkleen commented on Jul 29, 2026

    @mkleen
    Contributor

    untake

  4. added a commit that references this issue on Aug 5, 2026
    a0124a4
  5. added a commit that references this issue on Aug 12, 2026
    16dbb9d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions