Repository navigation
Missing field metadata for projections during physical planning #23529
Copy link
Copy link
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
DataFusion seems to drop field metadata for projections during physical planning. This may cause schema mismatch when the physical schema is later validated against the logical schema (e.g., when execution.skip_physical_aggregate_schema_check is false).
To Reproduce
Here is a simple test case to show this behavior.
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ToDFSchema;
use datafusion::common::metadata::FieldMetadata;
use datafusion::functions_aggregate::min_max::max_udaf;
use datafusion::logical_expr::expr::AggregateFunction;
use datafusion::logical_expr::{EmptyRelation, Expr, LogicalPlan};
use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner};
use datafusion::prelude::{SessionContext, col};
use datafusion_expr::logical_plan::{Aggregate, Projection};
#[tokio::test]
async fn test_aggregate_planning_fails_on_projection_with_field_metadata() {
let schema = Schema::new(vec![Field::new("value", DataType::Utf8, false)]);
let input = LogicalPlan::EmptyRelation(EmptyRelation {
produce_one_row: false,
schema: Arc::new(schema.to_dfschema().unwrap()),
});
let metadata = FieldMetadata::from(HashMap::from([("foo".to_string(), "bar".to_string())]));
let projection = LogicalPlan::Projection(
Projection::try_new(
vec![col("value").alias_with_metadata("value", Some(metadata))],
Arc::new(input),
)
.unwrap(),
);
let aggregate = LogicalPlan::Aggregate(
Aggregate::try_new(
Arc::new(projection),
vec![],
vec![Expr::AggregateFunction(AggregateFunction::new_udf(
max_udaf(),
vec![col("value")],
false,
None,
vec![],
None,
))],
)
.unwrap(),
);
DefaultPhysicalPlanner::default()
.create_physical_plan(&aggregate, &SessionContext::new().state())
.await
.unwrap();
}
}The test would fail with the following error:
called `Result::unwrap()` on an `Err` value: Internal("Physical input schema should be the same as the one converted from logical input schema. Differences: \n\t- field metadata at index 0 [value]: (physical) {} vs (logical) {\"foo\": \"bar\"}")
Expected behavior
DefaultPhysicalPlanner should handle field metadata in the logical plan, possibly inside create_project_physical_exec_with_props.
Additional context
No response
Reacted by Martin
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
mkleen commentedon Jul 14, 2026
take
mkleen commentedon Jul 29, 2026
untake
fix: preserve projection field metadata during physical planning (apa…
fix: preserve projection field metadata during physical planning (apa…