ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307

Is your feature request related to a problem or challenge?

ProjectionExec::project_statistics() only propagates column statistics for plain Column references and Literal values. Any ScalarFunctionExpr like get_field(), variant_get, or any UDF produces ColumnStatistics::new_unknown():

This means every column produced by a scalar function has Absent min/max/null_count/distinct_count, even when the function is a pure extraction (get_field, variant_get), a monotonic transformation (cast, abs), or any UDF where output stats are derivable from input stats

Describe the solution you'd like

Add an optional method to ScalarUdfImpl:

something like fn output_statistics(&self, input_statistics: &[ColumnStatistics]) -> Option<ColumnStatistics>

This way, ProjectionExec::project_statistics would call this before falling back to unknown

How this affects Struct + Variant queries

This matters for struct/variant queries especially, most output columns come through get_field or equivalent UDFs, so the cost model is effectively blind!

• cost based join ordering can't estimate cardinality for join keys produced by udfs
• aggregation planning can't estimate group count for group by on udf outputs
• filterexec selectivity - can't narrow row estimates for filters on udf outputs

Here's an MRE: https://github.com/apache/datafusion/compare/main...pydantic:datafusion:stats-propagation-mre?expand=1