All struct-aware optimizations are hardcoded to GetFieldFunc

Is your feature request related to a problem or challenge?

Every optimization for struct field querying (row level pushdown, leaf-level projection pruning, PushdownChecker, resolve_struct_field_leaves) is gated on the GetFieldFunc udf.

A custom UDF like variant_get gets none of these optimizations. There's no trait or capability API for a UDF to declare "I'm a struct field access."

This blocks Variant support entirely unless variant_get decomposes to get_field chains or the Parquet source learns a general mechanism for struct-access UDFs

There's no trait or capability API for a UDF to declare "I'm a struct field access."

There is:

datafusion/datafusion/expr/src/udf.rs

Lines 972 to 984 in 1416ed4

    
               /// Returns placement information for this function. 
        
               /// 
        
               /// This is used by optimizers to make decisions about expression placement, 
        
               /// such as whether to push expressions down through projections. 
        
               /// 
        
               /// The default implementation returns [`ExpressionPlacement::KeepInPlace`], 
        
               /// meaning the expression should be kept where it is in the plan. 
        
               /// 
        
               /// Override this method to indicate that the function can be pushed down 
        
               /// closer to the data source. 
        
               fn placement(&self, _args: &[ExpressionPlacement]) -> ExpressionPlacement { 
        
                   ExpressionPlacement::KeepInPlace 
        
               }

I have a query calling a UDF that returns a struct (e.g., fields a and b in the struct). If I manually unnest the struct (e.g., func(x)['a'] and func(x)['b']), datafusion will call my UDF twice and can not optimize it into one call (if I'm reading the execution plan and metrics correctly). It will do the same even if I take the UDF call to a CTE or separate view, so it seems more like an issue with optimization.

I'm not sure if this is the root cause of this behavior.

@liningpan

@liningpan I don't think that's related to this issue, but it is a very valid point. Could you maybe open an issue to track that? Off the top of my head it feels like something the CSE (Common Sub-expression Elimination) optimizer might be able to handle. A good first step would be an MRE / benchmark showing the perf issue (you could write a UDF that has a sleep in it or something).

Navigation Menu

Repository navigation

All struct-aware optimizations are hardcoded to `GetFieldFunc` #21306

Is your feature request related to a problem or challenge?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/// Returns placement information for this function.
	///
	/// This is used by optimizers to make decisions about expression placement,
	/// such as whether to push expressions down through projections.
	///
	/// The default implementation returns [`ExpressionPlacement::KeepInPlace`],
	/// meaning the expression should be kept where it is in the plan.
	///
	/// Override this method to indicate that the function can be pushed down
	/// closer to the data source.
	fn placement(&self, _args: &[ExpressionPlacement]) -> ExpressionPlacement {
	ExpressionPlacement::KeepInPlace
	}

All struct-aware optimizations are hardcoded to GetFieldFunc #21306

Description

Is your feature request related to a problem or challenge?

Activity

adriangb commented on Apr 1, 2026

liningpan commented on Jul 16, 2026

adriangb commented on Jul 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

All struct-aware optimizations are hardcoded to `GetFieldFunc` #21306