Add SessionConfig reference to ScalarFunctionArgs

@joseph-isaacs

Is your feature request related to a problem or challenge?

Now that Add ScalarUDFImpl::invoke_with_args to support passing the return type created for the udf instance #13290 is merged (thanks @joseph-isaacs) we have a place to add new

@Omega359 noted that by adding SessionConfig to the ScalarFunctionArgs unblock several tasks such as

Describe the solution you'd like

Add &SessionConfig to ScalarFunctionArgs, and ideally add a test that shows the config gets through.

Describe alternatives you've considered

No response

Additional context

We should try and get this done before DataFusion 44 is released so it isn't a breaking change

Release DataFusion 44.0.0 #13334

Well, small issue. ScalarFunctionArgs is in datafusion-expr which can't use SessionConfig directly since it's in datafusion-execution which depends on datafusion-expr which would cause a circular dependency. We can use ConfigOptions there which would give us access to all config except the opaque extensions in SessionConfig which I think is acceptable.

Great work!

Started delving into this trying to find a good way to impl. Trying to add config_options: ConfigOptions to ScalarFunctionExpr Having lots of fun with eq and hash with SessionConfig and f64 atm.

I wonder if we can use this trait: https://docs.rs/datafusion/latest/datafusion/catalog/trait.Session.html

We could but it wouldn't solve the issue. The issue is PhysicalExpr requires implementations to impl Eq and Hash or to have implementations for DynEq and DynHash. That is fine until something like SessionConfig which doesn't is introduced. I am looking at updating the either the 'config_namespace' macro or add explicit implementations for 'ScalarFunctionExpr' to try and impl either of the above and use f64.to_bits() and f64::from_bits(..) to handle the problematic f64's in the config. I think it's possible

It makes sense to me to move SessionConfig to common or common-runtime crate

take

It makes sense to me to move SessionConfig to common or common-runtime crate

I agree

~~I mistyped above - I meant we couldn't use SessionContext, not SessionConfig. Sorry about the confusion.~~ ... or not. I need more caffeine today. PR uses ConfigOptions.

I guess we can also add nullable info to ScalarFunctionArgs #11923

I guess we can also add nullable info to ScalarFunctionArgs #11923

It might already be present in ScalarFunctionArgs::data_type 🤔 :

datafusion/datafusion/expr/src/udf.rs

Line 336 in f2de2c4

pub return_type: &'a DataType,

Not really, DataType has no nullable info, we have to send nullable to ScalarFunctionArgs

datafusion/datafusion/physical-expr/src/scalar_function.rs

Lines 148 to 153 in 6c9355d

    
           // evaluate the function 
        
           let output = self.fun.invoke_with_args(ScalarFunctionArgs { 
        
               args, 
        
               number_rows: batch.num_rows(), 
        
               return_type: &self.return_type, 
        
           })?;

        // evaluate the function
        let nullable = self.nullable;
        let output = self.fun.invoke_with_args(ScalarFunctionArgs {
            args: inputs.as_slice(),
            number_rows: batch.num_rows(),
            return_type: &self.return_type,
            nullable,
        })?;

Not really, DataType has no nullable info, we have to send nullable to ScalarFunctionArgs

Ah I was thinking about the nullable info that is part of embedded Fields -- but that only affects Lists/Maps, etc

Add SessionConfig reference to ScalarFunctionArgs

This is pretty heavy dependency.
SessionConfig contains a LOT of information no scalar function should depend on and very little that has legitimate usage.
I would prefer not adding SessionConfig to ScalarFunctionArgs. Rather, we could add a stripped down config object dedicated to scalar functions. If time zone is all we need, we should add the time zone directly to ScalarFunctionArgs (like proposed in #16573).

https://docs.rs/datafusion/latest/datafusion/execution/config/struct.SessionConfig.html is mostly a wrapper around the ConfigOptions

If we just want specific fields, we could perhaps reuse the existing ExecutionProps

I think the challenge is that

The subset of ConfigOptions that functions might want to access is not clear to me
User defined functions may well want to use settings we haven't anticipated (and SessionConfig has a user defined way to pass it)

That doesn't mean we couldn't copy a subset of the fields into ScalarFunctionArgs, it just seems unecessary to try and pick that subset 🤔

If we just want specific fields, we could perhaps reuse the existing ExecutionProps

Are AliasGenerator and VarProviders relevant to function implementors?

The subset of ConfigOptions that functions might want to access is not clear to me

It's not clear to me either. We know about session time zone. Session start time may be relevant.
Do we need to know all of them from the start, or just anticipate we gonna evolve set of what's accessible as the need arises?

By avoiding full SessionConfig / ConfigOptions, I would like to retain the state where it's very easy to invoke a function, either from a test, a benchmark, or any other execution context.

Exposing sub-configs such SqlParserOptions, ExplainOptions, or OptimizerOptions to UDFs looks like breaking abstractions, or invitation to break abstractions.

I think my next attempt at solving this was to take the same approach that async-udf took - which is to move the invocation of the udf into an ExecutionPlan. That approach provides the full config_options.

Exposing sub-configs such SqlParserOptions, ExplainOptions, or OptimizerOptions to UDFs looks like breaking abstractions, or invitation to break abstractions.

In light of a push to reduce breaking changes like #16622 (and #16078, #16541, #13648) we could try to be more judicious about growing the public API. IF we don't need to expose full SessionConfig / ConfigOptions in ScalarFunctionArgs let's maybe not expose them.

In light of a push to reduce breaking changes like #16622 (and #16078, #16541, #13648) we could try to be more judicious about growing the public API. IF we don't need to expose full SessionConfig / ConfigOptions in ScalarFunctionArgs let's maybe not expose them.

My counter argument is that having to add new fields to ExecutionProps basically increases the public API over time. ConfigOptions is already public so it is my opinion that exposing just the ConfigOptions is less API surface area. I can see how there are alternate opinions to this

Here is what passing in the entire ConfigOptions looks like (it is pretty straightforward actually): #16661

BTW i solved my problem by adding the needed fields into the UDF struct during it construction and then I don't need any more info at simplify() & invoke() time. When doing so, I had this observation -- #16677

Closed in #16970

Navigation Menu

Repository navigation

Add `SessionConfig` reference to `ScalarFunctionArgs` #13519

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// evaluate the function
	let output = self.fun.invoke_with_args(ScalarFunctionArgs {
	args,
	number_rows: batch.num_rows(),
	return_type: &self.return_type,
	})?;

Add SessionConfig reference to ScalarFunctionArgs #13519

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

Omega359 commented on Nov 21, 2024

niebayes commented on Nov 21, 2024

Omega359 commented on Nov 21, 2024

alamb commented on Nov 21, 2024

Omega359 commented on Nov 21, 2024

jayzhan211 commented on Nov 22, 2024

Omega359 commented on Nov 22, 2024

alamb commented on Nov 22, 2024

Omega359 commented on Nov 22, 2024

jayzhan211 commented on Dec 8, 2024

alamb commented on Dec 8, 2024

jayzhan211 commented on Dec 9, 2024

alamb commented on Dec 9, 2024

findepi commented on Jun 26, 2025

alamb commented on Jun 26, 2025

findepi commented on Jun 26, 2025

Omega359 commented on Jun 26, 2025

findepi commented on Jul 1, 2025

alamb commented on Jul 2, 2025

findepi commented on Jul 4, 2025

alamb commented on Aug 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add `SessionConfig` reference to `ScalarFunctionArgs` #13519