Repository navigation
Some datafusion-spark expressions are missing the physical implementation #21959
Description
Describe the bug
I wanted to add base64 support in Comet using the expression from datafusion-spark, but it has no implementation, so I had to implement a custom version in Comet. This doesn't seem ideal.
fn invoke_with_args(&self, _args: ScalarFunctionArgs) -> Result<ColumnarValue> {
exec_err!("invoke should not be called on a simplified {} function", self.name())
} Impacted functions:
- base64
- unix
- trunc
- if
- date_diff
- time_trunc
- date_trunc
- add_months
- date_part
Additional context
https://discord.com/channels/885562378132000778/1413656077014143008/1499408538429030421
Activity
coderfender commentedon Apr 30, 2026
Let me take on this and also other expressions to completely remove simplify methods in df-spark crate given comet (and hopefully other downstream consumers) use it
coderfender commentedon May 5, 2026
Raised discussion thread : #22012
Jefffrey commentedon Aug 6, 2026
@coderfender if youre still keen on this issue, I've updated the issue body with the list of functions to tackle to close this issue
coderfender commentedon Aug 6, 2026
Awesome let handle this over the weekend
doc: add implementation guidelines for `datafusion-spark` (apache#24140)