Repository navigation
[DISCUSSION] Add separate crate to cover spark builtin functions #5600
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The discussion is to collect community thoughts on implementing spark builtin functions listed https://spark.apache.org/docs/3.2.0/api/sql/
We more often face the requests to implement Spark functions and the use case highly depends on person/company stack, one treats the spark more important, for others its opposite and PG compatibility is a priority
Builtin function list between Postgres and Spark are expectedly not the same. I believe it can be rare and worse cases when the function name the same but signature and/or return type is different.
The discussion goal is to find out how to organize DF and keep compatibility for majors like Spark, Postgres, and perhaps other systems
Describe the solution you'd like
@alamb in #5568 (comment) made a proposition to create an extensible crate for spark functions, or even it can be a separate subproject so the PG users have the possibility to exclude Spark functions.
Related Issues
- Spark-compatible CAST operation #11201
- Make modulos with negative float zero compat with other engines #11051
- Support "standard" / alternate format arguments for
to_timestamp#8915 - Decimal division compatibility mode with spark #7301
Describe alternatives you've considered
Not doing this
Additional context
Created after #5568 (comment)
Activity
andygrove commentedon Mar 14, 2023
It seems reasonable to me to have separate Spark and PG implementations of functions. This would make it more attractive for users wanting to migrate from Spark to DataFusion.
I like the idea of having a new
datafusion-sparkcrate with anything related to Spark compatibility and integration (such as Spark SQL dialect and Spark built-in functions). I wonder if @yjshen has any thoughts on this since he built a Spark + DataFusion integration in Blaze.yjshen commentedon Mar 14, 2023
Separating Spark functions into a special crate seems reasonable but supporting Spark UDFs requires significant effort. This is because many UDFs in Spark are designed to be compatible with Hive and handle corner cases differently than other databases like PG. These corner cases increase the workload of integrating Spark/Hive with DataFusion.
When developing Blaze, we must compare the implementations of both engines or port tests first to ensure that they have identical semantics before passing a UDF for execution by DataFusion.
alamb commentedon Mar 15, 2023
I also agree (unsurprisingly) that a separate crate for libraries of functions would be valuable.
Overall, keeping datafusion setup with a core and then a bunch of extension points, such that different sets of functionalty can be assembled for whatever usecase people want would be my ideal.
For example, there are already a bunch of functions that in theory should be "optional" via feature flags--
https://github.com/apache/arrow-datafusion/blob/6bfede0e1e1a09bd06ea0166e01c0ad4834a1721/datafusion/physical-expr/Cargo.toml#L35-L42
But I am not sure how well the vision and reality match (as in does anyone use datafusion without them)
comphead commentedon Mar 15, 2023
@alamb you are right, we can't provide optional builtin functions, that would be more than unexpected.
In wonderful example #5568 (Implement to_unixtime function) the same functionality can be achieved differently in
Spark
to_unix_timestamp(now())PG
extract(epoch from now())Trino
select to_unixtime(now())I'm not even sure if we need extra crates now. My personal feeling is Trino trying to adopt all possible syntaxes.
Perhaps we go that way and just keep extending the list of builtin functions and in case of conflicts(if any) the PG signature/return type could be as dominant
comphead commentedon Mar 24, 2023
@alamb do we need more discussion?
We can try to adopt syntaxes from major engines like Spark, PG, etc, this will make the migration to DF easier.
However The downside is supporting the same functionality in different implementations leading to code growth, and probably more time to support.
alamb commentedon Mar 25, 2023
So my opinion on this matter is that ideally DataFusion should be an extensible engine and so people using it can pick whatever parts they want to use and replace what they don't with their own implementations.
DataFusion includes a bunch of pre-built functionality (like the mostly compatible PG functions, parquet / json / etc readers, a memory catalog, etc) in order to get people started so they can focus on extending whatever is most important for their usecase.
So I think it would be great to have a separate crate with "spark compatible functions" (maybe also the same could be done for a "postgres compatible functions crate"). I think the BuiltInFunction thing is not required long term and it would be better if all functions could behave the same as user defined functions
Then the question becomes "where is that crate's code stored" -- it is probably fine initially to be in the main datafusion repo initially and if it gets too unweildy we could break it into its own repo or something.
But the ability to customize the functions available I think is key
alamb commentedon Feb 1, 2024
BTW #8045 tracks the work to break apart the function libraries into smaller pieces. Once this is done I think adding making a crate of spark compatible functions will be pretty straightforward
Omega359 commentedon Feb 1, 2024
I would like to recommend that each function crate also has a benches directory and any Cargo.toml file for those crates have criterion setup as a dev dependency.
Omega359 commentedon Feb 1, 2024
Well, I can say that for my current use case crypto and array functions are something that I'm unlikely to need. Every one's use case is different of course but it's a data point.
alamb commentedon Feb 1, 2024
Here is a proposal of how to extract / organize the functions: #9100
alamb commentedon Jan 26, 2025
FWIW we have now completed migrating all functions to User Defined Functions and I think there is growing interest in
BTW I think there are many people interested in spark compatible functions. For exmaple:
Maybe now is time to actually start collaborating on a spark compatible UDF function library. If anyone is interested in working on this, let me know and I can make a repo in https://github.com/datafusion-contrib
This could be similar to what @demetribu was working on in https://github.com/datafusion-contrib/datafusion-functions-extra (and maybe we can take a friendly look at how that was organized)
andygrove commentedon Jan 26, 2025
We already have a crate with Spark-compatible expressions maintained as part of the Comet subproject.
https://docs.rs/datafusion-comet-spark-expr/0.5.0/datafusion_comet_spark_expr/
It does not depend on Spark, JVM, or Comet—it is just a library of Spark-compatible DataFusion expressions.
We deliberately maintain them in one crate so that other DataFusion-based projects can use them.
andygrove commentedon Jan 26, 2025
The crate supports more than 100 expressions so far, most of which are listed here:
https://datafusion.apache.org/comet/user-guide/expressions.html
shehabgamin commentedon Jan 27, 2025
I love the idea of collaborating on Spark compatible
UDFs.As of writing,
243/402Spark functions doc-tests pass on Sail. We haven't focused on performance yet and instead have been focusing on just knocking all of them out because there are so many of them. Our implementations can be found:I will say that we have encountered numerous problems relying on downstream DataFusion-based crates, to the extent that we have removed all of them as dependencies (and included appropriate
[Credit]comments in various places to acknowledge the original sources). The issue isn't with the crates themselves but arises when it's time to upgrade DataFusion versions, requiring us to wait for each crate to update and release a new version.We haven't done as good a job as @andygrove and the Comet folks with documenting what we do and don't support (we're currently a small team of just two people). However, we do have test reports for every pull request to track coverage.
Sail has also put lots of efforts in supporting PySpark’s 10+ APIs for Python
UDFs/UDAFs/UDWFs/UDTFs. We support all the PySparkUDFtypes except one (the experimentalapplyInPandasWithState()method ofpyspark.sql.GroupedData):18 remaining items
kazuyukitanimura commentedon Jan 31, 2025
+1 for including spark functions in DataFusion main
@alamb just making sure if I understand, CI will still run and check
datafusion/src/functions-sparkeven it is not a dependency? For example, testing interface changes that affects bothdatafusion/src/functions-sparkanddatafusion/coreSounds like so based on your follow up comment
andygrove commentedon Jan 31, 2025
draftPR feat: Adddatafusion-sparkcrate #14392findepi commentedon Jan 31, 2025
I feel positive in principle about creating a crate as a home for Spark-compatible functions, especially if it gonna be maintained ("not dead project") and released together with core DF. My assumption is this is a totally optional feature for downstream projects (core dependencies do not change).
I didn't read the discussion above yet (just responding to Andy's call for opinions from mailing list). So it might there are some arguments that I should be aware of.
andygrove commentedon Jan 31, 2025
Correct. It is part of the workspace, but no other DataFusion crates depend on it.
Perhaps there is an argument for datafusion-cli having an optional dependency on it in the future.
alamb commentedon Jan 31, 2025
Yes that is my assumption
Yes that is what I was thinking. Exactly what those tests would be I am not 100% sure (e.g. there probably needs to be some way to test them without actually running spark
100%
Spaarsh commentedon Feb 9, 2025
I began contributing to this repo only from a month ago so please pardon an errs from my side, but I just wanted to suggest something.
Since we're planning of having a separate mode for spark wherein a user can access all spark functions and also not make the main code dependent on this crate, I was thinking if there was still a way of using these spark functions alongside our main df functions (i implied that all core functions won't be available in this mode since there are many similarly named functions). Could we allow optional import of our main df functions in the spark mode cli? This might be a makeshift solution for now but this way our users who migrate from spark can ALSO be exposed to our core functions and use them!
Or perhaps we could implement a "import function_name" kind of command that imports only a specific function or set of functions and checks for conflicting names.
From my work in the xxhash functions in the code, the mod.rs enlists all functions in that directory so this seems easier in theory.
Thanks.
alamb commentedon Feb 10, 2025
What I was thinking is that "spark mode" would override any necessary functions (but use the existing ones provided by DataFusion when that is not necessary)
I think that would be ok (maybe implement this in datafusion-cli)
Spaarsh commentedon Feb 10, 2025
I'm sorry if I'm got this wrong, you're suggesting that we could make an import command in the datafusion-cli independent of the current discussion, right?
I think can be included in an entirely new feature "Custom CLI mode" where except for certain core functions, the user manually imports the set of functions they require. This maybe trivial right now, but in the future if we increase our number of functions, this might come handy for users who wish to minimize the resource footprint of our cli.
If this looks worthy of further discussion and deviates from this discussion, I would be glad to open a new issue for this.
alamb commentedon Mar 11, 2025
There is quite a lot of related discussion on
datafusion-sparkcrate #14392Currently @shehabgamin I think plans to try and prototype this:
datafusion-sparkcrate #14392 (comment)Once they get a basic pattern down for spark compatible functions, this tickeet would then entail porting the rest of the function implementations along with tests, etc
shehabgamin commentedon Mar 11, 2025
WIP: #15168
Just gotta do some cleanup and add sufficient testing examples. I will have it ready for review by EOD tomorrow at the latest.
alamb commentedon Mar 11, 2025
BTW some backstory @shehabgamin is that Google Summer of Code applicants are starting to look for projects and this is one of the ones listed for DataFusion: https://datafusion.apache.org/contributor-guide/gsoc_project_ideas.html
In other words, I think we can rally potentially a bunch of community interest to help fill out the list of functions
It will be super valuable for you to have help set the pattern first however.
shehabgamin commentedon Mar 11, 2025
@alamb Oh I had no idea, that's amazing!
The Draft PR I linked sets up the structure for the entire crate. Only thing I haven't done yet is setup a well defined testing pattern, but that will be included too.
shehabgamin commentedon Mar 13, 2025
#15168 is ready for review! @alamb @andygrove
alamb commentedon May 1, 2025
We are tracking follow on work in
datafusion-sparkSpark Compatible Functions #15914