Repository navigation
[datafusion-spark] Test integrating datafusion-spark code into comet #1704
Description
What is the problem the feature request solves?
@shehabgamin added the datafusion-spark crate in apache/datafusion#15168
The goal is to help centralize the development of this function library in the core repository rather than duplicated effort
To verify this is feasible I would like to verify that the setup created in datafusion-spark / apache/datafusion#15168 can be used by downstream crates like comet before we go too far
Describe the potential solution
I would like someone to make a PR that shows we can remove one or more of the spark function in datafusion-spark can be used in comet (and reduce the code here)
Additional context
Reacted by Oleks V, Andy Grove and Bhargava Vadlamani
Activity
andygrove commentedon May 5, 2025
I created apache/datafusion#15947 to add Comet's
hexfunction todatafusion-spark. This involved converting it from aPhysicalExprto aScalarUDFImpl.Also, I created #1711 to use the
expm1function fromdatafusion-sparkin Comet.andygrove commentedon May 22, 2025
#1711 is now merged, so Comet is now using the datafusion-spark crate
andygrove commentedon May 22, 2025
I will keep this issue open until we actually remove an expression from comet and use a version from data fusion-spark
alamb commentedon May 22, 2025
EPIC! FYI @shehabgamin
[datafusion-spark] Add Spark-compatible ceil function (#20593)
[datafusion-spark] Add Spark-compatible ceil function (apache#20593)
[datafusion-spark] Add Spark-compatible ceil function (apache#20593)
dwsmith1983 commentedon Sep 11, 2026
The condition set here (remove a Comet expression and use the datafusion-spark version instead) was met in #2080 (d8c62f3), which dropped Comet's own
chrin favor of the datafusion-spark one, and the pattern has since become routine: native/Cargo.toml pullsdatafusion-sparkinto both native/core and native/spark-expr, and native/core/src/execution/jni_api.rs registers a growing set of its functions (SparkArrayContains,SparkArrayRepeat,SparkBitCount,SparkBitGet,SparkBitShift,SparkWidthBucket, and more) in the session context. Suggest closing.andygrove commentedon Sep 28, 2026
Closing this, since the condition I set above has been met. As @dwsmith1983 pointed out, #2080 replaced Comet's own chr with the datafusion-spark version, and bit_not (#2896), space (#3612), bit_count (#3616) and substring (#4161) have gone the same way since. Comet now uses more than 30 datafusion-spark functions.