Repository navigation
Release DataFusion 45.0.0 #14008
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Last release was https://crates.io/crates/datafusion/440.0 December 31, 2024 so next major release would be around Feb 1, 2025
Steps:
- Update version and changelog: Prepare for
45.0.0release: Version and Changelog #14397 - Voting Thread: https://lists.apache.org/thread/g20ywc9yto8xp07lcllmvgyn8g5z4420
- Release to crates.io: https://dist.apache.org/repos/dist/release/datafusion/datafusion-45.0.0
- Create ticket for next release: Release DataFusion
46.0.0#14123 - Port changelog to main: Minor: Update changelog for
45.0.0and tweakCHANGELOGdocs #14545
Pre-relese testing
- Test with Comet: Test DataFusion 45.0.0 with Comet #14274
- Test with delta.rs: chore: update to DataFusion 45.0.0 / arrow/parquet
54.0.0delta-io/delta-rs#3175 - Test with SailHQ: Test DataFusion 45.0.0 with Sail #14408
- Test with InfluxData
- Test with DataFusion Python: Test DataFusion 45 with datafusion-python #14410
Prior release tickets:
Please let me know if you would like to add any items on this list or move the categorization
Items to fix before release
- Upgrade to arrow-rs/parquet
54.0.0#14114 - Regression:
DataFrame::schemareturns incorrect schema for NATURAL JOIN #14058 - Regression in CASE expression since DF 44 #14277
- string_numeric_coercion in binary.rs does not support Utf8View #13359
- DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs #14230
- Regression:
Invalid comparison operation: Utf8 == Utf8Viewerror during LEFT ANTI JOIN #13510 - Type Coercion fails for List with inner type struct which has large/view types #14154
- [Epic] A Collection of Additional UTF8View support tickets #13504
-
encode(..., "hex")errors on non-UTF-8 binaries since Datafusion v43 #14055 - Improve deprecation message for MemoryExec #14322
- LimitPushdown rule uncorrect remove some GlobalLimitExec #14204
Items maybe to complete (not sure if they are blockers)
- Can no longer easily join duplicate schemas as of version 43 #14112
- Query produces different results after physical plan round trip to bytes #14253
- Potential performance regression with comparisions to scalar values #14291
- Error with type coercion with
CREATE TABLE AS SELECT... insertingVALUES#13124 - Bug: applying multiple times
EnforceDistributiongenerates invalid plan #14150
Nice to Have (but non blockers -- e.g. bugs but not regressions)
- feat: metadata columns #14057
- Exponential planning time (100s of seconds) with
UNIONandORDER BYqueries #13748 - External Error prefix is repeated multiple times #14080
- FFI support for versions and alternate tokio runtimes #13937
- Improve speed of
medianby implementing specialGroupsAccumulator#13681 -
FULL OUTER JOINandLIMITproduces wrong results #14335
Reacted by Shehab Amin, kamille, xudong.w and Oleks V
Activity
alamb commentedon Jan 13, 2025
@andygrove would you like to coordinate this release or would you like me to? (or does anyone else want to do so?)
alamb commentedon Jan 13, 2025
I also added some issues to the description above that I think would be worth fixing
andygrove commentedon Jan 13, 2025
I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.
alamb commentedon Jan 13, 2025
I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )
xudong963 commentedon Jan 14, 2025
Thanks, alamb, I booked 46 in advance!
alamb commentedon Jan 14, 2025
Awesome -- I filed #14123 to track 46
alamb commentedon Jan 15, 2025
I plan to start assembing the release candidate and test on the week of Jan 27 (in about 2 weeks time()
shehabgamin commentedon Jan 18, 2025
As promised, Sail is working on porting relevant tests into DataFusion.
A good starting point is a regression our tests caught in DataFusion 43, which still seems to persist in DataFusion 44. A regression was introduced in DataFusion 43.0.0 related to casting to UTF8 in various places. Upgrading to DataFusion 43.0.0 required adding explicit casting in several areas as a workaround. This PR (lakehq/sail#355) comments out those changes to expose the regression through the 12 additional failed tests compared to the main branch.
Once I’ve pinpointed the root cause(s) of the regression, I’ll create an issue in DataFusion to track the work. I want to ensure the issue accurately reflects the problem before filing it. I’m happy to address these regressions and port over the tests that cover them in the same PR. Hopefully, we can get this resolved in time for the DataFusion 45 release!
alamb commentedon Jan 18, 2025
Thank you very much @shehabgamin 🙏
I strongly suspect this is related to switching to Utf8View by default in Parquet; You can validate this theory by disabling the following config setting:
https://datafusion.apache.org/user-guide/configs.html
I think we are pretty close to closing out the Utf8View epic (now that we have upgraded to the latest arrow):
I'll add that to the list for 45 too
alamb commentedon Jan 18, 2025
I plan to start preparing / testing / pushing this release the week of Jan 27, aiming to get an release candidate early the next week
shehabgamin commentedon Jan 19, 2025
Thanks for the pointer @alamb!
I tried setting
datafusion.execution.parquet.schema_force_view_typestofalse, but unfortunately, none of the 12 failed tests passed.I'll take a deeper look into the issue after the weekend. Hope you have a great rest of your weekend!
shehabgamin commentedon Jan 22, 2025
Most of the regressions are related to this issue: #14230. I should be able to resolve them well before the
45release.While testing my local Sail code with the latest commit on DataFusion's main branch, I encountered several breaking changes that may make DataFusion 45 a jarring upgrade for some users. Given the previous discussion about wanting to make releases less jarring (#13334 (comment)), I wanted to bring this to your attention, @alamb.
Aside from that, there is one remaining regression I haven't investigated yet, which seems to be related to Parquet.
alamb commentedon Jan 22, 2025
Thanks @shehabgamin -- Can you enumerate these changes (or point me at a PR) so we can see if there is some way to make jarring
shehabgamin commentedon Jan 22, 2025
Yeah I'll work on that right now!
26 remaining items
jayzhan211 commentedon Feb 2, 2025
I found that we can't make this deprecated but a breaking change if the user override the
is_nullablefunction because it takesschema: &dyn ExprSchemawhich is not the case forreturn_type_from_exprstimsaucer commentedon Feb 2, 2025
I am currently working on building the 44 release. I hope to get that done this morning or tomorrow. It's generally run about a month behind this upstream repo
alamb commentedon Feb 3, 2025
In case anyone is interested, here is the release thread:
alamb commentedon Feb 3, 2025
Update on the release:
mainis open for new code (will be46.0.0)I have created a 45.0.0 branch from which to make the 45.0.0 release
I would very much like to get fixes for the following issues into the 45 branch as I think they are regressions that prevented some people from upgrading from 43 --> 44 and the fixes are fairly small:
CASE#14384DataFrameAPI #14387 (regression from 43)Utf8Viewtonumericcoercion #14377 (Utf8View regression in 43)REGEXP_MATCHscalar function to support Utf8View #11911 (Utf8View regression in 43) (mentioned by @Omega359 above)It would also be really nice to sort out this issue from @shehabgamin but it isn't clear to me we can fix that with a small non-risky patch
alamb commentedon Feb 3, 2025
@Omega359 has a fix for #11911
This afternoon I will make some backport PRs and hopefully get a release candidate created
alamb commentedon Feb 3, 2025
Backport PRs (for my own OCD sanity)
arrow/parquet54.1.0and fix clippy/ci (#14415) #14453Utf8Viewtonumericcoercion (#14377) #14455alamb commentedon Feb 3, 2025
Ok, I think we are ready with a release candidate:
tag here: https://github.com/apache/datafusion/releases/tag/45.0.0-rc1
Release voting thread: https://lists.apache.org/thread/g20ywc9yto8xp07lcllmvgyn8g5z4420
Content
This release candidate is based on commit: 26058ac 1
The proposed release tarball and signatures are hosted at 2.
The changelog is located at 3.
The standard verification procedure is documented at https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates.
shehabgamin commentedon Feb 3, 2025
@alamb I'll test on Sail again sometime today!
shehabgamin commentedon Feb 4, 2025
I updated #14408 (comment), but same issues as before with:
Test 2: Commit
26058ac(DataFusion45.0.0-rc1).shehabgamin commentedon Feb 4, 2025
I think we came to a temporary solution here #14268 (comment)
CC @alamb @Omega359 @jayzhan211
Omega359 commentedon Feb 4, 2025
I don't think I'll have the bandwidth to test a type coercion fix for UDF's myself this week to be honest. I'm about to fire off a full run of my application against the 45 branch but I likely can only do that once this week.
Omega359 commentedon Feb 5, 2025
Beyond the type coercion issues that I can work around my testing is working ... as long as I don't compile with 'release' target. That seems to segfault I think somewhere in DF code but I haven't yet been able to get a core dump from the crashing nodes to investigate further. I thought it was Rust 1.84.1 dependent but I retested with 1.83.0 and had the same issue.
I don't think it's a blocker but is something I'm going to continue to try and narrow down.
wiedld commentedon Feb 5, 2025
@alamb -- release candidate re-tested on InfluxData, and is good.
alamb commentedon Feb 7, 2025
The release has been approved and published to crates.io: https://lists.apache.org/thread/73plk4nhdjby58knd5rvwsmsjzpbcg7s