Repository navigation
Release DataFusion 44.0.0 #13334
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Last release was https://crates.io/crates/datafusion/43.0.0 November 8th, 2024 so next major release would be around December 8, 2024
Prior release tickets:
Items to possibly fix before release
- Regression:
Invalid comparison operation: Utf8 == Utf8Viewerror during LEFT ANTI JOIN #13510 - [Epic] A Collection of Additional UTF8View support tickets #13504
Improved upgrade experience:
- Make migration to
Signature::nullaryin 44.0.0 easier / less confusing #13763 - Make
DocBuildermigration in44.0.0easier #13764 - Making the
recursivedependency an optional feature #13766 - Deprecate ScalarUDFImpl::return_type #13717
Downstream project testing:
Activity
alamb commentedon Nov 21, 2024
I would love to get this one in:
Some things to highlight (that are already merged)
alamb commentedon Nov 21, 2024
Also this hopefully:
SessionConfigreference toScalarFunctionArgs#13519alamb commentedon Nov 25, 2024
@Omega359 and @andygrove suggested #13525 (comment) that for this release we
Perhaps for 44.0.0 we can try with own subprojects (Ballista, Comet, DF Python, DF Ray)
alamb commentedon Dec 6, 2024
I started gathering a list of items we think we should fix prior to the release in the description
alamb commentedon Dec 6, 2024
Does anyone have any opinion about holding the DataFusion 44 release for the next major arrow release?
54.0.0(December 2024) arrow-rs#6342That would fix a bunch of the StringView issues, but would likely delay DataFusion 44 for a few more weeks
Omega359 commentedon Dec 7, 2024
It's soon to be the holiday season so I'm all for cooking the release a bit more.
alamb commentedon Dec 7, 2024
I would personally love to see DataFusion 44.0.0 be lauded as "super stable" and have few upgrade issues (we would largely achieve this by testing upgrades with other projects prior to release)
The downside is that the changeset in DataFusion 44.0.0 might be larger
andygrove commentedon Dec 9, 2024
I'd like to see an upgrade guide for the 44 release (and am willing to take the lead on this). I am trying to upgrade Comet now and am running into some issues.
I am +1 for waiting for the next Arrow release.
andygrove commentedon Dec 9, 2024
I filed #13702
andygrove commentedon Dec 9, 2024
How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?
Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.
alamb commentedon Dec 9, 2024
In my opinion, we would ideally do the testing before we make the release candidate (as the overhead of making RCs is non trivial). However, I am not opposed to extending the voting timeline if that would get us more testing time
andygrove commentedon Dec 9, 2024
Testing before we create the RC makes sense
andygrove commentedon Dec 10, 2024
I'm still working on updating Comet to work with latest DF from main (upgrading from 43.0.0). This is quite a disruptive upgrade for us so far.
alamb commentedon Dec 13, 2024
I have been thinking about this upgrade and the next arrow release
54.0.0(December 2024) arrow-rs#6342Specifically I think we need this upgrade in order to resolve some of the issues in 44 related to string view
However, what I would like to propose is:
The downside of this plan is
The upsides are
If someone is critically waiting on the string view fixes, we can also contemplate making another arrow incremental release
27 remaining items
alamb commentedon Dec 24, 2024
Update, when preparing to make a final RC I found another issue:
swap_hash_joinis no longer public so comet doesn't compile #13898I have a proposed fix here:
swap_hash_jointo bepub#13899alamb commentedon Dec 24, 2024
Ok, we are clear for 🚀 from what I can tell -- the comet upgrade is working. I will merge the following PR
And then make a Release Candidate
alamb commentedon Dec 25, 2024
I made a release candidate: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp
shehabgamin commentedon Dec 26, 2024
I'll re-test Sail with the release candidate tonight!
shehabgamin commentedon Dec 27, 2024
Test 2 complete! Smooth Sailing testing commit
073a3b1from DataFusion Release Candidate branch.Test details here: #13855 (comment)
alamb commentedon Dec 27, 2024
Thanks @shehabgamin
timsaucer commentedon Dec 27, 2024
If we can make a new RC I would recommend cherry picking this fix for initcap: #13909
alamb commentedon Dec 28, 2024
As @timsaucer mentions above (and on the mailing list): https://lists.apache.org/thread/34xy1nm47tkrr8lszxzwcrlf5b47h333
There was a regression introduced into initcap that has been sinced fixed: #13909
I will make a new RC
alamb commentedon Dec 28, 2024
have created a new RC and started a new vote thread: https://lists.apache.org/thread/p0hyr78zs03f3snj0mh3vqm6c3kjwshn
alamb commentedon Dec 31, 2024
The release has been approved and published to crates.io: https://lists.apache.org/thread/7w64xqzn71plvmxy61v6px8bb994kr79 🎉
https://crates.io/crates/datafusion/44.0.0
I have also made a small PR to improve the docs for the next release:
niebayes commentedon Jan 2, 2025
@alamb Hi, is the release for datafusion-cli delayed?
alamb commentedon Jan 2, 2025
Good catch @niebayes -- I forgot to push this to crates.io. I have done so now and
datafusion-cli44.0.0 is available:https://crates.io/crates/datafusion-cli/44.0.0
alamb commentedon Jan 2, 2025
Here is a ticket to run ClickBench on DataFusion 44:
44.0.0#13983alamb commentedon Jan 4, 2025
Filed ticket for the next relese:
45.0.0#14008