Release DataFusion 50.1.0 (minor) #17594
Description
Is your feature request related to a problem or challenge?
- Follow on to Release DataFusion
50.0.0(Aug/Sep 2025) #16799
There are some things that we may way to backport into the 50.0.0 line, and this ticket tracks doing so
Steps:
- Update version and changelog: [branch-50] Prepare for 50.1.0 release #17748
- Test with DataFusion Python
- Test with DataFusion Comet
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread: https://lists.apache.org/thread/jch5tfvc9yzf7zrro61hcqbclqmsr4xr
- Publish to crates.io
- Create ticket for next release:
Potential Content:
Bug Fixes
- Exponential planning time when window function is partitioned by multiple columns #17401 / Prevent exponential planning time for Window functions #17563 / Prevent exponential planning time for Window functions - v2 #17684
- fix: ignore non-existent columns when adding filter equivalence info in
FileScanConfig#17546 - fix: Ensure the CachedParquetFileReader respects the metadata prefetch hint #17302
- Partial AggregateMode will generate duplicate field names which will fail DFSchema construct #17662
- [branch-50] Partial AggregateMode will generate duplicate field names which will fail DFSchema construct to branch-50 #17717
Minor Features:
Activity
[-]Release DataFusion `51.0.1` / `51.1.0` (patch) release[/-][+]Release DataFusion `51.0.1` (patch)[/+]alamb commentedon Sep 16, 2025
I can help run this release process unless someone else wants to
[-]Release DataFusion `51.0.1` (patch)[/-][+]Release DataFusion `50.0.1` (patch)[/+]milenkovicm commentedon Sep 16, 2025
Edit: incompatible change, does not make sense to go to patch release, sorry for the noise.
rkrishn7 commentedon Sep 16, 2025
@alamb #17600 to backport 17546
shehabgamin commentedon Sep 17, 2025
#17613
milenkovicm commentedon Sep 18, 2025
Due to #17596 ballista would not be able to support datafusion 50 I can provide a workaround but would mean we would need to release 50.1.0 could we consider 50.1 release instead?
Change would be adding two new methods to FunctionRegistry
alamb commentedon Sep 18, 2025
I think adding two new methods and releasing
50.1.0sounds good to me@findepi and I are still going back and forth a bit on the solution to
milenkovicm commentedon Sep 18, 2025
thanks @alamb, I'll try to make a PR tonight
milenkovicm commentedon Sep 18, 2025
i believe #17650 should do the job and unblock ballista, it would be great if we can merge it before we get chance to release #17596
milenkovicm commentedon Sep 18, 2025
Is there anything else i need to do to get this PR to patch release?
alamb commentedon Sep 19, 2025
Once #17650 is merged, we need to make another PR targeting the
branch-50branch, for example #17613Note it sounds like @berkaysynnada is going to try and work on a better solution to #17563 over the weekend so we still have some time
8 remaining items
alamb commentedon Sep 23, 2025
I have created a PR with version update and changelog:
alamb commentedon Sep 25, 2025
Ok, #17684 is merged in now
I made a backport PR here
And the PR for updating version and changelog are here
alamb commentedon Sep 26, 2025
Ok, one more PR to go and then I'll make the release candidate:
alamb commentedon Sep 28, 2025
I am just waiting on a committer to approve / merge this PR
And then I can make a RC
AdamGS commentedon Sep 29, 2025
Is
50.2.0planned, or is the next release going to be51.0.0?alamb commentedon Sep 29, 2025
I don't know of any plan for 50.2.0, but if you have fixes you want to include we can make one. Could you file a ticket if that is the case?
AdamGS commentedon Sep 29, 2025
Will do once #17808 merges.
alamb commentedon Sep 29, 2025
Voting thread for 50.1.0: https://lists.apache.org/thread/jch5tfvc9yzf7zrro61hcqbclqmsr4xr
AdamGS commentedon Sep 29, 2025
Tested with Vortex. We have a pretty big benchmarking matrix (its a bit noisy sometimes, but does help us catch significant changes and trends), results look solid for TPC-H, TPC-DS and ClickBench.
AdamGS commentedon Sep 30, 2025
Should I use the same template for the
50.2.0issue?mbrobbel commentedon Sep 30, 2025
We should include #17845 to prevent docs.rs build issues
alamb commentedon Sep 30, 2025
Yes please
alamb commentedon Oct 2, 2025
The 50.1.0 release was approved and released
I have also published to crates.io
Bump to DataFusion v50.0.1 (#370)