Repository navigation
Release DataFusion 49.0.0 (July 2025) #16235
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Previous release will be https://crates.io/crates/datafusion/47.0.0 (June, 2025) so next major release would be late June or Early July 2025
Steps:
- Update version and changelog: [branch-49] Update version to
49.0.0, add changelog #16822 - Create a release branch: https://github.com/apache/datafusion/tree/branch-49
- Test with DataFusion Python:
- Test with DataFusion Comet: chore: migrate to DF 49.0.0 datafusion-comet#2040
- Test with delta.rs: chore: update to DataFusion
49.0.0delta-io/delta-rs#3603 - Test with iceberg-rust:
- Test with LakeSail: chore: Upgrade to DataFusion 49.0.0 lakehq/sail#653
- Write upgrade guide:
- Test with parquet viewer
- Test with datafusion-materialized-views: Upgrade to DF49 datafusion-contrib/datafusion-materialized-views#75
- Voting Thread:
- Create ticket for next release: Release DataFusion
50.0.0(Aug/Sep 2025) #16799 - Forward port version and changelog to
main: [main] Update version to 49.0.0, add 49.0.0 changelog #16855
TODOs
- Update upgrade md for new unified config for sql string mapping to utf8view when we release datafusion 49.0.0 #16428
- Blog post for the DataFusion 49 release #16758
Prior release tickets:
Changes to add to upgrade guide
These PRs made changes that deserve a mention in the upgrade guide
Features to mention in the blog (if they make it)
- Introduce Async User Defined Functions #14837
- [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
- TopK dynamic filter pushdown attempt 2 #15770
- Support integration with Parquet modular encryption #15216
- [MAJOR] Equivalence System Overhaul #16217
- Use Tokio's task budget consistently, better APIs to support task cancellation #16398
- feat: Parquet modular encryption #16351
Bugs that need to be fixed
- Panic in
datafusion_expr::window_state::WindowAggState::update#16308 - SortQueryFuzzer found a failing case on main #16452
- Wrong results when
pushdown_filtersis enabled (starting in 48.0.0) #16588 - restore topk pre-filtering of batches and make sort query fuzzer less sensitive to expected non determinism #16501
- Extended tests are failing #16595
Bugs that would be good to fix / investigate
- 1000x slowdown opening parquet file due to partitions #16676
- How to write csv file to disk from a empty dataframe? #16240
- Release DataFusion
49.0.0(July 2025) #16235 (comment)
Community Wishlist
Reacted by Jonathan Chen, Yongting You, Qi Zhu, Adam Gutglick, Oleks V and feniljain
Activity
alamb commentedon Jul 12, 2025
I am thinking we should begin the 49.0.0 release next week. I'll publicize to the lists / etc
xudong963 commentedon Jul 12, 2025
@alamb Yeah, let's do it (time flies
adriangb commentedon Jul 13, 2025
We introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.
xudong963 commentedon Jul 14, 2025
I'm okay to wait for these changes. It's always good to not break users.
alamb commentedon Jul 14, 2025
Sounds good -- I added it to the checklist on the description of this ticket
adriangb commentedon Jul 14, 2025
We're trying to minimize breakage! Thanks to you and everyone else for your your patience as we make these exciting but difficult improvements.
adriangb commentedon Jul 15, 2025
xpost: #15057 (comment)
adriangb commentedon Jul 15, 2025
I've opened #16791 to address backwards compatibility
alamb commentedon Jul 16, 2025
I am hoping to start with some of the various release tasks tomorrow (like ensuring hte upgrade guide is in a good place) but I have been very busy with other things recently. Hopefully we can sort out #16791 soon and get the relese process started
shehabgamin commentedon Jul 17, 2025
Almost missed this thread. Will start testing when there is a
49branch!alamb commentedon Jul 17, 2025
I am still behind -- i will likely start in earnest on this task tomorrow (Friday) and over the weekend and hopefully we'll be reado to cut a Release Candidate (RC) early next week
15 remaining items
shehabgamin commentedon Jul 23, 2025
Tested on Sail with commit afb9099. Everything looks good!
lakehq/sail#653
comphead commentedon Jul 23, 2025
Testing on Comet, seeing some test failures
UPD: Moved to the latest DF RC branch and tracking progress apache/datafusion-comet#2040
One weird thing is the correctness issue on the queries with first/last with ignore null flag.
I can't find if Datafusion had changes for
ignore nullflag in 49 release 🤔UPD2: Looks like the cause is somewhere in
aggregate.rsorfirst_last.rsUPD3: @alamb I think it is good to go, the reason is the
ignore nullsresets the value if the grouping key lives in 2 separate parquet files. Same scenario works fine in DF, I assume some logic was moved from physical layer to logical causing Comet correctness issues when usingFirstValuefunction. I'm still investigating it, but again the same scenario works fine in DFUPD4: @alamb @ozankabak @berkaysynnada @mbutrovich Filed a PR https://github.com/apache/datafusion/pull/16918/files, appreciate for thoughts
xudong963 commentedon Jul 24, 2025
Test MV repo with DF49, datafusion-contrib/datafusion-materialized-views#75. Looks good
XiangpengHao commentedon Jul 24, 2025
I run into some compile issue in parquet viewer, still investigating...
Update 1: I have to disable the
encryptionfeature in Parquet to make it work: https://github.com/apache/datafusion/blob/main/Cargo.toml#L162It seems that it's not happy with
ring, but I also saw ring already have some special handling for wasmOmega359 commentedon Jul 24, 2025
I ran my test suite against df 49, all passed. I did have to make an update for
TableParquetOptionsthat wasn't listed in the upgrade doc (new crypto field).alamb commentedon Jul 24, 2025
Thanks @XiangpengHao -- do you think we should disable the crypto feature by default?
cc @corwinjoy @adamreeve
XiangpengHao commentedon Jul 24, 2025
@aoli-al figured out that this is an issue from my nix flake build, not a datafusion issue: XiangpengHao/parquet-viewer#50
(but I only see this error starting df49)
Now all tests passed with parquet viewer!
adamreeve commentedon Jul 24, 2025
It sounds like this is sorted but let me know if you'd like it disabled by default for the release. Corwin is on holiday but I can help with that if needed.
xudong963 commentedon Jul 27, 2025
The 49.0.0 is released, and I've published it to crate.io
alamb commentedon Jul 27, 2025
Thank you very much @xudong963 and everyone else who helped
I hope to publish the blog tomorrow:
49.0.0release post datafusion-site#91chore: bump datafusion to 49 (#4006)
alamb commentedon Aug 4, 2025
49.0.1(patch) #17036