Repository navigation
Release DataFusion 51.0.0 (Nov 2025) #17558
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Previous release will be #16799 (Sep, 2025) so next major release would be approximately late Nov 2025
Steps:
- Create a
branch-51branch: https://github.com/apache/datafusion/tree/branch-51 - Update version and changelog: [branch-51] Update version to 51.0.0, add Changelog #18551
- Test with DataFusion Python
- Test with DataFusion Comet
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread:
- Publish to crates.io
- Create ticket for next release: Release DataFusion
52.0.0(Dec 2025 / Jan 2026) #18566 - Create blog post: Blog post for the DataFusion
51.0.0release #18548
TODOs
Prior release tickets:
Related
- Boston USA Nov 2025 meetup: DISCUSSION: DataFusion Meetup in Boston, USA - Nov 12, 2025 #16703
Features to mention in the blog (if they make it)
- Support Decimal32/64 types #17501 from @AdamGS
- Support
WHERE,ORDER BY,LIMIT,SELECT,EXTENDpipe operators #17278 (comment) - Reduce number of object store requests when reading parquet files by default (set
metadata_size_hint) #18118 - [
datafusion-cli] Add a way to see what object store requests are made #17207 - [EPIC] A collection of items to improve CASE performance #18075
- feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size #16771
- Examples of extending SQL syntax #17824 from the @theirix
Issuses needed
Bugs that should be fixed
- Restore window sort optimizations without exponential planning time #17624
- Exponential planning time when window function is partitioned by multiple columns #17401
- Non-constant
DATE_TRUNCexpression regression for values before epoch #18334 - Optimizer 'common_sub_expression_eliminate' fails in a window function (SQLStorm) #17770 / fix: optimizer
common_sub_expression_eliminatefails in a window function #17852 - CI is failing on main /
current_date() = cast(now() as date);#18062 - NowFunc::new() seeds the timezone with "+00" while the canonical UTC offset remains "+00:00" in ConfigOptions::default(). #18219
- to_timestamp(double) gives different results depending on scalar/vectorized call context #16678
- Regression: error planning TPC-DS query: input schema nullability mismatch #17801
- Expects PARTITION BY expression to be ordered in DataFusion 51 release #18527
- [branch-51] bugfix: correct regression on TableType in into_view in DF51 #18618
- [branch-51] bugfix: select_columns should validate column names #18624
Community Wishlist
- Streaming Aggregate operator not being used in deduplication of pre-sorted Parquet files #16919
- Regression:
DataFrameWriteOptions::with_single_file_outputproduces a directory #13323 - fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496
- Respect execution timezone in
to_timestampand related functions #18025
Activity
mjgarton commentedon Sep 14, 2025
Should the title of this issue say 51.0.0 instead of 50.0.0?
[-]Release DataFusion `50.0.0` (Nov 2025)[/-][+]Release DataFusion `51.0.0` (Nov 2025)[/+]xudong963 commentedon Sep 15, 2025
yes, thank you, i changed it.
findepi commentedon Sep 15, 2025
I removed #16842 from "Changes to add to upgrade guide". I think it was completed in 50.0.0. If not, let's add it back.
xudong963 commentedon Oct 28, 2025
I think we could start working on the release next week?
rluvaton commentedon Oct 28, 2025
If we could get the following arrow fix merged and released as a patch and upgrade to that patch before the upcoming release it would be sweet:
alamb commentedon Oct 28, 2025
I think that one will be released in arrow 57.1.0 (a minor update that won't require a datafusion upgrade).
So once we have released datafusion 51 we'll be able to do a minor update to arrow 57.1.0
(in other words, I don't think the release timeline of DataFusion will impact the flow of that PR)
xudong963 commentedon Nov 3, 2025
FYI, I plan to cut the 51.0.0 in the next two days. Do we have some high-priority PRs waiting for review?
pepijnve commentedon Nov 3, 2025
#18444 and #18183 are the remaining two for the
CASEepic. Would be nice if we could still get those in.alamb commentedon Nov 3, 2025
@Omega359 do we have additional work needed for the default timezone fixes?
time_zonetoNone(was"+00:00") #18359Omega359 commentedon Nov 3, 2025
@alamb - the outstanding ones I've got on my radar are
to_timestampand related functions #18025I'd like to get the first one in for 51.0.0. The other two while nice to have I do not see as critical
alamb commentedon Nov 4, 2025
Thanks @Omega359 -- added it to the list above
I also think we need to fix this regression:
shehabgamin commentedon Nov 4, 2025
We're working on testing Sail on our end.
37 remaining items
xudong963 commentedon Nov 13, 2025
It seems like all PRs are added to branch-51, cc @comphead @timsaucer
Time to start the RC2
xudong963 commentedon Nov 14, 2025
I think we still need to update changelog before releasing, will do later
xudong963 commentedon Nov 14, 2025
Could anyone help do this? My network has been bad for a long time:
alamb commentedon Nov 14, 2025
Done!
BTW this happens to me when I don't use my github token (the rate limit is apparently much stricter). I made a PR to add a note to the docs:
[branch-51] Update changelog for 51.0.0 RC2 (#18705)
[main] Update changelog for 51.0.0 RC2 (#18710)
[main] Update changelog for 51.0.0 RC2 (apache#18710)
xudong963 commentedon Nov 15, 2025
@alamb thank you, just started the vote process https://lists.apache.org/thread/k2ygp0c1rp56ww44lljf0yg6qf60q5f7
Minor: Add docs to release/README.md about rate limits (#18704)
xudong963 commentedon Nov 19, 2025
Done
alamb commentedon Nov 19, 2025
Thank you @xudong963 -- amazing as alwasy
[main] Update changelog for 51.0.0 RC2 (apache#18710)
Minor: Add docs to release/README.md about rate limits (apache#18704)
Upgrade DF to 51 (#5189)