[DISCUSS] Release retrospective

@mbutrovich

This topic came up in the recent DataFusion community meeting. There is interest in having a retrospective of our current release practices.

Background

DataFusion major version 52 ended up with 4 releases and another underway:

At the same time we are trying to release 53.0.0. Trying to release patches on past versions while simultaneously releasing a new major version may split maintainer time and focus in undesirable ways.

Additional references

Discussion about the frequency of releasing: #16622

We are currently updating the release documentation: #21001

Request

This issue is to collect feedback from all community members about the recent release process. The discussion above is where the topic of frequency should happen.

There is a plan to have an in-person retrospective at the next community meeting, March 25. @mbutrovich has volunteered to try capturing notes from that meeting.

For people who want to share thoughts asynchronously or cannot attend the community meeting, please give us your feedback about the recent release process.

What went well?
What could be improved?
What did we learn?
What should we commit to changing?

A list of things to ensure nothing gets lost

git tag validation requested @coracuity
release management doc update
put publish crates list in correct order
Example for Download and unpack the official release tarball like wget from https://dist.apache.org/repos/dist/release/datafusion/datafusion-53.0.0/
Followup on Release DataFusion 53.0.0 (Feb 2026 / Mar 2026) #19692 (comment) Need to ensure package compiles before doing RC

@timsaucer

Thanks @timsaucer for driving this.

While fighting with DF 53 release I would like to share some experience that seemed reasonable to me

What went well?

Overall community reaction went well IMO, community made its parts brillianty:

responsibly tests RC
make quick turnaround on found bugs or provide workarounds
support with backports

What could be improved?

Release documentation can be improved:

some steps(more specific backports and some other minors) in release process can be elaborated, I'll create a PR soon to address this

Code quality can be improved:

it is fine to support more than 1 release, like Apache Spark supports versions 3 and 4 at the same time, but having so many hotfixes suggests some regressions are found on later stages.

What did we learn?

Still processing this

What should we commit to changing?

IMO release process needs to be refined
Prerelease verification might include fuzz testing(currently it is pretty limited set), or quasi fuzz testing where genAI generates thousands of valid queries with normal and extreme values
Prerelease verification can also include planner and query benchmarking to identify performance regressions

some steps(more specific backports and some other minors) in release process can be elaborated, I'll create a PR soon to address this

FYI I made one here to try and improve the backporting docs:

Add release management page to the documentation #21001

@timsaucer

What went well?

We fixed many bugs quickly and efficiently

It was a great team effort (we had multiple releases run by multiple different people - @timsaucer @comphead @xudong963 and myself 🎉

I actually also think the 53 release will be higher quality for having sat as a branch for a while (not immediately making a RC). While it was somewhat of a pain to backport PRs, that can be mostly automated and I think it gave us a chance to stabalize the release more.

What could be improved?

I think it would help to more explicitly focus on regression bugs (most of what triggered the 52 releases) -- in my mind this means

Having some way to flag regressions when they are filed in issues/PRs (so we can find and prioritize their reviews and fixes)
Making it clear that our priority is (at least my priority is) fixing regression bugs

What did we learn?

Running these releases takes a long time

What should we commit to changing?

I think we need to consider how to potentially slow down our release scheduled (to one major release every quarter, for example, rather than every month) or marking some as LTS releases where we have some way to make bug fixes without also new features.

I wanted to call attention to this situation as an opportunity to improve the release process: #19692 (comment)

Specifically, how did bad tags get pushed up? Can we prevent it or automate it such that it's impossible or improbable?

that can be mostly automated and I think it gave us a chance to stabalize the release more.

This sounds appealing! (backport is annoying during release for me haha)

This sounds appealing! (backport is annoying during release for me haha)

Agree, some strategy needed there, because a single backport involves:

backport PR
update change list
generate new tag
generate new RC
voting
repeat all above for all ongoing other hotfix branches

strategy needs to define when to make RC, lets say it can be once a day or something to avoid unnecessary work

marking some as LTS releases where we have some way to make bug fixes without also new features

+1 on this — having an LTS release could also reduce the number of backports needed for critical bugs and correctness issues, since you'd only need to target the LTS branch rather than multiple past releases.

Orthogonally, I think it simplifies the process too: any backport PR to the LTS branch should either reference the corresponding fix in main/latest, or explicitly note that the bug doesn't affect newer releases (e.g., already resolved by a behavioral change).

How would LTS style releases work when it comes to other common libraries with DataFusion deps? Mainly comet and iceberg, would the suggestion be for them to track LTS?

How would LTS style releases work when it comes to other common libraries with DataFusion deps? Mainly comet and iceberg, would the suggestion be for them to track LTS?

This is a really good question -- in my mind the answer would be yes but that would potentially slow down feature flow to those other libraries

Another thing we could consider would be to switch to doing major (breaking API) releases only every other month or every three months (similarly to how we do for arrow-rs) to give downstream crates more time to catch up

This would require more diligence on DataFusion's part to avoid breaking API changes as much 🤔 which is annoying for contributions to this library but makes life for downstream crates better

From my personal perspective I only ever track releases every 3 months at best so switching to having a LTS release would be beneficial.

As for the actual release process I've rarely had the time to test a release against my codebase within the actual release window. I'm just too busy most of the time to get to it within a week of the rc being cut. I suspect others are in a similar situation which leads to integration tests not being run that would otherwise have found issues.

I don't know if we have an official suite of benchmarks that are run on release candidates but if not I think it would be a really good idea to have those run comparing the results to the previous release(s). clickbench, tpch, tcpds, some limited # of criterion benchmarks (planning performance, etc) for example.

One thing that came up in the call today was that a lot of regressions come from lack of benchmarking. We're making efforts to add more benchmarks but Codspeed or similar system to track regressions across PR merges could help us catch these things before releases. I plan to make a tracking issue for this today. I know Bruce is working on this as well.

I don't know if we have an official suite of benchmarks that are run on release candidates but if not I think it would be a really good idea to have those run comparing the results to the previous release(s). clickbench, tpch, tcpds, some limited # of criterion benchmarks (planning performance, etc) for example.

Agreed. We often run benchmarks for a release's blog post after the fact, but some set of benchmarks should become a criteria for accepting a release candidate.

I opened #21165 to track benchmark improvements

Navigation Menu

Repository navigation

[DISCUSS] Release retrospective #21034

Background

Additional references

Request

A list of things to ensure nothing gets lost

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[DISCUSS] Release retrospective #21034

Description

Background

Additional references

Request

A list of things to ensure nothing gets lost

Activity

comphead commented on Mar 18, 2026

alamb commented on Mar 18, 2026

alamb commented on Mar 18, 2026

coracuity commented on Mar 18, 2026

xudong963 commented on Mar 19, 2026

comphead commented on Mar 19, 2026

hareshkh commented on Mar 20, 2026

xanderbailey commented on Mar 20, 2026

alamb commented on Mar 22, 2026

alamb commented on Mar 22, 2026

Omega359 commented on Mar 25, 2026

adriangb commented on Mar 25, 2026

mbutrovich commented on Mar 26, 2026

adriangb commented on Mar 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions