Repository navigation
[DISCUSS] Lower Friction / Lower Ceremony (faster?) releases #14428
Description
Is your feature request related to a problem or challenge?
Releasing DataFusion versions is described here:
We have been releasing about once a month for the last three years. You can see the history here:
However, there is some non trivial overhead to doing releases as @findepi has observed, such as the tarball creation and voting process.
Interestingly I believe that one of @jorgecarleitao 's concerns about arrow-rs when he created arrow2 was the (slow) pace of the Apache release model
Describe the solution you'd like
I would like to discuss what we can do to make releasing easier (and therefore potentially more frequent)
Describe alternatives you've considered
No response
Additional context
Reacted by Piotr Findeisen
Activity
alamb commentedon Feb 3, 2025
I have spent quite a bit of effort working on the
54.0.0However in my mind most of that additional effort comes from the pre-release testing we are doing with various downstream projects rather than the specifics of releases. See this for more detail
alamb commentedon Feb 3, 2025
Another thing @findepi has mentioned is potentially publishing "non official" / "nightly" releases to crates.io to make it easier for downstream projects to test pre-releases.
xudong963 commentedon Feb 3, 2025
The suggestion makes sense to me.
westonpace commentedon Feb 3, 2025
I don't think
crates.io(orcargo) have support for a nightly release concept? What we've done in lance, for Rust, is to create pre-release tags. We still trigger them manually (e.g. it's not a nightly thing) but it's just a button press and very low friction.Downstream users have to then specify the dependency with the tag:
Overall, I'd say it works well. It also avoids any implication of an official release which is nice.
westonpace commentedon Feb 3, 2025
(ah, I see you can do
-beta,-alphatags, so definitely supported, just not in the same way as python)findepi commentedon Feb 4, 2025
Not sure I deserve credit for this idea. It's tracked by #5023.
Regardless, I think we can easily agree that having lower-overhead releases is a good thing.
Also, per se, It is better not to have two different types of releases (official and unofficial).
alamb commentedon Feb 4, 2025
I think we could automate creating the tarball / tags perhaps.
For what it is worth, this past release by far most of my time was spent chasing down bugs / tickets /testing (aka the "deciding whether to release" in your description).
findepi commentedon Feb 7, 2025
can "deciding whether to release" be "automated" by marking mandatory issues and PRs as release-blockers?
The more external testing we want to do, the harder is to release. Yes, this gives more confidence in particular release, but it costs precious time resources.
(downstream projects should be well motivated to contribute good tests for Datafusion, so that eventually the net result is more confidence without having to chase)
alamb commentedon Feb 7, 2025
I agree this is the fundamental tradeoff