Repository navigation
Release DataFusion 43.0.0 #12470
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Last release was https://crates.io/crates/datafusion/42.0.0 September 17th, 2024 so next major release would be around October 20, 2024
Prior release tickets:
Desired Items that would be good to get into this release:
- (Andrew's goal) Enable reading StringView by default from Parquet (
schema_force_string_view) by default #11682 - (Andrew's goal) Enable parquet filter pushdown (
filter_pushdown) by default #3463
Activity
andygrove commentedon Oct 4, 2024
I'm keen to start the 43.0.0 release process as soon as we have upgraded to arrow-rs 53.1.0 since it will unblock apache/datafusion-ray#10
alamb commentedon Oct 4, 2024
The arrow upgrade is all ready here: #12724
Since the arrow upgrade is a minor version, you won't have to wait for the datafusion upgrade to get the latest version of arrow-rs
andygrove commentedon Oct 15, 2024
It has been 28 days since the last release to crates.io, so we should start planning this release.
@alamb would you still like to wait for the items mentioned in the description?
alamb commentedon Oct 15, 2024
Thanks @andygrove -- I think we should start planning. Here are some items I think we should include:
Required
I would love to get these as well
53.2.0(~November~ October 2024) arrow-rs#6341 -- though I could make a release sooner if you are willing to hold the DF release until next week)Other than #12788 we are on track I think to be ready to release in the next day or two
andygrove commentedon Oct 17, 2024
Thanks @alamb. I am in no rush for the release myself.
A "nice to have" for me is #12969 if it is ready in time, but it should not block the release (and could potentially be back-ported to a 42.x.x release)
alamb commentedon Oct 20, 2024
BTW I am going to accelerate the timeline to release arrow 53.2.0 so we can potentially include it in the next datafusion release: apache/arrow-rs#6341
I hope to make the RC today
alamb commentedon Oct 23, 2024
This came up on the sync call today
I would like consider doing this item before we release (as this check has caused significant pain downstream in InfluxDB 3.0 as well as delta-rs (see links on ticket):
Physical input schema should be the same as the one converted from logical schemaerror #13065If we did that I view everything else as nice to have, including
schema_force_string_view) by default #11682 as much as I would like to turn this on by default, I think it would be a bad idea to turn it on by default right before a release. It would be better to merge it to main and give it some bake timealamb commentedon Oct 25, 2024
Here is the PR to use StringView when reading from Parquet files: #13101
findepi commentedon Oct 26, 2024
There is a regression for CREATE TABLE #12864 (comment)
This should be resolved before release (or judged OK to break).
alamb commentedon Oct 26, 2024
Filed I have filed #13124 to track the
alamb commentedon Oct 29, 2024
I am getting worried about the number of changes that are accumulating unreleased.
Here is my proposal:
Physical input schema should be the same as the one converted from logical schemaerror #13065 (any volunteers to help with this?)alamb commentedon Nov 1, 2024
@andygrove what are your thoughts timingwise for 43? I feel like we are accumulating substantial API changes and it might be nice to release sooner rather than later
timsaucer commentedon Nov 4, 2024
Do you need an assist with any portion of this?
alamb commentedon Nov 4, 2024
Thank yoU @timsaucer ❤️
I am not sure -- I haven't heard from @andygrove -- perhaps we should ask him in Slack?
If he is willing, I think it would be great if you wanted to create the PR to update the versions and readme as documented in https://github.com/apache/datafusion/blob/main/dev/release/README.md#change-log
andygrove commentedon Nov 4, 2024
I am here (now). I can start the RC process this evening if we are not waiting on anything else
timsaucer commentedon Nov 4, 2024
I would like to get this in, but I have yet to find a reviewer: #13183
alamb commentedon Nov 10, 2024
The release has been approved and published to crates.io
Minor follow on:
Tracking
44.0.0:44.0.0#13334alamb commentedon Nov 10, 2024
Thanks again @andygrove