Release DataFusion 48.0.1 #16486

Activity

1. blaginin

   added

   enhancementNew feature or requestNew feature or request

   on Jun 20, 2025
2. blaginin commented on Jun 20, 2025

   

   blaginin

   on Jun 20, 2025

   MemberAuthor

   More actions

   posting here @alamb's comment for transparency:

   I think we can argue if that was a regression or not (it was a change of a default setting, though we are about to change it again in 49....)

   That sounds like a valid point to me. Nothing has seriously broken, and if it's too slow, users can tweak the config.

   Not sure if we want to fix the topk pushdown issues? Given that our own test suite is failing, can we sometimes produce incorrect results for users?
3. timsaucer commented on Jun 20, 2025

   

   timsaucer

   on Jun 20, 2025

   Member

   More actions

   If we do a patch release I would like to get in this bugfix that I found today: #16480
4. alamb commented on Jun 20, 2025

   

   alamb

   on Jun 20, 2025

   Contributor

   More actions

   I think the topK stuff was not included in 48.0.0 so there is no need to backport any fixes

   I volunteer to run the release process.

   In terms of fixing the regression in clickbench performance (which is due to statistics not being gathered by default for dataframes) with I think we have two options

   • backport the change of default value here: Set the default value of datafusion.execution.collect_statistics to true #16447 (will align with what 49.0.0 release)
   • revert the change in Make SessionContext::register_parquet obey collect_statistics config #16080 which made the defaults consistent
5. alamb commented on Jul 2, 2025

   

   alamb

   on Jul 2, 2025

   Contributor

   More actions

   I think this bug is enough to make me want to do s 48.0.1 release

   • Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588

   I will begin work on such a release later today

   ❤️React with ❤️1Reacted by Dmitrii Blaginin
6. alamb

   changed the title [-][Potentially] Release DataFusion `48.0.1`[/-] [+]Release DataFusion `48.0.1`[/+] on Jul 2, 2025
7. alamb

   self-assigned this

   on Jul 2, 2025
8. alamb commented on Jul 2, 2025

   

   alamb

   on Jul 2, 2025

   Contributor

   More actions

   The process will be:

   1. We will create backport PRs for fixes, following the instructions here https://github.com/apache/datafusion/blob/main/dev/release/README.md#how-to-backport-add-changes-to-branch--branch
   2. I'll plan to make an Release Candidate from branch-48 in the next day or two

   I will handle backporting

   • fix: column indices in FFI partition evaluator #16480
   • Make SessionContext::register_parquet obey collect_statistics config #16080
   • Fix parquet filter_pushdown: respect parquet filter pushdown config in scan #16646

   If anyone else has fixes or changes they would like to include in 48.0.1, please create a backport PR (targeting the branch-48 branch)

   ❤️React with ❤️1Reacted by Dmitrii Blaginin
9. alamb commented on Jul 2, 2025

   

   alamb

   on Jul 2, 2025

   Contributor

   More actions

   @xudong963 do you have additional suggestions for issues that should be backported?
10. alamb commented on Jul 2, 2025

    

    alamb

    on Jul 2, 2025

    Contributor

    More actions

    I have made some backport PRs:

    • [branch-48] Fix parquet filter_pushdown: respect parquet filter pushdown config i… #16656
    • [branch-48] fix: column indices in FFI partition evaluator (#16480) #16657

    I'll wait another day or two see if there is any more content needed for this release
11. blaginin commented on Jul 2, 2025

    

    blaginin

    on Jul 2, 2025

    Last edited by blaginin

    MemberAuthor

    More actions

    Also opened #16659 to discuss

    ❤️React with ❤️1Reacted by Andrew Lamb
12. alamb commented on Jul 4, 2025

    

    alamb

    on Jul 4, 2025

    Contributor

    More actions

    Ok, I have backported the three identified issues and I will now create a release candidate

    👍React with 👍2Reacted by Dmitrii Blaginin and Shehab Amin
13. alamb commented on Jul 5, 2025

    

    alamb

    on Jul 5, 2025

    Contributor

    More actions

    We have begun voting: https://lists.apache.org/thread/xobyfmpcvtcclcbgo0wz6c74b35mxcdx
14. xudong963 commented on Jul 6, 2025

    

    xudong963

    on Jul 6, 2025

    Member

    More actions

    @xudong963 do you have additional suggestions for issues that should be backported?

    Sorry, just come back, lgtm.

    👍React with 👍1Reacted by Andrew Lamb
15. alamb commented on Jul 7, 2025

    

    alamb

    on Jul 7, 2025

    Contributor

    More actions

    released to crates.io and all is good

    ❤️React with ❤️3Reacted by Douglas Anderson, Shehab Amin and Dmitrii Blaginin
16. alamb

    closed this as completedon Jul 7, 2025