Skip to content

[DISCUSS] More extensive pre-release testing #13661

New issue
New issue
Open
Open
[DISCUSS] More extensive pre-release testing#13661
Labels
enhancementNew feature or request

Description

@alamb
alamb
opened on Dec 5, 2024

Is your feature request related to a problem or challenge?

Up to now, when we have made DataFusion releases, we have mostly focused on validated that DataFusion's own unit tests have passed, (see dev/release/README.md) but haven't tested the upgrade with other downstream projects (like ballista, ray, InfluxDB IOx, etc) until after we have release the code

This results sometimes in downstream users finding issues after release. Some recent examples

Describe the solution you'd like

I would like to improve the testing / release process for DataFusion releases to reduce the number of regressions found after release.

This would likely take the form of updating the dev/release/README.md

Describe alternatives you've considered

One idea mentioned by @Omega359 and @andygrove on #13525 (comment)

Upgrading our own subprojects (Ballista, Comet, DF Python, DF Ray) as part of the DataFusion release process makes a lot of sense to validate that the upgrade guide is complete.

Additional context

No response

Activity

  1. alamb
    added
    enhancementNew feature or request
    on Dec 5, 2024
  2. alamb
    changed the title [-][DISCUSS] More deliberate pre-release testing[/-] [+][DISCUSS] More extensive pre-release testing[/+] on Dec 5, 2024

  3. alamb commented on Dec 5, 2024

    @alamb
    alamb
    on Dec 5, 2024
    ContributorAuthor

    BTW I often create WIP PRs to test releases of sqlparser-rs and arow-rs with DataFusion before the release

    For example:

    This caught at least one regression before release recently (apache/datafusion-sqlparser-rs#1556, fixed by @goldmedal 🙏 )

  4. findepi commented on Dec 5, 2024

    @findepi
    findepi
    on Dec 5, 2024
    Member

    I would like to improve the testing / release process for DataFusion releases to reduce the number of regressions found after release.

    I agree with the goal, but i am concerned about the cost/overhead involved. We don't have infinite bandwith at disposal.
    Additionally, orchestrating testing across several downstream projects creates new problems we didn't have before

    • which downstream projects can stop the release train? what kind of problems are able to stop the release train?
      • this is especially important question for closed-source downstream projects
    • how long are we willing to wait for external teams to report back? Downstream project maintainers will obviously be willing and motivated to help, but their availability cannot be assumed

    So what if we focused instead on:

    • improving testing within DF itself? if a downstream project is concerned about stability of feature X, they can contribute to improve test coverage for feature X (eg [DISCUSSION] More SqlLogicTest test coverage for queries, including join queries #13470)
    • easy low-ceremony low-overhead (automated) releases. If we see a big problem / regression after a major release, this can be patched on a maintenance branch. A maintenance branch can release daily without human intervention.

  5. alamb commented on Dec 5, 2024

    @alamb
    alamb
    on Dec 5, 2024
    ContributorAuthor

    I agree with the goal, but i am concerned about the cost/overhead involved. We don't have infinite bandwith at disposal.
    Additionally, orchestrating testing across several downstream projects creates new problems we didn't have before

    Yes, this is true. I imagine an incremental rollout type approach -- where we start with one project (datafusion-python is a natural example, and maybe we can get delta-rs to help too). In my (likely naieve) thinking the downstream projects will be willing to help as they are directly affected

  6. alamb commented on Dec 5, 2024

    @alamb
    alamb
    on Dec 5, 2024
    ContributorAuthor

    easy low-ceremony low-overhead (automated) releases. If we see a big problem / regression after a major release, this can be patched on a maintenance branch. A maintenance branch can release daily without human intervention

    It is my understanding that the apache voting / approval process prevents automated builds

    Creating a maintenance branch is also a compelling idea where we can focus on stability / shoring up test coverage 🤔

  7. findepi commented on Dec 7, 2024

    @findepi
    findepi
    on Dec 7, 2024
    Member

    It is my understanding that the apache voting / approval process prevents automated builds

    That's my understanding too, but i hope this process isn't nonnegotiable.
    Processes are there to serve the project & the community after all, not the other way around.

  8. alamb commented on Dec 9, 2024

    @alamb
    alamb
    on Dec 9, 2024
    ContributorAuthor

    It is my understanding that the apache voting / approval process prevents automated builds

    That's my understanding too, but i hope this process isn't nonnegotiable. Processes are there to serve the project & the community after all, not the other way around.

    I think as long as we make it clear that nightly builds are not "official" releases from the ASF point of view, we could create / publish them. 🤔

  9. findepi commented on Dec 10, 2024

    @findepi
    findepi
    on Dec 10, 2024
    Member

    That would work for me as long as these releases are the only once we publish. I would want automation for 'the releases' the people use. Especially if we have a maintenance branch, it would reasonable to release after every PR merge.
    I don't know what problems the manual release process solves that cannot be solved with automated releases.

    anyway, did we hijack the thread?

  10. alamb commented on Dec 10, 2024

    @alamb
    alamb
    on Dec 10, 2024
    ContributorAuthor

    anyway, did we hijack the thread?

    Yeah, we should probably file a separate discussion about more frequent releases if we want to pursue that option

  11. edmondop commented on Feb 10, 2025

    @edmondop
    edmondop
    on Feb 10, 2025
    Contributor

    @alamb I did some personal testing with Cargo mutants. We can have a manually runnable pipeline, or something scheduled every day, that could run mutation testing and inform us about the weakness of our tests

    I am happy to issue a PR if you think it's a good idea

  12. alamb commented on Feb 10, 2025

    @alamb
    alamb
    on Feb 10, 2025
    ContributorAuthor

    @alamb I did some personal testing with Cargo mutants. We can have a manually runnable pipeline, or something scheduled every day, that could run mutation testing and inform us about the weakness of our tests

    I am happy to issue a PR if you think it's a good idea

    I am not sure what mutation testing means -- I would be happy to check out a PR that had such a thing

    As for running, we could add it to the "extended" test maybe: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml

  13. edmondop commented on Feb 10, 2025

    @edmondop
    edmondop
    on Feb 10, 2025
    Contributor

    I have logged this @alamb there are some more details in that issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions