Repository navigation
[DISCUSS] Bug bash / slow down new features while we work on bugs #26865
Description
Recently there has been an explosion of new code going into DataFusion, as well as an explosion of new bug reports. Obviously these are both driven by new AI tools and we are all struggling to figure out how to adjust to the new software engineering realty
DataFusion is also getting close to the 56 release
We also have a number of outstanding wrong results bugs:
Thus I would like to discuss slowing down feature merging for a while ( a few weeks?) while we sort out the bugs and harden the features / testing / docs we do have.
I think we already risk this project running out of control / become unmaintainable (at least I would not be interesting in maintaining it) if we keep accepting new features at such a clip and accumulating maintenance surface and technical debt
I would suggest something like:
- maintainers focus on bug fixes, dev tooling improvements, simplifications, etc for the next few weeks
- Prepare new feature PRs / test them / review them of course, but don't merge them until after 56 is out
- Be even more aggressive about filtering out PRs that are risky or from non maintainers / existing community members
Activity
[-][DISCUSS] Bug bash / reduce new features for a while while we work on bugs[/-][+][DISCUSS] Bug bash / slow down new features while we work on bugs[/+]alamb commentedon Oct 28, 2026
Some other thoughts are to take this time to make sure important properties of the system are covered appropriately with tests, for example the IO patterns in parquet reading came up in #25811
edmondop commentedon Oct 28, 2026
Two years ago I suggested we implement mutation testing as a scheduled job but we absorbed the idea is there more appetite right now? #14590
alamb commentedon Oct 28, 2026
My pushback to more automated testing in the repo is that we need someone to actively watch it / maintain it. Is there any reason we can't use mutation testing outside the core repo ?
edmondop commentedon Oct 28, 2026
I think it's a fair concern and the answer should probably depend on our assessment of the situation. Given what you said about the project risks, I think a systematic solution would be more effective. A "one-off" effort could improve the situation in the short term, but if we believe there are forces that lead to this situation that aren't changing, it won't be sufficient
In favor of mutation testing in the repo
If we believe these bugs are the consequence of a decrease in the effectiveness of testing, because the system has become complex and/or because AI generates a part of the code, in this case I think it is a good idea to enable mutation testing. It will fail like so (taken from #14590)
this is telling us that the physical optimizer tests pass when changing a comparison function from OR to AND, and therefore we should improve these tests. We can run this on a schedule, and mutation testing will help us have better tests, and identify and fix bugs before they are observed at runtime.
Against mutation testing in the repo
If we believe these bugs are the inevitable defects that gets generated through software development, we don't think it is becoming harder to have write comprehensive and robust tests, then mutation testing would be too much of a burden
raghavyadav01 commentedon Oct 29, 2026
+1 from a contributor. Hardening what is there before 56 makes sense.
One thing I can take on that doesn't need review bandwidth: go through the wrong-results list in #26672 and the subquery list in #5492, rerun each repro on current main, and post which still reproduce and which are already fixed, so they can be closed or ranked. Happy to start if that's useful.
2010YOUY01 commentedon Oct 29, 2026
Thanks for starting the discussion! A feature freeze sounds like a good idea worth trying. Perhaps we could start with a two-week freeze before each major release? Initially, the freeze could be a guideline rather than a hard limit.
One catch is that even bug fixes can introduce technical debt. A typical case is a mostly AI-generated fix whose author don't fully understand the changes. AI tends to optimize for the easiest fix with minimal LoC, rather than the most maintainable solution.
With a flood of such PRs, our review bandwidth can't keep up. We can't always find the most qualified reviewer for each PR, and the sheer volume of PRs gradually lowers the bar for approval. This seem like another issue to deal with.
2010YOUY01 commentedon Oct 29, 2026
I think this concern is very reasonable for SQL fuzzing, their bug reports are typically very hard to investigate, and also have lots of false positives.
But mutation tests are much easier to understand and fix. The issue with the previous attempt, IIRC, was that the full test run took way too long. As long as someone can set up CI to run mutation tests only on new UTs in the PR, I'm happy to review it.