Repository navigation
[EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) #13648
Description
Is your feature request related to a problem or challenge?
This is broken out from a more general ticket here
🥳 In my opinion DataFusion is now good enough (performance and feature wise) for many people to have buit real systems and products
However, as more people build "real" systems using DataFusion, our historic "move fast and break things and hope you can keep up" mentality likely needs to adjust to a more mature "move as fast as possible, but minimize breakages" type response.
My summary of the discussion on #13525 from @findepi @scsmithr @waynexia @timsaucer @Rachelint @Omega359 @jonmmease @Dandandan and @andygrove was that many existing heavy users of DataFusion spend a lot of time during upgrades from one DataFusion release to another
Specifically, I think the core challenge I heard was NOT the mechnical API changes required, but the effort required to diagnose more suble issues such as:
- Queries that used to work stopped working
Describe the solution you'd like
I would like to improve the ease of upgrading DataFusion versions
There are many ways to do so and I would like to use this ticket to capture / organize the work in this area
Related Items
- [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525
- [Epic] A Collection of Additional UTF8View support tickets #13504
- Upgrade from 40 to 43 causes utf8 timestamp queries to fail #13625
- [DISCUSS] Document criteria for adding new features / what belongs in core DataFusion (e.g. sql syntax, functions, etc) #12357
- [Discuss] Release cadence / patch releases / Long Term Supported (lts) minor releases #5269
- Automatically check "invariants" #13652
- [EPIC] Run full sqllogic / sqlite test suite against DataFusion #13811
- [DISCUSS] More extensive pre-release testing #13661
- Use
cargo semver-checksfor release testing #13665 - Minor: improve the Deprecation / API health guidelines #13701
- [DISCUSS] DataFusion less frequent major / breaking releases (ease using multiple third-party extensions (like delta, or iceberg) ) #16622
- Adds script to detect breaking API changes/ semver #16541
Additional testing
- [DISCUSSION] More SqlLogicTest test coverage for queries, including join queries #13470
- Systematic fuzz testing for parquet predicate pushdown #12115
- Aggregation fuzz testing #12114
Activity
jonathanc-n commentedon Dec 4, 2024
Yes, I think one of the comments in that discussion mentioned that certain changes that would cause breakage should be mentioned in every release. So before release, we should list out the possible changes that would need to be made if a upgrade were to happen during the development process.
alamb commentedon Dec 4, 2024
I think it is a great idea. The challenge will be identifying such changes I thunk
comphead commentedon Dec 4, 2024
There is an interesting approach at MariaDB, they generate queries with different syntaxes to find regressions.
Basically we can take their main.sql file which is 7MB of different queries including join queries and adapt it to DF.
There is no answers check, just smoke test that query can run successfully
The example can be found https://github.com/mariadb-corporation/mariadb-qa/tree/master/pquery
@alamb WDYT? it looks like a low hanging fruit, we can take the file and run it in latest datafusion CLI as part of CI or major release verification process
alamb commentedon Dec 4, 2024
I think in general the more testing we have the better. This idea sounds good to me -- I think more fully leveraging @2010YOUY01 's integration into sqlancer is also quite interesting.
Let's try and write some tickets to capture these ideas too - I can spend some time working on this over the next day or two
Omega359 commentedon Dec 4, 2024
Almost absolutely NOT. https://github.com/mariadb-corporation/mariadb-qa/blob/master/LICENSE.md
https://www.apache.org/legal/resolved.html#category-x
comphead commentedon Dec 4, 2024
Thats frustrating. Lets see if sqlancer can generate something similar.
crepererum commentedon Dec 5, 2024
I would like to propose #13665 as well.
comphead commentedon Dec 5, 2024
Here is a cockroachDB article that might be found interesting on fuzz testing the DB with generating random(valid/not valid) statements through SQLSmith
https://www.cockroachlabs.com/blog/sqlsmith-randomized-sql-testing/
https://www.cockroachlabs.com/blog/testing-random-valid-sql-in-cockroachdb/
I understand SQLancer is the way, but we may consider this as an alternative just in case
alamb commentedon Dec 20, 2024
I found a good previous discussion by @andygrove about potential patch releases:
alamb commentedon Jun 30, 2025
I added a ticket to discuss less frequent major releases
jonmmease commentedon Jun 30, 2025
As I've come to use Claude Code more for development, I've found that it's pretty good at performing version updates for Rust crates using Release Note + Compiler Errors + Tests. If each breaking change were required to have migration instructions, with examples, in the release notes (or a separate migration guide), I think the upgrade process could be pretty reliable and automated with LLM assistance. And of course this would be useful for the humans as well.
alamb commentedon Jun 30, 2025
That is a great idea @jonmmease -- I wonder if you have seen the new upgrade notes for the last few releases
jonmmease commentedon Jun 30, 2025
Thanks, no I hadn't come across these, and the look great!. My habit was just to search for the CHANGELOG when upgrading. I'll definitely review these for the next upgrade, and they certainly look like great LLM context as well.
alamb commentedon Jun 30, 2025
We should probably update the changelog generator script to include a link to the upgrade guide 🤔 (filed in #16626)
alamb commentedon Oct 23, 2026
I think this one has served its usefulness and most items have been completed so closing