Update ClickBench benchmarks with DataFusion 54.0.0 (when released) #21505

Activity

1. alamb

   self-assigned this

   on Jun 1, 2026
2. alamb commented on Jun 1, 2026

   

   alamb

   on Jun 1, 2026

   ContributorAuthor

   More actions

   I have been working on this. The ClickBench scripts have been refactored and I hit a few issues gathering info

   • Some, but not all scripts report "load time" for stateless engines (time to download parquet and sync) ClickHouse/ClickBench#944
   • fix(datafusion): Update docs, skip concurrent_qps, Skip re-downloading partitioned files ClickHouse/ClickBench#943

   However, now I think I have enough to regenerate the results
3. alamb commented on Jun 1, 2026

   

   alamb

   on Jun 1, 2026

   ContributorAuthor

   More actions

   Here are the current results (still not quite at DuckDB) but close
   

   Here is the url I used locally

   Here is the full result:

   ClickBench-branch-54.zip

   ❤️React with ❤️2Reacted by feniljain and Tormod Gjeitnes Hellen
4. alamb commented on Jun 18, 2026

   

   alamb

   on Jun 18, 2026

   ContributorAuthor

   More actions

   I am starting to work on this

   Currently ClickBench has reports for these combinations:

   ┌────────────────┬───────────────────┬────────────────────────────────────────┐
   │ Instance type  │    Arch (AMI)     │                 Notes                  │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c6a.xlarge     │ x86_64 (AMD)      │ 4 vCPU                                 │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c6a.2xlarge    │ x86_64 (AMD)      │ 8 vCPU                                 │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c6a.4xlarge    │ x86_64 (AMD)      │ 16 vCPU — the canonical/reference size │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c6a.metal      │ x86_64 (AMD)      │ bare metal                             │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c7a.metal-48xl │ x86_64 (AMD)      │ bare metal, 192 vCPU                   │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c8g.4xlarge    │ arm64 (Graviton4) │ 16 vCPU                                │
   ├────────────────┼───────────────────┼────────────────────────────────────────┤
   │ c8g.metal-48xl │ arm64 (Graviton4) │ bare metal, 192 vCPU                   │
   └────────────────┴───────────────────┴────────────────────────────────────────┘
   

   I am working to get some numbers
5. alamb commented on Jun 19, 2026

   

   alamb

   on Jun 19, 2026

   ContributorAuthor

   More actions

   Preliminary Results from

   • Update Results for DataFusion version to 54.0.0 ClickHouse/ClickBench#948

   We are technically faster than DuckDB at the very highest end:
   

   Though for the classic c6a.4xlarge we are still a bit slower
   

   🚀React with 🚀2Reacted by Daniël Heres and Ratul Dawar
6. alamb commented on Jun 19, 2026

   

   alamb

   on Jun 19, 2026

   ContributorAuthor

   More actions

   🤔 maybe I can write a blog post about this anyways (and highlight the differences in the hardware 🤔 )

   👍React with 👍1Reacted by Kumar Ujjawal
7. alamb commented on Jun 19, 2026

   

   alamb

   on Jun 19, 2026

   ContributorAuthor

   More actions

   Ok,

   • Update Results for DataFusion version to 54.0.0 ClickHouse/ClickBench#948 is ready for review
8. alamb commented on Jul 1, 2026

   

   alamb

   on Jul 1, 2026

   ContributorAuthor

   More actions

   Results are live

   https://benchmark.clickhouse.com/#system=+curp|fqo|ti%20rud|Dist|kBP%20t|bt(%20t|aseaa|rsoP%20t|Sat%20&type=-&machine=+ca4e&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

   DataFusion is looking pretty good

   
9. alamb

   closed this as completedon Jul 1, 2026