Blog post for the DataFusion 51.0.0 release #18548

Activity

1. pepijnve commented on Nov 8, 2025

   

   pepijnve

   on Nov 8, 2025

   Last edited by pepijnve

   Contributor

   More actions

   Not a very exciting topic, but if you want some info for the blog post on the case work, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.

   I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.

   Just a case expression

   select case l_returnflag
     when 'R' then date_part('year', l_shipdate)
     when 'A' then date_part('year', l_commitdate)
     when 'N' then date_part('year',l_receiptdate)
     else null end
   from lineitem;
   

   50.0.0: ~0.492 seconds
   51.0.0: ~0.278 seconds

   *

   select * from lineitem
   

   50.0.0: 0.668 seconds
   50.1.0: 0.668 seconds

   * + a case expression

   select *, case l_returnflag
     when 'R' then date_part('year', l_shipdate)
     when 'A' then date_part('year', l_commitdate)
     when 'N' then date_part('year',l_receiptdate)
     else null end
   from lineitem;
   

   50.0.0: ~1.287 seconds -> case adds 0,619 seconds
   51.0.0: ~0.850 seconds -> case adds 0,182 seconds
2. alamb commented on Nov 9, 2025

   

   alamb

   on Nov 9, 2025

   ContributorAuthor

   More actions

   Not a very exciting topic, but if you want some info for the blog post on the case work, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.

   I am excited!

   I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.

   Thank you for the offer -- unless you already have one or would like to spend time writing up a more full features blog about how to optimize CASE evaluation, I think some unscientific numbers are sufficient

   Speaking of a blog, I am not aware of any practical writeup of implementing optimized CASE (conditional) evaluation in a vectorized query engine (aka like DataFusion). I would be willing to help write such a post, but if so I would want to make it a high quality one:

   1. background of CASE / conditional
   2. Basic evaluation strategy
   3. Why it is hard to make CASE fast
   4. All the crazy stuff that we did to actually make it fast in DataFusion
3. pepijnve commented on Nov 11, 2025

   

   pepijnve

   on Nov 11, 2025

   Contributor

   More actions

   I've placed a first draft written entirely by Claude AI at apache/datafusion-site#122 to get the ball rolling. It correctly captured the essence of the original issues and the changes the were made, but the post still needs quite some editing.
4. alamb commented on Nov 12, 2025

   

   alamb

   on Nov 12, 2025

   ContributorAuthor

   More actions

   thanks @pepijnve -- I'll try and review it over the next few days
5. Omega359 commented on Nov 12, 2025

   

   Omega359

   on Nov 12, 2025

   Contributor

   More actions

   PR's in this release (approx based on dates) - https://github.com/apache/datafusion/pulls?q=is%3Apr+merged%3A2025-09-16..2025-11-08

   👍React with 👍1Reacted by Andrew Lamb
6. alamb commented on Nov 13, 2025

   

   alamb

   on Nov 13, 2025

   ContributorAuthor

   More actions

   Another source of data is https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md
7. alamb

   assigned

   alamb

   and

   pepijnve

   on Nov 17, 2025
8. alamb

   closed this as completedin datafusion-site#124on Nov 25, 2025
9. alamb commented on Nov 25, 2025

   

   alamb

   on Nov 25, 2025

   ContributorAuthor

   More actions

   The blog post is now live: https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/