Make ClickBench Q23 Go Faster

Is your feature request related to a problem or challenge?

Part of [EPIC] ClickBench Improvements (Vanity Benchmark) #14586

Comparing ClickBench on DataFusion 45 and DuckDB (link)

You can see that for 23 DataFusion is almost 2x slower (around 10s where DuckDB is 5s)

You can run this query like this:

cd datafusion
cd benchmarks
# download data
./bench.sh data clickbench_partitioned
# run query with datafusion-cli (note escapes
datafusion-cli -c "SELECT * FROM 'data/hits_partitioned' WHERE \"URL\" LIKE '%google%' ORDER BY \"EventTime\" LIMIT 10;"

Here is the explain plan

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion/benchmarks$ datafusion-cli -c "EXPLAIN SELECT * FROM 'data/hits_partitioned' WHERE \"URL\" LIKE '%google%' ORDER BY \"EventTime\" LIMIT 10;"
DataFusion CLI v46.0.0
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Sort: data/hits_partitioned.EventTime ASC NULLS LAST, fetch=10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|               |   Filter: CAST(data/hits_partitioned.URL AS Utf8View) LIKE Utf8View("%google%")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|               |     TableScan: data/hits_partitioned projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], partial_filters=[CAST(data/hits_partitioned.URL AS Utf8View) LIKE Utf8View("%google%")]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| physical_plan | SortPreservingMergeExec: [EventTime@4 ASC NULLS LAST], fetch=10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|               |   SortExec: TopK(fetch=10), expr=[EventTime@4 ASC NULLS LAST], preserve_partitioning=[true]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|               |     CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|               |       FilterExec: CAST(URL@13 AS Utf8View) LIKE %google%                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|               |         DataSourceExec: file_groups={16 groups: [[Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_0.parquet:0..122446530, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_1.parquet:0..174965044, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_10.parquet:0..101513258, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_11.parquet:0..118419888, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_12.parquet:0..149514164, ...], [Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_14.parquet:108113265..151121699, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_15.parquet:0..103098894, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_16.parquet:0..101067219, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_17.parquet:0..116867853, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_18.parquet:0..133119589, ...], [Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_21.parquet:3887560..113455196, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_22.parquet:0..79775901, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_23.parquet:0..79631107, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_24.parquet:0..78257049, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_25.parquet:0..144169728, ...], [Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_28.parquet:106905624..162772407, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_29.parquet:0..79213288, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_3.parquet:0..192507052, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_30.parquet:0..124187913, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_31.parquet:0..123065410, ...], [Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_35.parquet:54087340..153632381, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_36.parquet:0..92487304, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_37.parquet:0..108247781, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_38.parquet:0..132005180, Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/hits_39.parquet:0..103522954, ...], ...]}, projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], file_type=parquet, predicate=CAST(URL@13 AS Utf8View) LIKE %google% |
|               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.056 seconds.

Something that immediately jumps out at me in the explain plan is this line

|               |         DataSourceExec: file_groups={16 groups: ...}, projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], file_type=parquet, predicate=CAST(URL@13 AS Utf8View) LIKE %google% |

"Projection" I think means that all of those columns are being read/ decoded from parquet, which makes sense as the query has a SELECT * on it.

However, in this case all but the top 10 rows are returned (out of 100M rows in the file)

So this means that most of the decoded data is decoded and thrown away immediately

Describe the solution you'd like

I would like to close the gap with DuckDB with some general purpose improvement

Describe alternatives you've considered

I think the way to improve performance here is to defer decoding ("Materializing") the other columns until we know what the top 10 rows are.

some wacky ideas:

Push the topk / ordering into the scan somehow
implement "late materialization"

Late materialization would look something like

decode only the EventTime column and a row_id
determine the top 10 row_id by sorting by EventTime
Decode only those 10 rows from the parquet file(s)

Additional context

No response

@adriangb

OOOO -- here is the duckdb plan and it shows what they are doing!

The key is this line:

│          Filters:         │
│ optional: Dynamic Filter  │
│        (EventTime)        │

What I think this is referring to is what @adriangb is describing in :

Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037

Specifically, the Top_N operator passes down a filter into the scan. The filter is "dynamic" in the sense that

the TOP_N operator knows what the smallest maximum value currently is
That means the scan can filter rows where the current timestamp is less than that number

┌─────────────────────────────┐
│┌───────────────────────────┐│
││       Physical Plan       ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│           TOP_N           │
│    ────────────────────   │
│          Top: 10          │
│                           │
│         Order By:         │
│ memory.main.hits.EventTime│
│             ASC           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│    ────────────────────   │
│  contains(URL, 'google')  │
│                           │
│       ~20000000 Rows      │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│       PARQUET_SCAN        │
│    ────────────────────   │
│         Function:         │
│        PARQUET_SCAN       │
│                           │
│        Projections:       │
│          WatchID          │
│         JavaEnable        │
│           Title           │
│         GoodEvent         │
│         EventTime         │
│         EventDate         │
│         CounterID         │
│          ClientIP         │
│          RegionID         │
│           UserID          │
│        CounterClass       │
│             OS            │
│         UserAgent         │
│            URL            │
│            ...            │
│      ParamCurrencyID      │
│    OpenstatServiceName    │
│     OpenstatCampaignID    │
│        OpenstatAdID       │
│      OpenstatSourceID     │
│         UTMSource         │
│         UTMMedium         │
│        UTMCampaign        │
│         UTMContent        │
│          UTMTerm          │
│          FromTag          │
│          HasGCLID         │
│        RefererHash        │
│          URLHash          │
│            CLID           │
│                           │
│          Filters:         │
│ optional: Dynamic Filter  │
│        (EventTime)        │
│                           │
│      ~100000000 Rows      │
└───────────────────────────┘

The topk dynamic filtering is described here:
https://www.snowflake.com/en/blog/super-fast-top-k-queries/

BTW apparently DuckDB uses the "late materialization" technique with its own native format. Here is an explain courtesy of Joe Issacs and Robert Kruszewski

┌─────────────────────────────┐
│┌───────────────────────────┐│
││       Physical Plan       ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #0            │
│__internal_decompress_integ│
│    ral_smallint(#1, 0)    │
│             #2            │
│__internal_decompress_integ│
│    ral_smallint(#3, 1)    │
│             #4            │
│             #5            │
│             #6            │
│             #7            │
│             #8            │
│             #9            │
│__internal_decompress_integ│
│    ral_smallint(#10, 0)   │
│__internal_decompress_integ│
│    ral_smallint(#11, 0)   │
│__internal_decompress_integ│
│    ral_smallint(#12, 0)   │
│            #13            │
│            #14            │
│            ...            │
│            #90            │
│            #91            │
│            #92            │
│            #93            │
│            #94            │
│            #95            │
│            #96            │
│            #97            │
│            #98            │
│            #99            │
│            #100           │
│__internal_decompress_integ│
│   ral_smallint(#101, 0)   │
│            #102           │
│            #103           │
│            #104           │
│                           │
│       ~99997497 Rows      │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│          ORDER_BY         │
│    ────────────────────   │
│    memory.main.hits_tb    │
│       .EventTime ASC      │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│             #0            │
│__internal_compress_integra│
│     l_utinyint(#1, 0)     │
│             #2            │
│__internal_compress_integra│
│     l_utinyint(#3, 1)     │
│             #4            │
│             #5            │
│             #6            │
│             #7            │
│             #8            │
│             #9            │
│__internal_compress_integra│
│     l_utinyint(#10, 0)    │
│__internal_compress_integra│
│     l_utinyint(#11, 0)    │
│__internal_compress_integra│
│     l_utinyint(#12, 0)    │
│            #13            │
│            #14            │
│            ...            │
│            #90            │
│            #91            │
│            #92            │
│            #93            │
│            #94            │
│            #95            │
│            #96            │
│            #97            │
│            #98            │
│            #99            │
│            #100           │
│__internal_compress_integra│
│    l_utinyint(#101, 0)    │
│            #102           │
│            #103           │
│            #104           │
│                           │
│       ~99997497 Rows      │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│    ────────────────────   │
│          WatchID          │
│         JavaEnable        │
│           Title           │
│         GoodEvent         │
│         EventTime         │
│         EventDate         │
│         CounterID         │
│          ClientIP         │
│          RegionID         │
│           UserID          │
│        CounterClass       │
│             OS            │
│         UserAgent         │
│            URL            │
│          Referer          │
│            ...            │
│      ParamCurrencyID      │
│    OpenstatServiceName    │
│     OpenstatCampaignID    │
│        OpenstatAdID       │
│      OpenstatSourceID     │
│         UTMSource         │
│         UTMMedium         │
│        UTMCampaign        │
│         UTMContent        │
│          UTMTerm          │
│          FromTag          │
│          HasGCLID         │
│        RefererHash        │
│          URLHash          │
│            CLID           │
│                           │
│       ~99997497 Rows      │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         HASH_JOIN         │
│    ────────────────────   │
│      Join Type: SEMI      │
│                           │
│        Conditions:        ├──────────────┐
│       rowid = rowid       │              │
│                           │              │
│       ~99997497 Rows      │              │
└─────────────┬─────────────┘              │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         SEQ_SCAN          ││           TOP_N           │
│    ────────────────────   ││    ────────────────────   │
│       Table: hits_tb      ││          Top: 10          │
│   Type: Sequential Scan   ││                           │
│                           ││         Order By:         │
│        Projections:       ││    memory.main.hits_tb    │
│            URL            ││       .EventTime ASC      │
│          WatchID          ││                           │
│         JavaEnable        ││                           │
│           Title           ││                           │
│         GoodEvent         ││                           │
│         EventTime         ││                           │
│         EventDate         ││                           │
│         CounterID         ││                           │
│          ClientIP         ││                           │
│          RegionID         ││                           │
│           UserID          ││                           │
│        CounterClass       ││                           │
│             OS            ││                           │
│         UserAgent         ││                           │
│            ...            ││                           │
│      ParamCurrencyID      ││                           │
│    OpenstatServiceName    ││                           │
│     OpenstatCampaignID    ││                           │
│        OpenstatAdID       ││                           │
│      OpenstatSourceID     ││                           │
│         UTMSource         ││                           │
│         UTMMedium         ││                           │
│        UTMCampaign        ││                           │
│         UTMContent        ││                           │
│          UTMTerm          ││                           │
│          FromTag          ││                           │
│          HasGCLID         ││                           │
│        RefererHash        ││                           │
│          URLHash          ││                           │
│            CLID           ││                           │
│                           ││                           │
│       ~99997497 Rows      ││                           │
└───────────────────────────┘└─────────────┬─────────────┘
                             ┌─────────────┴─────────────┐
                             │         PROJECTION        │
                             │    ────────────────────   │
                             │             #1            │
                             │             #2            │
                             │                           │
                             │       ~19999499 Rows      │
                             └─────────────┬─────────────┘
                             ┌─────────────┴─────────────┐
                             │           FILTER          │
                             │    ────────────────────   │
                             │  contains(URL, 'google')  │
                             │                           │
                             │       ~19999499 Rows      │
                             └─────────────┬─────────────┘
                             ┌─────────────┴─────────────┐
                             │         SEQ_SCAN          │
                             │    ────────────────────   │
                             │       Table: hits_tb      │
                             │   Type: Sequential Scan   │
                             │                           │
                             │        Projections:       │
                             │            URL            │
                             │         EventTime         │
                             │                           │
                             │          Filters:         │
                             │ optional: Dynamic Filter  │
                             │        (EventTime)        │
                             │                           │
                             │       ~99997497 Rows      │
                             └───────────────────────────┘

This looks cool! Very interested in this.

There's two optimizations here that go together, if you check clickbench results duckdb on their own format is significantly faster than parquet. The two optimizer rules that do this is 1) TopN https://github.com/duckdb/duckdb/blob/main/src/optimizer/topn_optimizer.cpp#L105 2) Late materialization https://github.com/duckdb/duckdb/blob/main/src/optimizer/late_materialization.cpp#L180 (join back the filter result to obtain rest of the columns)

@XiangpengHao

Note that late materialization (the join / semi join rewrite) needs join operator support that DataFusion doesn't yet have (we could add it but it will take non trivial effort)

My suggested order of implementation is:

Enable parquet filter pushdown (filter_pushdown) by default #3463 with @XiangpengHao (so that we can actually evaluate the topk filter during scan)
Then implement topk filtering Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037

I actually think that will likely get us quite fast. I am not sure how much more improvement late materialized joins will get without a specialized file format.

I don't have time to help plan out late materializing joins at the moment, but I am quite interested in pushing along the predicate pushdown

There is a similar thought named prewhere: https://clickhouse.com/docs/sql-reference/statements/select/prewhere.

Even though it aims to filter, the idea is similar, for example:

Table t has 100 columns, one of them is a, for sql: select * from t where a = 1, it'll do the following steps:

First, read only the data in column a
Apply a = 1 filter to filter out matching rows.
Read the remaining 99 columns only for those matching rows.

Back to topk, select * from t order by a limit 10

First, read only the data in column a
Perform a sort to find the row_id of the top 10 rows.
Row identifiers as determined by 2 and selectively read only the other columns of these 10 rows.

We can spilt the idea to the query:

WITH ids AS (SELECT row_id, a FROM t ORDER BY a LIMIT 10)
SELECT t.* FROM t JOIN ids WHERE t.row_id IN (SELECT row_id FROM ids)

@xudong963

We can spilt the idea to the query:

I agree -- this is what I meant by "late materialization" . Your example / explanation is much better than mine @xudong963 🙏

@xudong963

Note that late materialization (the join / semi join rewrite) needs join operator support that DataFusion doesn't yet have (we could add it but it will take non trivial effort)

I did not fully get this part. DF has semi join support and some rewrites to utilize it in similar cases?
The query transformation in SQL as given by @xudong963 is optimized to a SEMI join + TopK, so I think it could be implemented as logical optimization rule (i.e. adding a filter with subquery on the ids).

> CREATE TABLE t (a int, b int, row_id int);
0 row(s) fetched. 
Elapsed 0.004 seconds.

> EXPLAIN (WITH ids AS (SELECT row_id, a FROM t ORDER BY a LIMIT 10)
SELECT t.* FROM t JOIN ids WHERE t.row_id IN (SELECT row_id FROM ids));
+---------------+--------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                       |
+---------------+--------------------------------------------------------------------------------------------+
| logical_plan  | LeftSemi Join: t.row_id = __correlated_sq_1.row_id                                         |
|               |   Cross Join:                                                                              |
|               |     TableScan: t projection=[a, b, row_id]                                                 |
|               |     SubqueryAlias: ids                                                                     |
|               |       Projection:                                                                          |
|               |         Sort: t.a ASC NULLS LAST, fetch=10                                                 |
|               |           TableScan: t projection=[a]                                                      |
|               |   SubqueryAlias: __correlated_sq_1                                                         |
|               |     SubqueryAlias: ids                                                                     |
|               |       Projection: t.row_id                                                                 |
|               |         Sort: t.a ASC NULLS LAST, fetch=10                                                 |
|               |           Projection: t.row_id, t.a                                                        |
|               |             TableScan: t projection=[a, row_id]                                            |
| physical_plan | CoalesceBatchesExec: target_batch_size=8192                                                |
|               |   HashJoinExec: mode=Partitioned, join_type=LeftSemi, on=[(row_id@2, row_id@0)]            |
|               |     CrossJoinExec                                                                          |
|               |       DataSourceExec: partitions=1, partition_sizes=[0]                                    |
|               |       ProjectionExec: expr=[]                                                              |
|               |         SortExec: TopK(fetch=10), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false] |
|               |           DataSourceExec: partitions=1, partition_sizes=[0]                                |
|               |     ProjectionExec: expr=[row_id@0 as row_id]                                              |
|               |       SortExec: TopK(fetch=10), expr=[a@1 ASC NULLS LAST], preserve_partitioning=[false]   |
|               |         DataSourceExec: partitions=1, partition_sizes=[0]                                  |
|               |                                                                                            |
+---------------+--------------------------------------------------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.005 seconds.

>

@xudong963

Ah actually, the query given by @xudong963 is I think slightly off, I think it should be the following (without the explicit join).

This yields the same plan as DuckDB:

> EXPLAIN (WITH ids AS (SELECT row_id, a FROM t ORDER BY a LIMIT 10)
SELECT t.* FROM t WHERE t.row_id IN (SELECT row_id FROM ids));
+---------------+------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                     |
+---------------+------------------------------------------------------------------------------------------+
| logical_plan  | LeftSemi Join: t.row_id = __correlated_sq_1.row_id                                       |
|               |   TableScan: t projection=[a, b, row_id]                                                 |
|               |   SubqueryAlias: __correlated_sq_1                                                       |
|               |     SubqueryAlias: ids                                                                   |
|               |       Projection: t.row_id                                                               |
|               |         Sort: t.a ASC NULLS LAST, fetch=10                                               |
|               |           Projection: t.row_id, t.a                                                      |
|               |             TableScan: t projection=[a, row_id]                                          |
| physical_plan | CoalesceBatchesExec: target_batch_size=8192                                              |
|               |   HashJoinExec: mode=Partitioned, join_type=LeftSemi, on=[(row_id@2, row_id@0)]          |
|               |     DataSourceExec: partitions=1, partition_sizes=[0]                                    |
|               |     ProjectionExec: expr=[row_id@0 as row_id]                                            |
|               |       SortExec: TopK(fetch=10), expr=[a@1 ASC NULLS LAST], preserve_partitioning=[false] |
|               |         DataSourceExec: partitions=1, partition_sizes=[0]                                |
|               |                                                                                          |
+---------------+------------------------------------------------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.004 seconds.

I think the difference is that DuckDB dynamically pushes down the current state of the TopK heap into file opening as described in #15037 and implemented in #15301

@xudong963

I did not fully get this part. DF has semi join support and some rewrites to utilize it in similar cases?

The query transformation in SQL as given by @xudong963 is optimized to a SEMI join + TopK, so I think it could be implemented as logical optimization rule (i.e. adding a filter with subquery on the ids).

@Dandandan --

I think it would be interesting to try and rewrite q23 manually to that pattern and see how it goes fast

I suspect (but have not measured), if we implemented this rewrite we would find it runs much more slowly than the existing code because what would happen is that the entire input file (all columns) would be decoded and all but 10 rows are thrown away

To avoid this we need to push the join filters into the scan (and get predicate pushdown on by default)

Edit: although now I say this maybe it would be much better as we have to decode all the columns now....

I tried the rewrite into a Semi join and indeed it is over 2x slower (5.3sec vs 12sec)

> SELECT * from 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;
Elapsed 5.320 seconds.

Here is what I think the rewrite is

> SELECT * from 'hits_partitioned' WHERE "WatchID" IN (
  SELECT "WatchID" FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10
);

Elapsed 12.023 seconds.

WatchID is a unique key

> select count(distinct "WatchID"), count(*) from 'hits_partitioned';
+------------------------------------------+----------+
| count(DISTINCT hits_partitioned.WatchID) | count(*) |
+------------------------------------------+----------+
| 99997493                                 | 99997497 |
+------------------------------------------+----------+

I also double checked the output

## orig
datafusion-cli -c "SELECT * FROM 'hits_partitioned' WHERE \"URL\" LIKE '%google%' ORDER BY \"EventTime\" LIMIT 10;" > orig.out

## rewrite
 datafusion-cli -c "SELECT * from 'hits_partitioned' WHERE \"WatchID\" IN (SELECT \"WatchID\" FROM 'hits_partitioned' WHERE \"URL\" LIKE '%google%' ORDER BY \"EventTime\" LIMIT 10);" > rewrite.out

## check
sort orig.out > orig.out.sort
sort rewrite.out > rewrite.out.sort
diff orig.out.sort rewrite.out.sort

7c7
< Elapsed 5.649 seconds.
---
> Elapsed 11.067 seconds.

I am not really sure where the time is going 🤔
output of explain analyze: explain.txt

@alamb

Thanks for checking @alamb !

I think a large portion is spent in the hash join (repartitioning the right side input) - I think because it runs as Partitioned hash join, instead of realizing it could use CollectLeft (see repartition_time=55.648133838s) for the small side (10 rows).

Thanks for checking @alamb !

I think a large portion is spent in the hash join (repartitioning the right side input) - I think because it runs as Partitioned hash join, instead of realizing it could use CollectLeft (see repartition_time=55.648133838s) for the small side (10 rows).

I also think the Url, WatchID and EventDate columns will be decoded twice. To make that join go really fast we would need something like:

Push Dynamic Join Predicates into Scan ("Sideways Information Passing", etc) #7955

@adriangb

BTW combined with @adriangb's PR here

Add dynamic pruning filters from TopK state #15301

It will likely go crazy fast 🚀

I think a large portion is spent in the hash join (repartitioning the right side input) - I think because it runs as Partitioned hash join, instead of realizing it could use CollectLeft (see repartition_time=55.648133838s) for the small side (10 rows).

I traced this down to an issue in the planner, which uses PartitionMode::Auto iff stats are collected (datafusion.execution.collect_statistics)
We can however still use file and plan-derived statistics, so let's change that.
#15339

@alamb

I tried the rewrite into a Semi join and indeed it is over 2x slower (5.3sec vs 12sec)

SELECT * from 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;
Elapsed 5.320 seconds.
Here is what I think the rewrite is

SELECT * from 'hits_partitioned' WHERE "WatchID" IN (
SELECT "WatchID" FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10
);

Elapsed 12.023 seconds.
WatchID is a unique key

select count(distinct "WatchID"), count() from 'hits_partitioned';
+------------------------------------------+----------+
| count(DISTINCT hits_partitioned.WatchID) | count() |
+------------------------------------------+----------+
| 99997493 | 99997497 |
+------------------------------------------+----------+
I also double checked the output

orig

datafusion-cli -c "SELECT * FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;" > orig.out

rewrite

datafusion-cli -c "SELECT * from 'hits_partitioned' WHERE "WatchID" IN (SELECT "WatchID" FROM 'hits_partitioned' WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10);" > rewrite.out

check

sort orig.out > orig.out.sort
sort rewrite.out > rewrite.out.sort
diff orig.out.sort rewrite.out.sort

7c7
< Elapsed 5.649 seconds.

Elapsed 11.067 seconds.

I recently took a detailed look at this optimization in ClickHouse, and it might offer you some insights @alamb .

rewrite SQL in ClickHouse

First, in ClickHouse, each row of data can be located using the two virtual columns _part and _part_offset (this applies only to MergeTree tables).
Specifically, for the optimization you mentioned, you can compare the performance of the following two queries in ClickHouse:

-- Q1:
SELECT * from hits WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;
10 rows in set. Elapsed: 34.907 sec. Processed 18.63 million rows, 11.77 GB (533.61 thousand rows/s., 337.25 MB/s.)
Peak memory usage: 1.31 GiB.
-- Q2:
SELECT * FROM hits WHERE (_part,_part_offset) in 
(SELECT _part,_part_offset from hits WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10);
10 rows in set. Elapsed: 7.262 sec. Processed 18.68 million rows, 3.13 GB (2.57 million rows/s., 431.28 MB/s.)
Peak memory usage: 190.45 MiB.

I measured that Q1 took 34 seconds, while Q2 only took 7.2 seconds (both cleared the page cache before running).
This performance improvement may be attributed to ClickHouse correctly utilizing the result of the subquery to filter parts and quickly pinpoint the relevant data.

However, in earlier versions of ClickHouse (such as 23.12), the aforementioned query Q2 would actually degrade in performance. But at that time, if I split Q2 into two separate statements and executed them manually, it would still work perfectly fine.

-- Initial (Clear page cache)
    -- step 1
SELECT _part,_part_offset  FROM hits WHERE "URL" LIKE '%google%' ORDER BY "EventTime" LIMIT 10;
10 rows in set. Elapsed: 6.254 sec. Processed 18.63 million rows, 3.10 GB (2.98 million rows/s., 495.76 MB/s.)
Peak memory usage: 190.79 MiB.
    -- step 2
SELECT * FROM hits WHERE
    (_part = 'all_1_210_3' AND _part_offset IN (20223140, 20223142, 20223144, 19725555, 15188338, 13322137, 19741966, 3076201))
    OR
    (_part = 'all_211_216_1' AND _part_offset IN (692957, 692958));
10 rows in set. Elapsed: 0.731 sec. Processed 65.04 thousand rows, 36.83 MB (89.02 thousand rows/s., 50.41 MB/s.)
Peak memory usage: 46.23 MiB.

-- The total time is 6.98 seconds.

Conclusion

Accessing row_id might require some optimization techniques, such as merging row_id conditions to quickly skip through RowGroups
After implementing this optimization, the performance improvement in ClickBench may not be significant. I noticed that when columns are fully loaded into the page cache, the performance gain from lazy materialization is not very noticeable. Additionally, ClickBench only clears the page cache before executing the first SQL query. Detailed experiments show that even when fully loaded into the page cache, this optimization still has a significant effect. However, Q23 is not only affected by this optimization but also influenced by prewhere. Therefore, after being fully cached in memory, prewhere still significantly reduces the amount of data read, which in turn makes the effect of this optimization less noticeable.

Relevant: https://clickhouse.com/blog/clickhouse-gets-lazier-and-faster-introducing-lazy-materialization

Relevant: https://clickhouse.com/blog/clickhouse-gets-lazier-and-faster-introducing-lazy-materialization

Thank you so much for sharing this blog link—it’s truly an excellent learning resource! I hadn’t come across it before; I was just looking at the source code of this related optimization PR and running performance tests on the hits dataset. They introduced a new type called ColumnLazy(Interestingly, the comments also mention optimizing the join process through this structure.) to implement lazy materialization of columns. However, in my testing, I’ve noticed that performance starts to degrade when the limit value becomes large. For instance, degradation begins at limit 100000 when the page cache is cleared. In fully cached scenarios, performance degradation occurs as early as limit 256. This is why they’ve set query_plan_max_limit_for_lazy_materialization=10 as the default value to determine whether to apply the optimization.

However, I noticed that using the following SQL rewrite approach hardly leads to any performance degradation. I even tested it with limit 400000 while clearing the page cache, and it still maintained excellent performance. Below are the performance test results for a limit of 100,000 (clear page cache):

-- Q1:
SELECT * from hits ORDER BY "EventTime" LIMIT 100000;
100000 rows in set. Elapsed: 70.314 sec. Processed 103.89 million rows, 70.28 GB (1.48 million rows/s., 999.50 MB/s.)
Peak memory usage: 27.66 GiB.
-- Q2:
SELECT * FROM hits WHERE (_part,_part_offset) in 
(SELECT _part,_part_offset from hits ORDER BY "EventTime" LIMIT 100000);
100000 rows in set. Elapsed: 5.639 sec. Processed 82.02 million rows, 3.39 GB (14.55 million rows/s., 601.81 MB/s.)
Peak memory usage: 715.57 MiB.

The direct SQL rewrite approach mentioned above might still have some issues in ClickHouse at the moment. They are currently exploring the possibility of using it alongside projections (a feature in ClickHouse akin to materialized views) to create secondary indexes and similar functionalities. Relevant PR: ClickHouse/ClickHouse#78429 (comment).

I find this truly remarkable, and I’m also investigating why the direct SQL rewrite performs faster than ColumnLazy.

I find this truly remarkable, and I’m also investigating why the direct SQL rewrite performs faster than ColumnLazy.

I have already discovered the reason:ClickHouse/ClickHouse#79645

They are currently exploring the possibility of using it alongside projections (a feature in ClickHouse akin to materialized views) to create secondary indexes and similar functionalities. Relevant PR: ClickHouse/ClickHouse#78429 (comment).

There's a follow-up PR that makes the query in that comment work consistently: ClickHouse/ClickHouse#79471

Also, I had asked for something that sounds equivalent to dynamic filter in TopK in ClickHouse in ClickHouse/ClickHouse#75098 and ClickHouse/ClickHouse#75774 (comment), so it seems like CH has yet to perform that optimization at the moment, it just has lazy materialization and soon the manual version of it via query rewriting and "projection indexes".

@alamb

@alamb should we close this now that Q23 is 23x faster?

YES!

Navigation Menu

Repository navigation

Make ClickBench Q23 Go Faster #15177

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

orig

rewrite

check

7c7
< Elapsed 5.649 seconds.

rewrite SQL in ClickHouse

Conclusion

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make ClickBench Q23 Go Faster #15177

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

alamb commented on Mar 12, 2025

alamb commented on Mar 12, 2025

alamb commented on Mar 12, 2025

suibianwanwank commented on Mar 12, 2025

robert3005 commented on Mar 12, 2025

alamb commented on Mar 12, 2025

xudong963 commented on Mar 12, 2025

alamb commented on Mar 12, 2025

Dandandan commented on Mar 20, 2025

Dandandan commented on Mar 20, 2025

adriangb commented on Mar 20, 2025

alamb commented on Mar 20, 2025

alamb commented on Mar 20, 2025

alamb commented on Mar 20, 2025

Dandandan commented on Mar 20, 2025

alamb commented on Mar 20, 2025

alamb commented on Mar 20, 2025

Dandandan commented on Mar 20, 2025

acking-you commented on Apr 23, 2025

orig

rewrite

check

7c7 < Elapsed 5.649 seconds.

rewrite SQL in ClickHouse

Conclusion

adriangb commented on Apr 23, 2025

acking-you commented on Apr 23, 2025

acking-you commented on Apr 27, 2025

EmeraldShift commented on Apr 29, 2025

adriangb commented on Oct 15, 2025

alamb commented on Oct 15, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

7c7
< Elapsed 5.649 seconds.