Repository navigation
Parallelize infer_schema #19970
Description
Is your feature request related to a problem or challenge?
Currently infer_schema runs in a single thread.
Also it uses buffered instead of buffer_unordered (the latter will be better if some reads are taking much longer than others).
You can see below it takes ~130ms just to load the schema/metadata for the (100) clickbench_partitioned files.
Describe the solution you'd like
We should be able to use all threads.
Describe alternatives you've considered
No response
Additional context
No response
Activity
alamb commentedon Jan 25, 2026
FWIW the actual clickbench runs (on benchmark.clickhouse.com) don't count the time to
CREATE EXTERNAL TABLESo if the schema inference isn't part of of the actual query, this optimization may not help the benchmark results
It is not a bad thing to do, I am just commenting regarding the benchmark
Dandandan commentedon Jan 25, 2026
Ah yes, you are right.
Looking at the benchmark, the difference between cold and warm run is quite high (notacibly higher than some others engines) - I believe
list_files_for_scan#19971 is part of the cold run?https://benchmark.clickhouse.com/#system=+ti%20rud&type=-&machine=+g4e&cluster_size=-&opensource=-&hardware=-&tuned=+n&metric=cold&queries=-
zhuqi-lucas commentedon Mar 8, 2026
Hi @Dandandan @alamb, is this still being actively pursued?
We have a use case that would benefit from this — inferring schemas from complex nested JSON files can be quite slow today, especially when there are many files or deeply nested structures. Faster schema inference would meaningfully improve our workflow.
Happy to contribute or help test if this is still moving forward!
alamb commentedon Mar 10, 2026
Dandandan commentedon Mar 16, 2026
Go ahead, I think it's a nice issue. Did you also see (perhaps you can help reviewing?):
apache/arrow-rs#9494
One other thing I saw that also might be worth looking at is that we always create > 32 threads when doing metadata reading based on the configuration. Not a huge problem per-se but it adds to the memory usage (as the threads won't be removed) and probably reduces locality a bit while running queries.