Repository navigation
1000x slowdown opening parquet file due to partitions #16676
Description
Describe the bug
parquet metadata load time was 15 seconds with default configuration. Setting target_partitions to 1 brought it down to 15 milliseconds.
With the default config I was seeing 20 file groups in the DataSourceExec (since my machine has 20 cores) and loading the metadata was taking forever. Forcing target_partitions to 1 fixed it: I now see just 1 file group in the DataSourceExec, metadata load time is down. (Added bonus: the plan no longer requires a MergeExec, which was the slowest part of the whole query).
To Reproduce
The query I was running is very basic: just filter + sort + select. The parquet file is 130.00 MiB
with 1.08 MiB of metadata (6.42 MiB when expanded in memory).
Expected behavior
I wouldn't have thought going from 1 file group to 20 would slow down metadata parsing by 1000x.
Additional context
No response
Activity
jatin510 commentedon Jul 5, 2025
take
asayers commentedon Jul 6, 2025
I think the correct solution was for me to enable the metadata cache (I haven't confirmed this). So perhaps the "bug" (if there is one) is just that the metadata cache is off by default?
alamb commentedon Jul 7, 2025
Perhaps this is related to the default value of statistics too:
#16447
Do you have a reproducer you can share @asayers ?
asayers commentedon Jul 8, 2025
I can't share the data I was hitting this case with. I could try to make a synthetic reproducer, but work is very busy right now so I might not get to it for a few months.
alamb commentedon Aug 14, 2025
We have added a new metadata cache (will ship in 50.0.0):
I wonder if you could try your reproducer with what is on main and see if the query runs faster
alamb commentedon Nov 26, 2025
I think this issue should now be closed. Please reopen if it you don't see any improvement