Repository navigation
Add a default ListFilesCache implementation for the ListingTable #18827
Description
Is your feature request related to a problem or challenge?
When using the ListingTable the objects backing the table are listed on each query. This inherently adds additional latency to every query, and this latency is especially notable when using high-latency storage such as remote Object Stores. DataFusion's cache infrastructure for ListingTables has a hook for a ListFilesCache that users can implement, however there is no default implementation. I would like to improve out-of-the-box query performance by providing a default implementation for the ListFilesCache.
Describe the solution you'd like
I would like a default implementation of the ListFilesCache that mirrors the functionality of the recently developed MetadataCache. This implementation should have a user configurable memory bound and users who do not want to use the cache should be able to disable it.
Describe alternatives you've considered
No response
Additional context
No response
Activity
alchemist51 commentedon Nov 20, 2025
@alamb @BlakeOrth if somebody has not already started, I would like to take this issue!
alamb commentedon Nov 20, 2025
I do not know of any work that has started but I think @BlakeOrth maybe was interested in it
BlakeOrth commentedon Nov 20, 2025
@alchemist51 I'm glad others are interested in this effort! I already have a decent chunk of a POC for the cache itself that I put together a little while back when I was exploring if this effort even made sense. That being said, I'm more than willing to try to collaborate with other contributors on this effort as a whole where it makes sense.
I think it might make sense for me to get my existing POC to a state where I can open a draft PR so everyone can understand the general path and how large of a change set this issue might take. After we understand that, it might make sense to split that POC into smaller pieces that are "real" PRs (with docs, tests etc.), and if so that could be a good opportunity for us to try to divide up this work. If it doesn't end up making sense to split up this work there will be additional work immediately following this effort to allow configuration and introspection of the cache, both of which will be necessary to really close out this work. Does this make sense and sound ok to everyone?
alchemist51 commentedon Nov 20, 2025
Sure sounds good! Let me know once you raise the draft.
alamb commentedon Dec 3, 2025
In order to add a default cache we will also need to add a way to limit its size:
Adds memory-bound DefaultListFilesCache (#18855)
alamb commentedon Dec 8, 2025
@BlakeOrth added an implementation here
What remains for this ticket is to set the DefaultListFilesCache in the default session context and update tests
BlakeOrth commentedon Dec 8, 2025
I think the work to set the
DefaultListFilesCachein the default session also needs to come with the capability to disable the cache (this will be very important for thesqllogictests that need updated)alamb commentedon Dec 8, 2025
I think this is tracked in
BlakeOrth commentedon Dec 15, 2025
@alamb with #19108 being merge-ready I will start working on a PR that enables this cache by default (and all the test fallout this is going to bring) so we can close this one out!
Enables DefaultListFilesCache by default (#19366)