Repository navigation
Reuse file descriptors for LocalFileSystem object storage #19983
Description
Is your feature request related to a problem or challenge?
The ObjectStore implementation is stateless and doesn't allow for passing File descriptors.
This means for every request it's doing a File::open (and close) call, which creates overhead (mainly when doing a lot of small requests, e.g. with predicate pushdown enabled).
Describe the solution you'd like
Implement an object store implementation for DataFusion that opens / reuses the Files for each path.
We can use Arc<File> stored in hashmap for each file / path + read_at (stateless file read) to minimize overhead and maximize concurrency.
Describe alternatives you've considered
No response
Additional context
Also see
apache/arrow-rs-object-store#622
apache/arrow-rs-object-store#614
Reacted by Adam Gutglick
Reacted by Kyle Barron
Activity
[-]Save file descriptors for LocalFileSystem object storage[/-][+]Reuse file descriptors for LocalFileSystem object storage[/+]AdamGS commentedon Jan 27, 2026
glad to take this if you didn't get started
Dandandan commentedon Jan 27, 2026
Sure! I did vibe-code something simple to test the performance - on the Clickbench benchmark couldn't see a dramatic performance, but I didn't yet check other areas like filter pushdown where I guess it could matter more (i.e. many small requests).
I do sometimes see
openorlstator (metadata) or seek in CPU profiles, most of the time it's only taking ~0.1% / 1 sample of total time, the most time is always in read and/or allocating theVecto write to.AdamGS commentedon Jan 28, 2026
I do remember it being an issue in the past with Vortex, so I'm hoping I can see a difference
alchemist51 commentedon Jan 29, 2026
We were facing file discriptor opening taking time when we were trying to open the file multiple times for smaller partitions. Using a Cached file reader did help us get over this bottleneck. Tagging @bharath-techie who was doing the experimentation on this!
AdamGS commentedon Jan 29, 2026
@Dandandan Is there a reason to have it as part of DataFusion and not part of
object-store? As I'm working through apache/arrow-rs-object-store#628 it seems like there is already a lot of useful code there, and making the change there makes its impact spread wider.I get that the implementation is stateless, but they might be open to changing that?
Dandandan commentedon Jan 29, 2026
Ok! No if object store is willing to take this it might be better (and help other use cases as well potentially)!
I created the issue here as it's easier to experiment with DataFusion without changing the upstream code / interface.