Repository navigation
Support compression in spill files #16130
Description
Is your feature request related to a problem or challenge?
Part of #16065
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
Describe the solution you'd like
- Add a configuration for spill compression type
- Pass an option to Arrow-IPC writer/reader for compression
Describe alternatives you've considered
No response
Additional context
No response
Reacted by ding-young
Activity
2010YOUY01 commentedon May 21, 2025
I think this issue can be added to your current TODO list @ding-young.
In the future, if we introduce a dedicated spill format, we might revise the implementation to apply compression to the entire spill file.
But I think it's good to be implemented now since the code footprint is small (should be just passing an option into Arrow IPC reader/writer)
ding-young commentedon May 21, 2025
Sure. I also think that introducing compression option first is good to start though we may try more complicated techniques.
ding-young commentedon May 21, 2025
take
Omega359 commentedon May 27, 2025
related/prior art: #14078 (comment)