You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Track total memory allocation used by DataFusion plans #898
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
When running DataFusion as part of a Rust program that has other substantial uses of memory (for example Buffers in IOx) we would like to know how much memory is allocated when running DataFusion plans so we can:
Allocate sufficient Memory to DataFusion / limit other users of memory
Describe the solution you'd like
A counter (perhaps AtomicUsize tied to the ExecutionContext somehow) that tracks, across all DataFusion plans running in that context, how much memory has been allocated. This counter's value should be available both during the plan execution as well as after it has completed.
The counter should include:
Memory allocated in RecordBatches created by DataFusion operators
Memory used in intermediate buffers (e.g. HashTables, Sort buffers, etc) - should be "capacity" rather than "size" to reflect the heap usage of the program
Decremented when memory is deallocated
Initially, a counter that gets the major allocations of memory would be good.
Describe alternatives you've considered
Implement a per-operator allocation tracking scheme (perhaps based on metrics, see #866 and #679).
Per-operator tracking of memory is likely also valuable and is tracked by #899
Additional context
This is likely a pre-requisite for actually limiting memory usage for DataFusion plans as described in #587
Reacted by Raphael Taylor-Davies, Ruihang Xia and Nga Tran
Activity
alamb commentedon Aug 4, 2023
I am happy to report this is now possible by using a memory pool (simply ask for how much space it has left)
https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#tymethod.reserved
You can use https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.GreedyMemoryPool.html if you only want to track the memory