Skip to content

Track total memory allocation used by DataFusion plans #898

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

When running DataFusion as part of a Rust program that has other substantial uses of memory (for example Buffers in IOx) we would like to know how much memory is allocated when running DataFusion plans so we can:

  1. Allocate sufficient Memory to DataFusion / limit other users of memory
  2. Start turning / working to limit memory usage by DataFusion (e.g. [Epic] Optionally Limit memory used by DataFusion plan #587 )

Describe the solution you'd like
A counter (perhaps AtomicUsize tied to the ExecutionContext somehow) that tracks, across all DataFusion plans running in that context, how much memory has been allocated. This counter's value should be available both during the plan execution as well as after it has completed.

The counter should include:

  1. Memory allocated in RecordBatches created by DataFusion operators
  2. Memory used in intermediate buffers (e.g. HashTables, Sort buffers, etc) - should be "capacity" rather than "size" to reflect the heap usage of the program
  3. Decremented when memory is deallocated

Initially, a counter that gets the major allocations of memory would be good.

Describe alternatives you've considered
Implement a per-operator allocation tracking scheme (perhaps based on metrics, see #866 and #679).

Per-operator tracking of memory is likely also valuable and is tracked by #899

Additional context
This is likely a pre-requisite for actually limiting memory usage for DataFusion plans as described in #587

Activity

  1. alamb commented on Aug 4, 2023

    @alamb
    ContributorAuthor

    I am happy to report this is now possible by using a memory pool (simply ask for how much space it has left)

    https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#tymethod.reserved

    You can use https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.GreedyMemoryPool.html if you only want to track the memory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions