Repository navigation
Add Memory Profiling Functionality #14510
Description
Is your feature request related to a problem or challenge?
When executing long-running batch processes or streaming queries, it is difficult to diagnose and monitor memory usage. People need an easy way to inspect memory consumption to understand resource usage and troubleshoot potential OOM. issues.
Describe the solution you'd like
Implement a built-in memory profiling feature
(similar to DuckDB’s duckdb_memory() and duckdb_temporary_files() functions)
┌──────────────────┬────────────────────┬─────────────────────────┐
│ tag │ memory_usage_bytes │ temporary_storage_bytes │
│ varchar │ int64 │ int64 │
├──────────────────┼────────────────────┼─────────────────────────┤
│ BASE_TABLE │ 168558592 │ 0 │
│ HASH_TABLE │ 0 │ 0 │
│ PARQUET_READER │ 0 │ 0 │
│ CSV_READER │ 0 │ 0 │
│ ORDER_BY │ 0 │ 0 │
│ ART_INDEX │ 0 │ 0 │
│ COLUMN_DATA │ 0 │ 0 │
│ METADATA │ 0 │ 0 │
│ OVERFLOW_STRINGS │ 0 │ 0 │
│ IN_MEMORY_TABLE │ 0 │ 0 │
│ ALLOCATOR │ 0 │ 0 │
│ EXTENSION │ 0 │ 0 │
├──────────────────┴────────────────────┴─────────────────────────┤
│ 12 rows 3 columns │
└─────────────────────────────────────────────────────────────────┘
FROM duckdb_temporary_files();
┌────────────────────────────────┬───────────┐
│ path │ size │
│ varchar │ int64 │
├────────────────────────────────┼───────────┤
│ .tmp/duckdb_temp_storage-0.tmp │ 967049216 │
└────────────────────────────────┴───────────┘
that allows to query:
- The memory usage of different streams or even other objects (e.g., hash tables, intermediate results, buffers).
- Current temporary file usage, if applicable.
Describe alternatives you've considered
No response
Additional context
No response
Reacted by Yongting You
Activity
comphead commentedon Feb 5, 2025
That is nice @berkaysynnada I love it.
To implement it we have to use a
memory reservationin most of intensive operations, like Sorting, Join, hash, scan, etc, now we have quite few operations supporting memory reservation, the spill support even fewer.To break down this into manageable pieces we need to
ExecutionPlanand we can access metrics throughEXPLAIN ANALYZEso probably this can be a substitute for the builtin function? Should we output per partition or aggregated, max value or mean?berkaysynnada commentedon Feb 6, 2025
Thank you @comphead for the great elaboration! I don't have a clear answer to your question yet, but since this is the final part of our work list, I believe someone can complete the first two items before deciding on how to collect and display.
PokIsemaine commentedon Feb 25, 2025
I am interested in this function.
Tag memory that is allocated through the buffer manager, and add duckdb_memory() function by Mytherin · Pull Request #10496 · duckdb/duckdb
DuckDBadds memory tags when the buffer manager allocates memory, making it easier to trace memory usage.Does
DataFusionalso need to implement it in a similar way?I have just started to learn
DataFusion’s memory management and I noticed:https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html
Do we need to trace all memory allocations or just focus on the part managed by
MemoryPool?comphead commentedon Feb 27, 2025
Hi @PokIsemaine I'm planning to experiment with this.
One thing as you mentioned absolutely correct is to use MemoryReservation which is the helper when memory allocated through the pool and can be read in metrics using
EXPLAIN ANALYZE querystatement. Challenge is our memory pool coverage needs to be improved and get a lot of transformation covered.Another dirty trick which might work is to try to track the max process memory per the each transformation using https://crates.io/crates/sysinfo
This approach shows the process memory usage globally not just per specific node of physical plan, but using the
Having max memory fingerprint it would be easy to find trends and heavyweight operations
2010YOUY01 commentedon Mar 6, 2025
No, they're different. I believe duckdb's memory pool is a textbook buffer pool, which manages memory spilling and reading back automatically for the operators.
For DataFusion,
MemoryPoolperhaps is a misleading naming, it's actually a 'memory tracker', operators have to use this tracker to estimate if it has exceeded the memory limit. If so they will explicitly do the spilling, or fail with a user friendly error message.Regarding tracking those small allocations by internal
MemoryPool, maybe not. However, if we can use the system memory profiler, or internal metrics from a memory allocator (e.g. mimalloc), it would be great to also integrate them.comphead commentedon Mar 18, 2025
The PR in arrow rs to avoid mem overcount for shared buffers apache/arrow-rs#7303
alamb commentedon Aug 12, 2025
This was merged and is now available in DataFusion
alamb commentedon Aug 12, 2025
Also it seems there is a related ticket
@kosiew has a PR / proposal
zheniasigayev commentedon Aug 12, 2025
Very exciting progress! One thing that would be incredibly helpful is to extend the configuration flag,
--top-memory-consumersfrom the DataFusion CLI into the configs (https://datafusion.apache.org/user-guide/configs.html) to get informative error messages. This would display N number of top memory consumers when the query fails due to memory exhaustion. By default it can be set to 0 to disable this.I think a big leap forward would be to implement continuous profiling of DataFusion, since it's been quite difficult to do so with https://datafusion.apache.org/library-user-guide/profiling.html, unless more capabilities have been added.
Thinking even farther out, some nice-to-haves would be equivalent tools such as https://github.com/DBatUTuebingen/duckdb-explain-visualizer.
alamb commentedon Aug 18, 2025
I think that is what already happens. If not, can you show an example of what message you are proposing to change?
I agree this would be great.
DataFusion already has all the APIs needed to implement this type of tool (including getting the current values of all statisics). All that is needed now is for someone to actually implement the UI / modify an existing one