Repository navigation
Prometheus metric for memory/time used per task prefix #7341
Description
We have some metrics around number of tasks completed over time. I think that we could use a couple of others:
- Amount of time spent in each prefix. I think that we already store this data in all_durations (I brought this up when @ntabris did his work before as probably what we actually wanted, rather than number of tasks completed).
- Amount of memory currently used by each prefix. This is probably not a counter, but instead an instantaneous measurement of current state
This would be useful for after-the-fact debugging of clusters. @dchudz was curious about this. @fjetter is this easy for someone on your team to do?
Activity
mrocklin commentedon Nov 21, 2022
Currently queued/processing per prefix might also be interesting.
ntabris commentedon Nov 21, 2022
For (1) if we can get that as a cumulative total, that would be great.
For the memory gauge, what's the time resolution at which that's useful data?
mrocklin commentedon Nov 22, 2022
I think that this is already stored in
all_durationson the TaskGroup/Prefix objectsI think instantaneous is fine. For example when looking at a cluster that appears to be globally paused (what we saw earlier) I'm curious, which task prefixes are responsible for all of this data? Which task prefixes have yet to run?
dchudz commentedon Dec 5, 2022
@fjetter any sense of where this fits into your next few days/weeks/months?
add prometheus metrix for time used per task prefix
add prometheus metric for time used per task prefix
add prometheus metric for time used per task prefix
crusaderky commentedon Dec 22, 2022
Do we already have number of completed tasks per prefix? We need it in order to calculate mean task runtime grouped by prefix
mrocklin commentedon Dec 22, 2022
Yes
We actually already record an exponentially weighted moving average of task runtime by prefix. This was actually the very first motivation for task prefixes. It's pretty core to the scheduling heuristics. This functionality actually pre-dates the TaskPrefix class itself 🙂
crusaderky commentedon Dec 22, 2022
@mrocklin is your answer specific to Prometheus, or are you talking about internal metrics in the scheduler?
mrocklin commentedon Dec 22, 2022
Internal metrics on the scheduler.
crusaderky commentedon Dec 22, 2022
We need to export the count of completed tasks per prefix to Prometheus, so that grafana/whatever can calculate mean runtimes.
add prometheus metric for time used per task prefix
add prometheus metric for time used per task prefix
add prometheus metric for time and memory used per task prefix
add prometheus metric for time and memory used per task prefix
add prometheus metric for time and memory used per task prefix