Repository navigation
Number tile on a gauge changes with the time range: sum adds up every sample in short ranges #3500
Description
What happened?
A number tile over the same gauge data gives a different value depending on how wide the time range is.
Data: two gauge series, one point a minute for 10 minutes. Their latest values are 80 and 4. Number tile, aggFn: sum, on main (2bfc630):
| Time range (contains the same 10 minutes of data) | Result |
|---|---|
| exactly the 10 minutes | 515.75: every sample summed |
| 6 hours | 84: the latest value per series |
| 7 days | 84 |
A number tile has no granularity, so the gauge CTE buckets the data at 'auto' granularity, which is derived from the date range. It takes LastValue per series per bucket, and the outer aggregation then runs over all the buckets. A short range gives small buckets, and in the end one bucket per sample:
- https://github.com/hyperdxio/hyperdx/blob/2bfc6307dc0c9b84d0f3ff68e1beeaf0b4fcc877/packages/common-utils/src/core/renderChartConfig.ts#L2053-L2066
- https://github.com/hyperdxio/hyperdx/blob/2bfc6307dc0c9b84d0f3ff68e1beeaf0b4fcc877/packages/common-utils/src/core/renderChartConfig.ts#L2134-L2141
Expected: the same data gives the same number for any range that contains it. For a gauge, that would be the last value of each series in the range, then the aggregation across series (84 here). That matches a PromQL instant query, and the last point of the same chart as a line.
We only tested sum. avg, min and max go through the same CTE, so we expect them to change with the range too, just less visibly.
Steps to reproduce
Reproduces on play.hyperdx.io? Didn't try.
An integration test in the style of renderChartConfig.int.test.ts, on main:
- Insert gauge
repro.cpufor hostsa(50, 25, 12.5, 6.25, 100, 75, 10, 80, 40, 80) andb(1, 2, 3, 4, 5, 6, 5, 4, 3, 4), one point a minute. renderChartConfigwithdisplayType: Number,select: [{ aggFn: 'sum', metricType: 'gauge', metricName: 'repro.cpu', valueExpression: 'Value' }], no granularity.dateRange= the 10 minutes → 515.75.dateRange= 6 h ending at the same point → 84.
How are you running HyperDX?
Helm (ClickStack chart 2.1.0) with the hyperdx:2-nightly image (2.42.0 plus the Node bump from #3456), our own ClickHouse 26.2.5.45. Reproduced on main 2bfc630.
Where does it show up?
Dashboard number tiles. We hit it while rebuilding dashboards through the MCP tools: a "connected clients" tile (sum of a per-pod Redis gauge) showed a multiple of the real count, so we rebuilt it in SQL.
Debug info
N/A.
Logs
None.
Written with help from an AI assistant (Claude Code). The behaviour comes from our own deployment and an integration test on main.
Activity
bcanvural commentedon Oct 31, 2026
We'd like to fix this. One refinement first, because "last value per series in the range" (as I wrote above) isn't right either once series come and go.
Example: 3 pods with 10 Redis clients each are replaced by 3 new ones halfway through a 10-minute range.
Proposal: on a non-time-series tile (number, and tables without granularity), a gauge select works like a PromQL instant query. The
aggFncombines series at the end of the range, assum(metric)/avg(metric)/max(metric)do in PromQL:AttributesHash), take its newest point in a lookback window before the end of the range: 5 minutes, Prometheus' default lookback delta.bitAnd(Flags, 1) = 1), so a vanished series drops out immediately, not after 5 minutes.aggFnacross the remaining series.The result no longer depends on the range width: the example in the issue gives 84 for every range, and the churn example gives 30. Line charts are unchanged. Aggregating over time (like PromQL's
avg_over_time) would be a separate, explicit option; this PR doesn't add one.Related, but a separate fix: today the gauge query reads staleness markers as real values of 0 (
Flagsis projected but not filtered), which pullsavgandmindown in the bucket where a series ends. We can file that on its own.Plan for the PR:
renderChartConfig, when the chart has no granularity: replace the auto-bucketedBucketedCTE with one row per series,argMax(Value, TimeUnix)andargMax(Flags, TimeUnix)over[end - 5m, end], drop markers, and keep the outer aggregation as it is.renderChartConfig.int.test.ts): the same data gives the same number for a 10-minute, 6-hour and 7-day range; the churn example gives the current count; a series ending in a marker is excluded.Does that work for you?
fix(common-utils): skip metric points that have no recorded value (#3583