Repository navigation
OpenMetrics writer exposes _count/_sum for classic histograms with negative bucket thresholds #2631
Description
Location (prometheus/client_java @ 7695470):
prometheus-metrics-exposition-textformats/.../OpenMetricsTextFormatWriter.java:227-230:writeClassicHistogramBucketswrites_count/_sumwheneverhasCount() && hasSum(), for histograms and gauge histograms alike.prometheus-metrics-model/.../DistributionDataPointSnapshot.java:33-36: the model notes the rule but leaves it to someone else.prometheus-metrics-core/.../Histogram.java:793-800:Histogram.Builder.classicUpperBounds(...)accepts negative bounds, so a regularHistogramcan produce such data points.
// In OpenMetrics format, histogram _count and _sum are either both present or both absent.
if (data.hasCount() && data.hasSum()) {
writeCountAndSum(writer, countName, sumName, data, exemplars, scheme, nameValidLegacy);
} private void validate() {
// If a histogram or summary observes negative values the sum could be negative.
// According to OpenMetrics sum should be omitted in that case, but we don't enforce this here.
}Problem
The OpenMetrics 1.0 spec says for Histogram:
Negative threshold buckets MAY be used, but then the Histogram MetricPoint MUST NOT contain a sum value as it would no longer be a counter semantically.
and for the text format:
If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix "_count".
OpenMetricsTextFormatWriter does not check the bucket thresholds, so a classic histogram whose first bucket is negative (e.g. Histogram.builder().classicUpperBounds(-1, -0.5, 0, 0.5, 1)) is exposed with _sum (and _count), which is not valid OpenMetrics. The writer's own comment already says _count and _sum come as a pair, so both should be left out in that case; the buckets still carry the count via le="+Inf".
Scope of this report:
- Only the OpenMetrics 1.0 text writer. The Prometheus text format, protobuf and the experimental OpenMetrics 2.0 writer are not affected by this rule.
- Gauge histograms are not affected: the spec says "If negative threshold buckets are present, then sum MAY be negative" for GaugeHistogram, so
_gcount/_gsumshould stay. - I have not checked whether a particular ingester rejects the current output.
Reproduction
NegativeBucketHistogramTest.java (prometheus-metrics-exposition-textformats)
package io.prometheus.metrics.expositionformats;
import static org.assertj.core.api.Assertions.assertThat;
import io.prometheus.metrics.model.snapshots.ClassicHistogramBuckets;
import io.prometheus.metrics.model.snapshots.HistogramSnapshot;
import io.prometheus.metrics.model.snapshots.MetricSnapshots;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.Test;
class NegativeBucketHistogramTest {
@Test
void histogramWithNegativeBucketsOmitsCountAndSum() throws IOException {
HistogramSnapshot histogram =
HistogramSnapshot.builder()
.name("my_histogram")
.dataPoint(
HistogramSnapshot.HistogramDataPointSnapshot.builder()
.sum(-0.5)
.classicHistogramBuckets(
ClassicHistogramBuckets.builder()
.bucket(-1.0, 0)
.bucket(-0.5, 1)
.bucket(0.0, 0)
.bucket(0.5, 0)
.bucket(1.0, 0)
.bucket(Double.POSITIVE_INFINITY, 0)
.build())
.build())
.build();
assertThat(write(histogram))
.isEqualTo(
"# TYPE my_histogram histogram\n"
+ "my_histogram_bucket{le=\"-1.0\"} 0\n"
+ "my_histogram_bucket{le=\"-0.5\"} 1\n"
+ "my_histogram_bucket{le=\"0.0\"} 1\n"
+ "my_histogram_bucket{le=\"0.5\"} 1\n"
+ "my_histogram_bucket{le=\"1.0\"} 1\n"
+ "my_histogram_bucket{le=\"+Inf\"} 1\n"
+ "# EOF\n");
}
@Test
void gaugeHistogramWithNegativeBucketsKeepsGsum() throws IOException {
HistogramSnapshot histogram =
HistogramSnapshot.builder()
.name("my_gauge_histogram")
.gaugeHistogram(true)
.dataPoint(
HistogramSnapshot.HistogramDataPointSnapshot.builder()
.sum(-7.0)
.classicHistogramBuckets(
ClassicHistogramBuckets.builder()
.bucket(-1.0, 4)
.bucket(Double.POSITIVE_INFINITY, 1)
.build())
.build())
.build();
assertThat(write(histogram))
.isEqualTo(
"# TYPE my_gauge_histogram gaugehistogram\n"
+ "my_gauge_histogram_bucket{le=\"-1.0\"} 4\n"
+ "my_gauge_histogram_bucket{le=\"+Inf\"} 5\n"
+ "my_gauge_histogram_gcount 5\n"
+ "my_gauge_histogram_gsum -7.0\n"
+ "# EOF\n");
}
private static String write(HistogramSnapshot histogram) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
OpenMetricsTextFormatWriter.create().write(out, MetricSnapshots.of(histogram));
return out.toString(StandardCharsets.UTF_8);
}
}./mvnw test -pl prometheus-metrics-exposition-textformats -am -Dtest=NegativeBucketHistogramTest -Dsurefire.failIfNoSpecifiedTests=false
The histogram case fails (the gauge histogram case passes already):
[ERROR] NegativeBucketHistogramTest.histogramWithNegativeBucketsOmitsCountAndSum:38
expected:
"# TYPE my_histogram histogram
my_histogram_bucket{le="-1.0"} 0
my_histogram_bucket{le="-0.5"} 1
my_histogram_bucket{le="0.0"} 1
my_histogram_bucket{le="0.5"} 1
my_histogram_bucket{le="1.0"} 1
my_histogram_bucket{le="+Inf"} 1
# EOF
"
but was:
"# TYPE my_histogram histogram
my_histogram_bucket{le="-1.0"} 0
my_histogram_bucket{le="-0.5"} 1
my_histogram_bucket{le="0.0"} 1
my_histogram_bucket{le="0.5"} 1
my_histogram_bucket{le="1.0"} 1
my_histogram_bucket{le="+Inf"} 1
my_histogram_count 1
my_histogram_sum -0.5
# EOF
"
Environment:
- Commit:
76954702079ded14718b84cbdde7f3bcf1bc03bd; the files involved are unchanged on currentmain(e574d32) - Toolchain: OpenJDK 25.0.5, Apache Maven 3.9.16
Already fixed in prometheus/client_python
PR: prometheus/client_python#496 — "Allow for negtive histogram buckets." (merged 2019-12-19T17:09:34Z, merge commit 87d08deaa94). It stops exposing _sum for histograms whose first bucket is negative, and keeps _gsum for gauge histograms, where a negative value is allowed when there are negative buckets. (It keeps _count; the "if and only if" rule above suggests dropping it too, which is what the existing comment in the Java writer already says.)
How client_python fixed it (relevant hunks, verbatim):
--- a/prometheus_client/metrics.py
+++ b/prometheus_client/metrics.py
@@ -566,7 +566,8 @@ def _child_samples(self):
acc += self._buckets[i].get()
samples.append(('_bucket', {'le': floatToGoString(bound)}, acc))
samples.append(('_count', {}, acc))
- samples.append(('_sum', {}, self._sum.get()))
+ if self._upper_bounds[0] >= 0:
+ samples.append(('_sum', {}, self._sum.get()))
samples.append(('_created', {}, self._created))
return tuple(samples)
--- a/prometheus_client/metrics_core.py
+++ b/prometheus_client/metrics_core.py
@@ -217,10 +217,13 @@ def add_metric(self, labels, buckets, sum_value, timestamp=None):
exemplar,
))
# +Inf is last and provides the count value.
- self.samples.extend([
- Sample(self.name + '_count', dict(zip(self._labelnames, labels)), buckets[-1][1], timestamp),
- Sample(self.name + '_sum', dict(zip(self._labelnames, labels)), sum_value, timestamp),
- ])
+ self.samples.append(
+ Sample(self.name + '_count', dict(zip(self._labelnames, labels)), buckets[-1][1], timestamp))
+ # Don't iunclude sum if there's negative buckets.
+ if float(buckets[0][0]) >= 0 and sum_value is not None:
+ self.samples.append(
+ Sample(self.name + '_sum', dict(zip(self._labelnames, labels)), sum_value, timestamp))
+Suggested fix
In the OpenMetrics writer, skip the _count/_sum pair for non-gauge histograms whose lowest bucket threshold is negative (classic buckets are sorted, so checking the first one is enough):
diff --git a/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java b/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
--- a/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
+++ b/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
@@ -179,11 +179,11 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
if (snapshot.isGaugeHistogram()) {
writeMetadata(writer, "gaugehistogram", metadata, scheme);
writeClassicHistogramBuckets(
- writer, metadata, "_gcount", "_gsum", snapshot.getDataPoints(), scheme);
+ writer, metadata, "_gcount", "_gsum", true, snapshot.getDataPoints(), scheme);
} else {
writeMetadata(writer, "histogram", metadata, scheme);
writeClassicHistogramBuckets(
- writer, metadata, "_count", "_sum", snapshot.getDataPoints(), scheme);
+ writer, metadata, "_count", "_sum", false, snapshot.getDataPoints(), scheme);
}
}
@@ -192,6 +192,7 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
MetricMetadata metadata,
String countSuffix,
String sumSuffix,
+ boolean gaugeHistogram,
List<HistogramSnapshot.HistogramDataPointSnapshot> dataList,
EscapingScheme scheme)
throws IOException {
@@ -225,7 +226,9 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
writeScrapeTimestampAndExemplar(writer, data, exemplar, scheme);
}
// In OpenMetrics format, histogram _count and _sum are either both present or both absent.
- if (data.hasCount() && data.hasSum()) {
+ // A histogram with negative buckets must not have a sum (a gauge histogram may).
+ boolean negativeBuckets = buckets.getUpperBound(0) < 0;
+ if (data.hasCount() && data.hasSum() && (gaugeHistogram || !negativeBuckets)) {
writeCountAndSum(writer, countName, sumName, data, exemplars, scheme, nameValidLegacy);
}
writeCreated(writer, name, data, scheme, nameValidLegacy);Verified on 7695470 (OpenJDK 25.0.5, Maven 3.9.16): the test above fails without this patch and passes with it; no other test results change (config, model and exposition-textformats module tests).
I can submit a PR for this.
Checked: #646 (documenting negative observations) and #1912 (OpenMetrics 2.0 support) are related but do not cover this; searches for negative buckets _sum, histogram sum negative, negative bucket, classicUpperBounds negative found no report of this.
Activity