Skip to content

OpenMetrics writer exposes _count/_sum for classic histograms with negative bucket thresholds #2631

Open

Description

@hidingmyname

Location (prometheus/client_java @ 7695470):

      // In OpenMetrics format, histogram _count and _sum are either both present or both absent.
      if (data.hasCount() && data.hasSum()) {
        writeCountAndSum(writer, countName, sumName, data, exemplars, scheme, nameValidLegacy);
      }
  private void validate() {
    // If a histogram or summary observes negative values the sum could be negative.
    // According to OpenMetrics sum should be omitted in that case, but we don't enforce this here.
  }

Problem

The OpenMetrics 1.0 spec says for Histogram:

Negative threshold buckets MAY be used, but then the Histogram MetricPoint MUST NOT contain a sum value as it would no longer be a counter semantically.

and for the text format:

If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix "_count".

OpenMetricsTextFormatWriter does not check the bucket thresholds, so a classic histogram whose first bucket is negative (e.g. Histogram.builder().classicUpperBounds(-1, -0.5, 0, 0.5, 1)) is exposed with _sum (and _count), which is not valid OpenMetrics. The writer's own comment already says _count and _sum come as a pair, so both should be left out in that case; the buckets still carry the count via le="+Inf".

Scope of this report:

  • Only the OpenMetrics 1.0 text writer. The Prometheus text format, protobuf and the experimental OpenMetrics 2.0 writer are not affected by this rule.
  • Gauge histograms are not affected: the spec says "If negative threshold buckets are present, then sum MAY be negative" for GaugeHistogram, so _gcount/_gsum should stay.
  • I have not checked whether a particular ingester rejects the current output.

Reproduction

NegativeBucketHistogramTest.java (prometheus-metrics-exposition-textformats)
package io.prometheus.metrics.expositionformats;

import static org.assertj.core.api.Assertions.assertThat;

import io.prometheus.metrics.model.snapshots.ClassicHistogramBuckets;
import io.prometheus.metrics.model.snapshots.HistogramSnapshot;
import io.prometheus.metrics.model.snapshots.MetricSnapshots;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.Test;

class NegativeBucketHistogramTest {

  @Test
  void histogramWithNegativeBucketsOmitsCountAndSum() throws IOException {
    HistogramSnapshot histogram =
        HistogramSnapshot.builder()
            .name("my_histogram")
            .dataPoint(
                HistogramSnapshot.HistogramDataPointSnapshot.builder()
                    .sum(-0.5)
                    .classicHistogramBuckets(
                        ClassicHistogramBuckets.builder()
                            .bucket(-1.0, 0)
                            .bucket(-0.5, 1)
                            .bucket(0.0, 0)
                            .bucket(0.5, 0)
                            .bucket(1.0, 0)
                            .bucket(Double.POSITIVE_INFINITY, 0)
                            .build())
                    .build())
            .build();

    assertThat(write(histogram))
        .isEqualTo(
            "# TYPE my_histogram histogram\n"
                + "my_histogram_bucket{le=\"-1.0\"} 0\n"
                + "my_histogram_bucket{le=\"-0.5\"} 1\n"
                + "my_histogram_bucket{le=\"0.0\"} 1\n"
                + "my_histogram_bucket{le=\"0.5\"} 1\n"
                + "my_histogram_bucket{le=\"1.0\"} 1\n"
                + "my_histogram_bucket{le=\"+Inf\"} 1\n"
                + "# EOF\n");
  }

  @Test
  void gaugeHistogramWithNegativeBucketsKeepsGsum() throws IOException {
    HistogramSnapshot histogram =
        HistogramSnapshot.builder()
            .name("my_gauge_histogram")
            .gaugeHistogram(true)
            .dataPoint(
                HistogramSnapshot.HistogramDataPointSnapshot.builder()
                    .sum(-7.0)
                    .classicHistogramBuckets(
                        ClassicHistogramBuckets.builder()
                            .bucket(-1.0, 4)
                            .bucket(Double.POSITIVE_INFINITY, 1)
                            .build())
                    .build())
            .build();

    assertThat(write(histogram))
        .isEqualTo(
            "# TYPE my_gauge_histogram gaugehistogram\n"
                + "my_gauge_histogram_bucket{le=\"-1.0\"} 4\n"
                + "my_gauge_histogram_bucket{le=\"+Inf\"} 5\n"
                + "my_gauge_histogram_gcount 5\n"
                + "my_gauge_histogram_gsum -7.0\n"
                + "# EOF\n");
  }

  private static String write(HistogramSnapshot histogram) throws IOException {
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    OpenMetricsTextFormatWriter.create().write(out, MetricSnapshots.of(histogram));
    return out.toString(StandardCharsets.UTF_8);
  }
}

./mvnw test -pl prometheus-metrics-exposition-textformats -am -Dtest=NegativeBucketHistogramTest -Dsurefire.failIfNoSpecifiedTests=false

The histogram case fails (the gauge histogram case passes already):

[ERROR]   NegativeBucketHistogramTest.histogramWithNegativeBucketsOmitsCountAndSum:38 
expected: 
  "# TYPE my_histogram histogram
  my_histogram_bucket{le="-1.0"} 0
  my_histogram_bucket{le="-0.5"} 1
  my_histogram_bucket{le="0.0"} 1
  my_histogram_bucket{le="0.5"} 1
  my_histogram_bucket{le="1.0"} 1
  my_histogram_bucket{le="+Inf"} 1
  # EOF
  "
 but was: 
  "# TYPE my_histogram histogram
  my_histogram_bucket{le="-1.0"} 0
  my_histogram_bucket{le="-0.5"} 1
  my_histogram_bucket{le="0.0"} 1
  my_histogram_bucket{le="0.5"} 1
  my_histogram_bucket{le="1.0"} 1
  my_histogram_bucket{le="+Inf"} 1
  my_histogram_count 1
  my_histogram_sum -0.5
  # EOF
  "

Environment:

Already fixed in prometheus/client_python

PR: prometheus/client_python#496 — "Allow for negtive histogram buckets." (merged 2019-12-19T17:09:34Z, merge commit 87d08deaa94). It stops exposing _sum for histograms whose first bucket is negative, and keeps _gsum for gauge histograms, where a negative value is allowed when there are negative buckets. (It keeps _count; the "if and only if" rule above suggests dropping it too, which is what the existing comment in the Java writer already says.)

How client_python fixed it (relevant hunks, verbatim):

--- a/prometheus_client/metrics.py
+++ b/prometheus_client/metrics.py
@@ -566,7 +566,8 @@ def _child_samples(self):
             acc += self._buckets[i].get()
             samples.append(('_bucket', {'le': floatToGoString(bound)}, acc))
         samples.append(('_count', {}, acc))
-        samples.append(('_sum', {}, self._sum.get()))
+        if self._upper_bounds[0] >= 0:
+            samples.append(('_sum', {}, self._sum.get()))
         samples.append(('_created', {}, self._created))
         return tuple(samples)
 
--- a/prometheus_client/metrics_core.py
+++ b/prometheus_client/metrics_core.py
@@ -217,10 +217,13 @@ def add_metric(self, labels, buckets, sum_value, timestamp=None):
                 exemplar,
             ))
         # +Inf is last and provides the count value.
-        self.samples.extend([
-            Sample(self.name + '_count', dict(zip(self._labelnames, labels)), buckets[-1][1], timestamp),
-            Sample(self.name + '_sum', dict(zip(self._labelnames, labels)), sum_value, timestamp),
-        ])
+        self.samples.append(
+                Sample(self.name + '_count', dict(zip(self._labelnames, labels)), buckets[-1][1], timestamp))
+        # Don't iunclude sum if there's negative buckets.
+        if float(buckets[0][0]) >= 0 and sum_value is not None:
+            self.samples.append(
+                    Sample(self.name + '_sum', dict(zip(self._labelnames, labels)), sum_value, timestamp))
+

Suggested fix

In the OpenMetrics writer, skip the _count/_sum pair for non-gauge histograms whose lowest bucket threshold is negative (classic buckets are sorted, so checking the first one is enough):

diff --git a/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java b/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
--- a/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
+++ b/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetricsTextFormatWriter.java
@@ -179,11 +179,11 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
     if (snapshot.isGaugeHistogram()) {
       writeMetadata(writer, "gaugehistogram", metadata, scheme);
       writeClassicHistogramBuckets(
-          writer, metadata, "_gcount", "_gsum", snapshot.getDataPoints(), scheme);
+          writer, metadata, "_gcount", "_gsum", true, snapshot.getDataPoints(), scheme);
     } else {
       writeMetadata(writer, "histogram", metadata, scheme);
       writeClassicHistogramBuckets(
-          writer, metadata, "_count", "_sum", snapshot.getDataPoints(), scheme);
+          writer, metadata, "_count", "_sum", false, snapshot.getDataPoints(), scheme);
     }
   }
 
@@ -192,6 +192,7 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
       MetricMetadata metadata,
       String countSuffix,
       String sumSuffix,
+      boolean gaugeHistogram,
       List<HistogramSnapshot.HistogramDataPointSnapshot> dataList,
       EscapingScheme scheme)
       throws IOException {
@@ -225,7 +226,9 @@ public class OpenMetricsTextFormatWriter implements ExpositionFormatWriter {
         writeScrapeTimestampAndExemplar(writer, data, exemplar, scheme);
       }
       // In OpenMetrics format, histogram _count and _sum are either both present or both absent.
-      if (data.hasCount() && data.hasSum()) {
+      // A histogram with negative buckets must not have a sum (a gauge histogram may).
+      boolean negativeBuckets = buckets.getUpperBound(0) < 0;
+      if (data.hasCount() && data.hasSum() && (gaugeHistogram || !negativeBuckets)) {
         writeCountAndSum(writer, countName, sumName, data, exemplars, scheme, nameValidLegacy);
       }
       writeCreated(writer, name, data, scheme, nameValidLegacy);

Verified on 7695470 (OpenJDK 25.0.5, Maven 3.9.16): the test above fails without this patch and passes with it; no other test results change (config, model and exposition-textformats module tests).

I can submit a PR for this.

Checked: #646 (documenting negative observations) and #1912 (OpenMetrics 2.0 support) are related but do not cover this; searches for negative buckets _sum, histogram sum negative, negative bucket, classicUpperBounds negative found no report of this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions