Proposal: Query NHCB as classic histograms in PromQL #7915

Proposal: Query NHCB as classic histograms in PromQL

Problem

When users migrate from classic histograms to Native Histograms with Custom Buckets (NHCB), their existing PromQL queries break. Classic histogram queries use suffixed metric names (metric_bucket, metric_sum, metric_count), while NHCB uses native histogram PromQL syntax (histogram_count(), histogram_sum(), etc.).

This creates a migration barrier: operators cannot switch their ingestion pipeline to NHCB without coordinating a simultaneous rewrite of all downstream queries, dashboards, and alerting rules. In a multi-tenant system like Cortex, this is especially painful because the operator may control the ingestion format but individual tenants own their queries.

Goal

Allow tenants to query NHCB series using classic histogram PromQL syntax, transparently. When a user queries metric_bucket{le="0.5"}, Cortex should also look for an NHCB series named metric and synthesize the classic histogram samples from it. This makes the NHCB migration invisible to the query side.

Requirements

• Per-tenant configuration: Enabled/disabled via runtime config, off by default.
• No query rewrite required: Existing classic histogram queries work unchanged.
• Transparent to users: No new query syntax or API changes needed.
• Correct during migration: Must handle the transient state where both classic and NHCB series exist for the same metric. Classic series take precedence when both are present at the same timestamp. Per-step conflict resolution is needed for range queries spanning the migration window.
• Suffix handling: Support _bucket, _count, and _sum suffix queries against the underlying NHCB series.

Scope

In scope:

• Instant and range vector selectors in PromQL instant/range queries.
• Synthesizing _bucket, _count, _sum float samples from NHCB.
• Correct conflict resolution when classic and NHCB coexist.

Out of scope (future work):

• Emulating classic histogram suffixes from exponential native histograms.
• Regex matchers on __name__ that match suffixed names.
• Non-PromQL API endpoints (e.g. /api/v1/series, /api/v1/labels).

Design Considerations

Prometheus implemented this feature in the storage interface layer (between TSDB and the PromQL engine) via a NHCBAsClassicQuerier wrapper. In Cortex, the analogous insertion point is the querier layer — after merging results from ingesters and store-gateway, before returning to the PromQL engine.

Key areas to evaluate:

1. Where to insert the conversion: The wrapper needs to sit in the query path where it can intercept Select calls, issue a parallel Select for the base metric name (without suffix), and merge/deduplicate results.

2. Performance: Each suffixed query may trigger an additional Select for the base metric. In a multi-tenant environment with high query concurrency, the overhead of these extra lookups needs measurement. Caching whether a metric has NHCB data could mitigate repeated lookups.

3. Ingester + store-gateway fan-out: The additional Select must go through the same ingester and store-gateway path. The wrapper should be placed after the merge point so the extra query is issued once, not once-per-ingester.

Test Plan

• Correctness: Query classic histogram suffixes against NHCB-only data and verify results match equivalent native histogram PromQL functions.
• Migration: Test time ranges spanning classic-only → classic+NHCB → NHCB-only transitions.
• Performance: Compare query latency and resource usage of emulated classic queries vs. native histogram queries on the same data.
• Conflict resolution: Verify classic series win when both classic and NHCB exist at the same timestamp.