Skip to content

[C++][Python] tdigest and hash_tdigest return NaN for q outside [0, 1]; quantile raises #55372

Open

Description

@chrikrah

Describe the bug, including details regarding any error messages, version, and platform.

pyarrow.compute.tdigest and hash_tdigest accept a q outside [0, 1] and return NaN, while quantile rejects the same value with ArrowInvalid. TDigestOptions documents the range as a requirement.

import pyarrow as pa
import pyarrow.compute as pc

print(pa.__version__)
a = pa.array([1.0, 2.0, 3.0])
for fn in (pc.quantile, pc.tdigest):
    try:
        print(fn.__name__, fn(a, q=[0.5, 1.5]).to_pylist())
    except pa.ArrowInvalid as e:
        print(fn.__name__, type(e).__name__, e)

t = pa.table({"k": [1, 1, 1], "v": [1.0, 2.0, 3.0]})
agg = t.group_by("k").aggregate([("v", "tdigest", pc.TDigestOptions(q=[7.0]))])
print(agg.to_pylist())
$ python -I td.py    # pyarrow 26.0.0 wheel, Python 3.12, Linux x86_64
26.0.0
quantile ArrowInvalid Quantile must be between 0 and 1
tdigest [2.0, nan]
[{'k': 1, 'v_tdigest': [nan]}]

contract: TDigestOptions.q in python/pyarrow/_compute.pyx:2388 ("All values must be in [0, 1]"), and cpp/src/arrow/compute/api_aggregate.h ("probability level of quantile must be between 0 and 1 inclusive").

CheckQuantileOptions (cpp/src/arrow/compute/kernels/aggregate_quantile.cc:482, a99464f on main) returns Status::Invalid("Quantile must be between 0 and 1") for any q outside the range. Neither aggregate_tdigest.cc nor GroupedTDigestImpl in hash_aggregate_numeric.cc (the separate hash_tdigest kernel) checks q, and TDigest::Quantile in cpp/src/arrow/util/tdigest.cc returns NAN for q < 0 || q > 1.

Expected: ArrowInvalid for any q outside [0, 1], from a shared check called by both the tdigest and hash_tdigest inits, with the message quantile uses.

I read the source at a99464f and ran the 26.0.0 wheel, not a source build. I did not check how q=NaN should behave: quantile also lets it through.

I can open a PR with the check and tests in test_compute.py, kernels/aggregate_test.cc and acero/hash_aggregate_test.cc. Would you take it?

Component(s)

C++, Python

Activity

  1. Abhi2939 commented on Nov 4, 2026

    @Abhi2939

    Hi @chrikrah, this looks like a clean fix. Are you still planning to open the PR? If you’d rather not, I’d be happy to take it. I’d add the check next to CheckQuantileOptions, call it from both the tdigest and hash_tdigest inits with the same message, and add tests in the three places you listed. I’ll leave q=NaN as it is now unless a maintainer wants it changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions