Repository navigation
RFC: Lossless fixed-width integer bit-packing codec for ADC-style data #813
Description
Hi! This is an exploratory design discussion based on a working prototype and benchmarks.
I’d really appreciate feedback on API shape and integration direction.
Motivation
Many scientific datasets store integer signals where only a subset of bits are meaningful
(e.g. 10–12 bit ADC data stored in uint16). Zarr/numcodecs currently rely on byte-level
compression for such data, which does not explicitly remove unused bits.
Observations
In an ADC-style benchmark (uint16, effective 12 bits, 10M samples), default Zarr compression
reduced storage from ~19 MB to ~7 MB, but further gains plateaued. Existing bit-level tools
in numcodecs (e.g. PackBits) do not support lossless integer bit-width packing.
Proposal
Introduce a lossless integer bit-packing codec/filter that:
- Packs fixed-width integer values using exactly N bits
- Operates per chunk
- Is fully reversible
- Can be composed with existing compression
Prototype
I implemented a pure-Python prototype to validate feasibility:
- Correct round-trip verified
- Storage reduction proportional to effective bit-width
- Zarr v2 compatible (self-describing stream)
Results (summary)
- Bit-packing alone reduces storage predictably (e.g. 12/16 → ~0.75×)
- Bit-packing + Blosc/Zstd achieves size near default compression
- Python implementation is CPU-heavy → optimized backend likely needed
Questions
- Should this live as a codec or filter in numcodecs?
- How should bit-width metadata be handled (header vs external)?
- Is limiting initial scope to uint16 reasonable?
- Any guidance on aligning this with Zarr v3’s codec pipeline?
Activity
LDeakin commentedon Dec 25, 2025
There is a codec specification for a Zarr V3
packbitscodec that can do this. https://github.com/zarr-developers/zarr-extensions/tree/main/codecs/packbitssakshi2433 commentedon Dec 26, 2025
Thanks for the pointer — that’s very helpful.
I took a look at the Zarr v3 PackBits codec specification. As I understand it, that codec is primarily intended for packing bit arrays / boolean-like data, whereas the use case I’m exploring is lossless fixed-width packing of integer values (e.g. storing uint16 ADC data using exactly 10–12 bits per value).
One of the motivations for the RFC was that existing PackBits-style approaches don’t directly address integer bit-width reduction while preserving exact values. My prototype currently handles this by packing integer values bit-wise and carrying minimal metadata for reversibility.
Would the preferred direction be to:
I’d appreciate any guidance on how you’d envision this fitting into the v3 codec model.
LDeakin commentedon Dec 26, 2025
Maybe I am misunderstanding, but doesn't the
last_bitparameter ofpackbitssupport your use case?{ "data_type": "uint16", "codecs": [ { "name": "packbits", "configuration": { "last_bit": 10 } } ], ... }sakshi2433 commentedon Dec 27, 2025
Yes — at the specification level, I agree that
packbitswithfirst_bit/last_bitdoes express the use case I’m exploring (e.g. usinglast_bitto limit effective bit-width foruint16data).What motivated the RFC was less about the expressiveness of the spec itself, and more about the current practical gap between the v3 design and usable tooling today. In particular:
first_bit/last_bitfor integer data (as opposed to boolean/bit arrays),My prototype and benchmarks were intended as a way to explore these questions empirically and help inform how this capability should be implemented, exposed, and documented in practice.
If the preferred direction is to treat this entirely as an implementation and adoption question around the existing v3
packbitscodec, I’m very happy to align with that and focus on prototyping, benchmarking, and integration guidance rather than proposing a new abstraction.