[zarr-metadata] runtime type checking #4447

this is a bit of a retroactive issue, but if you've been following the PRs recently there's been a large volume of AI-written changes against zarr-metadata. Sorry for the noise, but there's a big effort going on. I'll explain what that is in this issue.

for motivation, i'm tired of stupid JSON parsing bugs in zarr. I call these "stupid" bugs because they are caused by bad tooling. Python offers typeddict types for modelling JSON payloads, which is great. But in order to narrow an arbitrary JSON input to a typeddict, you have to exhaustively check all the constraints defined by the typeddict, such as the presence / absence of required keys, the type of each value (this is likely recursive), the type of undeclared keys (the spec defines rules for this). And after that, you have to check the spec-defined constraints for the values, like "int between 0 and 10" as a refinement of "int". There are a lot of ways to create drifting representations of the same thing, and lots of opportunities to omit a crucial type check that lets garbage in, or to produce garbage. zarr should not produce garbage, so we need something to address this systematically.

tl;dr, I'm adding a runtime type checker + extensive validation logic to zarr-metadata. That means zarr-metadata will have routines that check the following invariants of the stuff in zarr.json, ordered by depth:

• it's well-formed JSON

• the outer structure of that JSON complies with the spec. Is a "codecs" field present, is it a JSON array, etc.

• the outer structure of the various JSON fields complies with their respective specs.

  This is actually a bit deeper than you might think, because the spec admits many different spellings of the same thing: "bytes", {"name": "bytes"}, {"name": "bytes", "must_understand": False}, {"name": "bytes", "configuration": {}}, {"name": "bytes", "must_understand": False, "configuration": {}} are all valid ways to declare the exact same bytes codec. The must_understand ones are incoherent, but that's a spec bug.

  Second, this kind of check must operate under a closed set of codecs / chunk key encodings / chunk grids, etc, because evaluating the validity of the sharding codec requires evaluating the validity of the codecs it defines, so you have to start your sharding codec validation procedure with a set of possible codecs in hand. and codec JSON doesn't contain the codec type (array -> array, array -> bytes, etc) so checking that invariant requires resolving codec JSON to a codec definition.

• the constraints between fields are satisfied. e.g., the fill_value is consistent with the data_type. The shape is consistent with all dimensionality-sensitive fields like transpose codec configuration, chunk grid, etc.

• the action of the array -> array codecs are well-defined, given the final array configuration (fill value, data type, and chunk grid). this one is particularly sensitive because the rectilinear chunk grid defines a large set of chunk shapes, and all chunk shapes must satisfy some codec-specific predicates of codecs like sharding_indexed. getting this wrong means silently corrupted data.

and all this has to be extensible. we need data structures that allow a maintainer to easily write a new codec definition without a ton of boilerplate. we need to ensure that the JSON types (typeddicts, necessary for modelling JSON) don't drift relative to the dataclasses (necessary for giving users something easy to work with, and for coupling validation routines to object construction).

the obvious question is why do this in-house. why not use pydantic or msgspec. my answer is:

• we don't need a lot of the things pydantic and msgspec can do, and we want zarr-metadata in particular to be as dependency-lite as possible. bringing in pydantic as a required dependency would prevent consumers from using it.
• some of the things we need, mostly pertaining to typeddicts, msgspec cannot do properly. typeddicts in particular seem to move faster than runtime type checkers can adapt, and I don't want to wait ages for msgspec to figure out recursive type aliases (table stakes for JSON values!). I assume pydantic does a better job, but since so many zarr consumers might want to use their own version of pydantic, I would rather not make it a core dependency. but type-checking JSON is a core need for zarr.
• AI coding agents make this code straightforward to write and maintain. we have plenty of exemplars to draw inspiration from (zod, pydantic, msgspec) and the problem is bounded.

so anyway, this feature is coming in the next release of zarr-metadata. After that we can start looking at bringing this functionality into zarr, and then we have fewer dumb bugs.