ShardingCodec.validate incorrectly assumes that Zdtype.default_scalar is always a valid value #4511

Zarr version

v3.4.1

Numcodecs version

0.17.0

Python Version

3.12.3

Operating System

Linux

Installation

using pip into a virtual environment

Description

When creating an array, the ShardingCodec.validate function checks that the codec configuration is compatible with the array metadata.

It does this by trying to encode the dtype.default_scalar() value as the fill_value for an ArraySpec and then checking that each codec can validate the spec.

However, it is not always the case that the dtype.default_scalar() is a valid value for a given array and this may cause the array creation to fail.

For example:

• I want to create an array of float32 values that I know are within the range of 100-200 (Note that this range does not include zero)
• I want to encode these values as uint16 on disk (in order to save space) by scaling them between 0 and 65535 and then casting them to uint16 (using the ScaleOffset and CastValue codecs).

In this example, trying to encode a value of zero (the Float32.default_scalar() value) will scale to a negative value which cannot be encoded as uint16 and the array creation will fail.
In this example though I have no need to store zero values and so this validation is checking the wrong thing.

For now I can work around this by explicitly clamping zero to a valid value within my expected range but that is an unnecessary step and makes the array codec definition unnecessarily complicated. It also potentially hides future errors if an actual out-range-value gets added to the array later on.

It would be better if we could provide an alternative value (other than dtype.default_scalar()) to the ShardingCodec that it can use for validation in these sort of cases.

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr

minimum = 100
maximum = 200

zarr.create_array(
  store={}, 
  shape=(0,), 
  dtype="float32", 
  filters=[
    zarr.codecs.ScaleOffset(scale=(65535 / (maximum - minimum)), offset=minimum),
    zarr.codecs.CastValue(data_type="uint16", rounding="nearest-even")
  ]
)

# raises ValueError: Value -65535 out of range [0, 65535]. Set out_of_range='clamp' or out_of_range='wrap' to handle this

Additional output

Note that to reproduce the above example, the cast-value-rs pip package must also be installed.