FEAT: Implement <dj_zarr> Custom Type Adaptor

Feature Request

Problem

Scientific research, particularly in fields like neuroscience and imaging, generates massive n-dimensional arrays that are too large to be stored efficiently in traditional blob fields. The Zarr format is an industry standard for storing chunked, compressed array data, enabling parallel I/O and efficient partial access (slicing). Without native support for Zarr, users are forced to manage these datasets manually by storing file paths, which disconnects the data from the DataJoint pipeline and forfeits all the benefits of integrated data management and integrity checks.

Requirements

A successful implementation of this improvement should provide a built-in Custom Type Adaptor for handling Zarr arrays, leveraging the object type for external storage. This implementation must adhere to the DataJoint 2.0 Specification.

The core requirements are:

Create a dj.CustomType Adaptor:

A new class must be implemented that inherits from dj.CustomType as a plugin

Implement the Standard Interface:

The type_name property must return the string <dj_zarr>.
The stored_type property must return the object.
The put method must accept a Zarr-compatible array object (e.g., a NumPy array) and write it to the configured external object store as a Zarr dataset.
The get method must read the Zarr dataset from the external store and return it as a lazy-loading Zarr array object.

Default Registration:

The <dj_zarr> adaptor should be registered by default with the DataJoint client, making it available out-of-the-box.

Justification

Providing native support for Zarr arrays via a type adaptor will be a transformative feature for DataJoint. It will enable the seamless management of petabyte-scale array data in a format that is optimized for high-performance, parallel computing and cloud environments. This directly addresses the needs of modern data-intensive science and solidifies DataJoint's position as a cutting-edge platform for scientific data management.

This issue is stale because it has been open for 45 days with no activity.

This will be implemented as a separate project and tested for integration here.

+1 this from my side! I am working a lot with zarr nowadays.
I particularly like the idea of using some kind of lazy loading, i.e. that there is a zarr object saved in the cloud (as external object in dj) that can be indexed from downstream processes in the pipeline without loading the whole zarr array to disk first - i.e. using zarr's indexability from within datajoint if that makes sense.

@dimitri-yatsenko

Implementation Plan for DataJoint 2.0

The DataJoint 2.0 Codec system provides the infrastructure to implement <zarr> support. Here's how it would work:

Codec Implementation

class ZarrCodec(dj.Codec):
    """
    Store NumPy arrays as Zarr datasets in external object storage.
    
    Usage:
        data : <zarr@store>      # Store as Zarr in named store
        data : <zarr@>           # Store as Zarr in default store
    """
    
    name = "zarr"
    
    def get_dtype(self, is_external: bool) -> str:
        # Zarr is always external (uses object storage)
        return "<object>"
    
    def encode(self, value, *, key=None, store_name=None):
        """Write array to Zarr format in object storage."""
        import zarr
        import numpy as np
        
        # Convert to numpy if needed
        if not isinstance(value, np.ndarray):
            value = np.asarray(value)
        
        # Write to temporary Zarr, then upload
        # The <object> codec handles the actual storage
        return {"_zarr": True, "shape": value.shape, "dtype": str(value.dtype)}
    
    def decode(self, stored, *, key=None):
        """Return lazy-loading Zarr array from object storage."""
        import zarr
        
        # Return zarr.Array backed by fsspec for lazy loading
        # This enables slicing without loading entire array
        return zarr.open(stored["path"], mode="r")

Key Features

Lazy Loading: Returns zarr.Array object, not loaded data. Users can slice efficiently:

arr = (MyTable & key).fetch1("data")  # Returns zarr.Array
chunk = arr[100:200, :, :]  # Only loads this slice from cloud

Cloud-Native: Uses <object@store> for storage, leveraging fsspec backends (S3, GCS, Azure)
Chunked Storage: Zarr's chunking enables parallel writes and partial reads

Implementation Options

Option A: Built-in codec (in datajoint.builtin_codecs)

Available out-of-box
Adds zarr as optional dependency

Option B: Plugin package (datajoint-zarr)

Separate install: pip install datajoint-zarr
Auto-registers via entry points
Keeps core datajoint lightweight

Given @dimitri-yatsenko's comment about implementing as a separate project, Option B seems preferred. The entry point system in DataJoint 2.0 supports this:

# In datajoint-zarr/pyproject.toml
[project.entry-points."datajoint.codecs"]
zarr = "datajoint_zarr:ZarrCodec"

This allows <zarr@store> to work automatically once datajoint-zarr is installed.

Implemented in DataJoint 2.0

Zarr support is implemented via the <object> storage type:

Implementation:

The <object> type provides native Zarr support through fsspec:

@schema
class LargeData(dj.Computed):
    definition = """
    -> Upstream
    ---
    zarr_data : <object>
    """
    
    def make(self, key):
        # Create Zarr array directly in object storage
        with self.staged_insert(key, 'zarr_data', extension='.zarr') as ref:
            store = zarr.DirectoryStore(ref.path)
            z = zarr.open(store, mode='w', shape=(10000, 10000))
            z[:] = compute_data()

# Fetch returns ObjectRef with zarr-compatible interface
data = LargeData.fetch1()
z = zarr.open(data['zarr_data'].fsmap, mode='r')  # Lazy loading

Features:

Staged insert: Write large Zarr arrays directly to storage
fsmap interface: Compatible with zarr.open()
Lazy loading: Data loaded on demand
Cloud support: Works with S3, GCS, Azure via fsspec

See src/datajoint/objectref.py for the ObjectRef implementation with fsmap support.

Navigation Menu

Repository navigation

FEAT: Implement <dj_zarr> Custom Type Adaptor #1263

Feature Request

Problem

Requirements

Default Registration:

Justification

Implementation Plan for DataJoint 2.0

Codec Implementation

Key Features

Implementation Options

Implemented in DataJoint 2.0

Implementation:

Features:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FEAT: Implement <dj_zarr> Custom Type Adaptor #1263

Description

Feature Request

Problem

Requirements

Default Registration:

Justification

Activity

github-actions commented on Oct 3, 2025

dimitri-yatsenko commented on Dec 22, 2025

horsto commented on Jan 8, 2026

dimitri-yatsenko commented on Jan 8, 2026

Implementation Plan for DataJoint 2.0

Codec Implementation

Key Features

Implementation Options

dimitri-yatsenko commented on Jan 9, 2026

Implemented in DataJoint 2.0

Implementation:

Features:

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions