FEAT: dj_blob type plugin for DataJoint 2.0

Feature Request

Problem

Many existing DataJoint pipelines rely on the blob attribute type to store arbitrary serialized Python objects (e.g., NumPy arrays, dictionaries, custom class instances). While DataJoint 2.0 is moving towards a more structured approach with object types and specialized adaptors, there is a critical need for a backward-compatible solution. Without a standardized <dj_blob> adaptor, migrating legacy pipelines becomes a significant challenge, requiring users to manually update their code to handle serialization, which defeats the purpose of a seamless upgrade path.

Requirements

A successful implementation of this improvement should provide a built-in Custom Type Adaptor that replicates the functionality of the legacy blob type for storing general Python objects. This implementation must adhere to the DataJoint 2.0 Specification.

The core requirements are:

Create a dj.CustomType Adaptor for classic datajoint blobs.�

A new class must be implemented that inherits from dj.CustomType.

Implement the Standard Interface:

The input_type property must return the type name to be used in the definition: dj_blob.
The stored_type property must return the type name to convert into for storage (can be another CustomType.
The put method must accept an arbitrary Python object and serialize it into a binary format suitable for storage in a blob database field.
The get method must accept the binary data from the blob field and deserialize it back into the original Python object.

The <dj_blob> adaptor should be registered by default with the DataJoint client, making it available out-of-the-box without requiring user intervention.

Justification

The primary justification for this feature is backward compatibility. It provides a direct and simple migration path for countless existing pipelines that depend on the ability to store serialized Python objects in blob fields. Furthermore, it offers a standardized, safe, and officially supported method for handling generic Python objects, ensuring consistency and preventing the proliferation of ad-hoc serialization solutions in new pipelines.

This issue is stale because it has been open for 45 days with no activity.

DataJoint 2.0 Implementation

This feature is implemented in DataJoint 2.0 via the <blob> codec (using <blob> syntax rather than <dj_blob>).

Implementation Details

The BlobCodec class in src/datajoint/builtin_codecs.py provides the requested functionality:

class BlobCodec(Codec):
    """
    Serialize Python objects using DataJoint's blob format.
    
    Supports both internal and external storage:
    - <blob>: Stored in database (bytes → LONGBLOB)
    - <blob@>: Stored externally via <hash@> with deduplication
    - <blob@store>: Stored in specific named store
    """
    
    name = "blob"

    def get_dtype(self, is_external: bool) -> str:
        return "<hash>" if is_external else "bytes"

    def encode(self, value, *, key=None, store_name=None) -> bytes:
        from . import blob
        return blob.pack(value, compress=True)

    def decode(self, stored: bytes, *, key=None) -> Any:
        from . import blob
        return blob.unpack(stored, squeeze=False)

Requirements Checklist

Requirement	Status	Implementation
Built-in codec for classic DJ blobs	✅	`BlobCodec` class
`input_type` property	✅	`name = "blob"` (used as `<blob>`)
`stored_type` property	✅	`get_dtype()` returns `bytes` or `<hash>`
`put` method (serialize)	✅	`encode()` using `blob.pack()`
`get` method (deserialize)	✅	`decode()` using `blob.unpack()`
Registered by default	✅	Auto-registers on module load

Usage

@schema
class ProcessedData(dj.Manual):
    definition = '''
    data_id : int
    ---
    small_result : <blob>       # internal (in database)
    large_result : <blob@>      # external (default store, deduplicated)
    archive : <blob@cold>       # external (specific store)
    '''

# Insert any serializable object (NumPy arrays, dicts, etc.)
ProcessedData.insert1({'data_id': 1, 'small_result': {'scores': [0.9, 0.8]}})

Naming Convention

The implementation uses <blob> rather than <dj_blob> for cleaner syntax. The three-layer type system in 2.0 distinguishes:

Core type blob: Raw bytes (no serialization)
Codec <blob>: DataJoint's serialization format (NumPy, dicts, etc.)

This approach is cleaner and consistent with other codecs (<hash>, <object>, <attach>, etc.).

Backward Compatibility

The <blob> codec uses the same blob.pack()/blob.unpack() functions as legacy DataJoint, ensuring full backward compatibility with existing serialized data.

Implemented in DataJoint 2.0

The <blob> codec is built-in and provides backward-compatible blob functionality:

Implementation:

BlobCodec class: src/datajoint/builtin_codecs.py
Auto-registered: Available out-of-the-box
Backward compatible: Serializes arbitrary Python objects (NumPy arrays, dicts, etc.)
DataJoint native format: Uses DataJoint's blob serialization

Usage:

@schema
class Results(dj.Manual):
    definition = """
    id : int
    ---
    data : <blob>  # Stores any Python object
    """

# Insert NumPy array, dict, custom objects
Results.insert1({'id': 1, 'data': np.array([1, 2, 3])})
Results.insert1({'id': 2, 'data': {'key': 'value'}})

The <blob> codec is the default for DataJoint's native object serialization, providing seamless migration from legacy blob fields.

Navigation Menu

Repository navigation

FEAT: `dj_blob` type plugin for DataJoint 2.0 #1262

Feature Request

Problem

Requirements

Justification

DataJoint 2.0 Implementation

Implementation Details

Requirements Checklist

Usage

Naming Convention

Backward Compatibility

Implemented in DataJoint 2.0

Implementation:

Usage:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FEAT: dj_blob type plugin for DataJoint 2.0 #1262

Description

Feature Request

Problem

Requirements

Justification

Activity

github-actions commented on Oct 11, 2025

dimitri-yatsenko commented on Jan 8, 2026

DataJoint 2.0 Implementation

Implementation Details

Requirements Checklist

Usage

Naming Convention

Backward Compatibility

dimitri-yatsenko commented on Jan 9, 2026

Implemented in DataJoint 2.0

Implementation:

Usage:

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

FEAT: `dj_blob` type plugin for DataJoint 2.0 #1262