FEAT: Object-augmented schemas -- File Type

Object Column Type Specification

Overview

The object type introduces a new paradigm for managed file storage in DataJoint. Unlike existing attach@store and filepath@store types that reference named stores, the object type uses a unified storage backend that is tightly coupled with the schema and configured at the pipeline level.

The object type supports both files and folders. Content is copied to storage at insert time, referenced via handle on fetch, and deleted when the record is deleted.

Immutability Contract

Objects stored via the object type are immutable after finalization. Users agree to:

Insert (copy): Copy existing content to storage
Insert (staged): Reserve path, write directly, then finalize
Fetch: Read content via handle (no modification)
Delete: Remove content when record is deleted (only way to remove)

Once an object is finalized (either via copy-insert or staged-insert completion), users must not directly modify it in the object store.

Two Insert Modes

Mode	Use Case	Workflow
Copy	Small files, existing data	Local file → copy to storage → insert record
Staged	Large objects, Zarr/HDF5	Reserve path → write directly to storage → finalize record

Storage Architecture

Single Storage Backend Per Pipeline

Each DataJoint pipeline has one associated storage backend configured in datajoint.json. DataJoint fully controls the path structure within this backend.

Supported Backends

DataJoint uses fsspec to ensure compatibility across multiple storage backends:

Local storage – POSIX-compliant file systems (e.g., NFS, SMB)
Cloud-based object storage – Amazon S3, Google Cloud Storage, Azure Blob, MinIO

Project Structure

A DataJoint project creates a structured hierarchical storage pattern:

📁 project_name/
├── datajoint_store.json         # store metadata (not client config)
├── 📁 schema_name/
│   ├── 📁 Table1/
│   │   ├── data.parquet         # tabular data export (future)
│   │   └── 📁 objects/          # object storage for this table
│   │       ├── pk1=val1/pk2=val2/field1_token.dat
│   │       └── pk1=val1/pk2=val2/field2_token.zarr
│   ├── 📁 Table2/
│   │   ├── data.parquet
│   │   └── 📁 objects/
│   │       └── ...

Object Storage Keys

When using cloud object storage:

s3://bucket/project_name/schema_name/Table1/objects/pk1=val1/field_token.dat
s3://bucket/project_name/schema_name/Table1/objects/pk1=val1/field_token.zarr

Configuration

Settings Structure

Object storage is configured in datajoint.json using the existing settings system:

{
    "database.host": "localhost",
    "database.user": "datajoint",

    "object_storage.project_name": "my_project",
    "object_storage.protocol": "s3",
    "object_storage.endpoint": "s3.amazonaws.com",
    "object_storage.bucket": "my-bucket",
    "object_storage.location": "my_project",
    "object_storage.partition_pattern": "{subject_id}/{session_id}"
}

For local filesystem storage:

{
    "object_storage.project_name": "my_project",
    "object_storage.protocol": "file",
    "object_storage.location": "/data/my_project",
    "object_storage.partition_pattern": "{subject_id}/{session_id}"
}

Settings Schema

Setting	Type	Required	Description
`object_storage.project_name`	string	Yes	Unique project identifier (must match store metadata)
`object_storage.protocol`	string	Yes	Storage backend: `file`, `s3`, `gcs`, `azure`
`object_storage.location`	string	Yes	Base path or bucket prefix
`object_storage.bucket`	string	For cloud	Bucket name (S3, GCS, Azure)
`object_storage.endpoint`	string	For S3	S3 endpoint URL
`object_storage.partition_pattern`	string	No	Path pattern with `{attribute}` placeholders
`object_storage.token_length`	int	No	Random suffix length for filenames (default: 8, range: 4-16)
`object_storage.access_key`	string	For cloud	Access key (can use secrets file)
`object_storage.secret_key`	string	For cloud	Secret key (can use secrets file)

Environment Variables

Settings can be overridden via environment variables:

DJ_OBJECT_STORAGE_PROTOCOL=s3
DJ_OBJECT_STORAGE_BUCKET=my-bucket
DJ_OBJECT_STORAGE_LOCATION=my_project
DJ_OBJECT_STORAGE_PARTITION_PATTERN="subject{subject_id}/session{session_id}"

Secrets

Credentials can be stored in the .secrets/ directory:

.secrets/
├── object_storage.access_key
└── object_storage.secret_key

Partition Pattern

The partition pattern is configured per pipeline (one per settings file). Placeholders use {attribute_name} syntax and are replaced with primary key values.

{
    "object_storage.partition_pattern": "subject{subject_id}/session{session_id}"
}

Example with partitioning:

s3://my-bucket/my_project/subject_id=123/session_id=45/schema_name/Recording/objects/raw_data_Ax7bQ2kM.dat

If no partition pattern is specified, files are organized directly under {location}/{schema}/{Table}/objects/.

Store Metadata (`datajoint_store.json`)

Each object store contains a metadata file at its root that identifies the store and enables verification by DataJoint clients. This file is named datajoint_store.json to distinguish it from client configuration files (datajoint.json).

Location

{location}/datajoint_store.json

For cloud storage:

s3://bucket/my_project/datajoint_store.json

Content

{
    "project_name": "my_project",
    "created": "2025-01-15T10:30:00Z",
    "format_version": "1.0",
    "datajoint_version": "0.15.0",
    "database_host": "db.example.com",
    "database_name": "my_project_db"
}

Schema

Field	Type	Required	Description
`project_name`	string	Yes	Unique project identifier
`created`	string	Yes	ISO 8601 timestamp of store creation
`format_version`	string	Yes	Store format version for compatibility
`datajoint_version`	string	Yes	DataJoint version that created the store
`database_host`	string	No	Database server hostname (for bidirectional mapping)
`database_name`	string	No	Database name (for bidirectional mapping)

The optional database_host and database_name fields enable bidirectional mapping between object stores and databases. This is informational only - not enforced at runtime. Administrators can alternatively ensure unique project_name values across their namespace, and managed platforms may handle this mapping externally.

Store Initialization

The store metadata file is created when the first object attribute is used:

┌─────────────────────────────────────────────────────────┐
│ 1. Client attempts first file operation                 │
├─────────────────────────────────────────────────────────┤
│ 2. Check if datajoint_store.json exists                 │
│    ├─ If exists: verify project_name matches            │
│    └─ If not: create with current project_name          │
├─────────────────────────────────────────────────────────┤
│ 3. On mismatch: raise DataJointError                    │
└─────────────────────────────────────────────────────────┘

Client Verification

DataJoint performs a basic verification on connect to ensure store-database cohesion:

On connect: Client reads datajoint_store.json from store
Verify: project_name in client settings matches store metadata
On mismatch: Raise DataJointError with descriptive message

# Example error
DataJointError: Object store project name mismatch.
  Client configured: "project_a"
  Store metadata: "project_b"
  Ensure all clients use the same object_storage.project_name setting.

Administrative Responsibility

A 1:1 correspondence is assumed between:

Database location + project_name in client settings
Object store + project_name in store metadata

DataJoint performs basic verification but does not enforce this mapping. Administrators are responsible for ensuring correct configuration across all clients.

Syntax

@schema
class Recording(dj.Manual):
    definition = """
    subject_id : int
    session_id : int
    ---
    raw_data : object          # managed file storage
    processed : object         # another object attribute
    """

Note: No @store suffix needed - storage is determined by pipeline configuration.

Database Storage

The object type is stored as a JSON column in MySQL containing:

File example:

{
    "path": "my_schema/Recording/objects/subject_id=123/session_id=45/raw_data_Ax7bQ2kM.dat",
    "size": 12345,
    "hash": null,
    "ext": ".dat",
    "is_dir": false,
    "timestamp": "2025-01-15T10:30:00Z",
    "mime_type": "application/octet-stream"
}

File with optional hash:

{
    "path": "my_schema/Recording/objects/subject_id=123/session_id=45/raw_data_Ax7bQ2kM.dat",
    "size": 12345,
    "hash": "sha256:abcdef1234...",
    "ext": ".dat",
    "is_dir": false,
    "timestamp": "2025-01-15T10:30:00Z",
    "mime_type": "application/octet-stream"
}

Folder example:

{
    "path": "my_schema/Recording/objects/subject_id=123/session_id=45/raw_data_pL9nR4wE",
    "size": 567890,
    "hash": null,
    "ext": null,
    "is_dir": true,
    "timestamp": "2025-01-15T10:30:00Z",
    "item_count": 42
}

JSON Schema

Field	Type	Required	Description
`path`	string	Yes	Full path/key within storage backend (includes token)
`size`	integer	Yes	Total size in bytes (sum for folders)
`hash`	string/null	Yes	Content hash with algorithm prefix, or null (default)
`ext`	string/null	Yes	File extension (e.g., `.dat`, `.zarr`) or null
`is_dir`	boolean	Yes	True if stored content is a directory
`timestamp`	string	Yes	ISO 8601 upload timestamp
`mime_type`	string	No	MIME type (files only, auto-detected from extension)
`item_count`	integer	No	Number of files (folders only)

Content Hashing

By default, no content hash is computed to avoid performance overhead for large objects. Storage backend integrity is trusted.

Optional hashing can be requested per-insert:

# Default - no hash (fast)
Recording.insert1({..., "raw_data": "/path/to/large.dat"})

# Request hash computation
Recording.insert1({..., "raw_data": "/path/to/important.dat"}, hash="sha256")

Supported hash algorithms: sha256, md5, xxhash (xxh3, faster for large files)

Staged inserts never compute hashes - data is written directly to storage without a local copy to hash.

Folder Manifests

For folders (directories), a manifest file is created alongside the folder in the object store to enable integrity verification without computing content hashes:

raw_data_pL9nR4wE/
raw_data_pL9nR4wE.manifest.json

Manifest content:

{
    "files": [
        {"path": "file1.dat", "size": 1234},
        {"path": "subdir/file2.dat", "size": 5678},
        {"path": "subdir/file3.dat", "size": 91011}
    ],
    "total_size": 567890,
    "item_count": 42,
    "created": "2025-01-15T10:30:00Z"
}

Design rationale:

Stored in object store (not database) to avoid bloating the JSON for folders with many files
Placed alongside folder (not inside) to avoid polluting folder contents and interfering with tools like Zarr
Enables self-contained verification without database access

The manifest enables:

Quick verification that all expected files exist
Size validation without reading file contents
Detection of missing or extra files

Filename Convention

The stored filename is always derived from the field name:

Base name: The attribute/field name (e.g., raw_data)
Extension: Adopted from source file (copy insert) or optionally provided (staged insert)
Token: Random suffix for collision avoidance

Stored filename = {field}_{token}{ext}

Examples:
  raw_data_Ax7bQ2kM.dat     # file with .dat extension
  raw_data_pL9nR4wE.zarr    # Zarr directory with .zarr extension
  raw_data_kM3nP2qR         # directory without extension

This convention ensures:

Consistent, predictable naming across all objects
Field name visible in storage for easier debugging
Extension preserved for MIME type detection and tooling compatibility

Path Generation

Storage paths are deterministically constructed from record metadata, enabling bidirectional lookup between database records and stored files.

Path Components

Location - from configuration (object_storage.location)
Partition attributes - promoted PK attributes (if partition_pattern configured)
Schema name - from the table's schema
Table name - the table class name
Object directory - objects/
Primary key encoding - remaining PK attributes and values
Suffixed filename - {field}_{token}{ext}

Path Template

Without partitioning:

{location}/{schema}/{Table}/objects/{pk_attr1}={pk_val1}/{pk_attr2}={pk_val2}/.../{field}_{token}{ext}

With partitioning:

{location}/{partition_attr}={val}/.../schema/{Table}/objects/{remaining_pk_attrs}/.../{field}_{token}{ext}

Note: The objects/ directory follows the table name, allowing each table folder to also contain tabular data exports (e.g., data.parquet) alongside the objects.

Partitioning

The partition pattern allows promoting certain primary key attributes to the beginning of the path (after location). This organizes storage by high-level attributes like subject or experiment, enabling:

Efficient data locality for related records
Easier manual browsing of storage
Potential for storage tiering by partition

Configuration:

{
    "object_storage.partition_pattern": "{subject_id}/{experiment_id}"
}

Partition attributes are extracted from the primary key and placed at the path root. Remaining PK attributes appear in their normal position.

Example Without Partitioning

For a table:

@schema
class Recording(dj.Manual):
    definition = """
    subject_id : int
    session_id : int
    ---
    raw_data : object
    """

Inserting {"subject_id": 123, "session_id": 45, "raw_data": "/path/to/recording.dat"} produces:

my_project/my_schema/Recording/objects/subject_id=123/session_id=45/raw_data_Ax7bQ2kM.dat

Note: The filename is raw_data (field name) with .dat extension (from source file).

Example With Partitioning

With partition_pattern = "{subject_id}":

my_project/subject_id=123/my_schema/Recording/objects/session_id=45/raw_data_Ax7bQ2kM.dat

The subject_id is promoted to the path root, grouping all files for subject 123 together regardless of schema or table.

Deterministic Bidirectional Mapping

The path structure (excluding the random token) is fully deterministic:

Record → File: Given a record's primary key, construct the path prefix to locate its file
File → Record: Parse the path to extract schema, table, field, and primary key values

This enables:

Finding all files for a specific record
Identifying which record a file belongs to
Auditing storage against database contents

The random token is stored in the JSON metadata to complete the full path.

Primary Key Value Encoding

Primary key values are encoded directly in paths when they are simple, path-safe types:

Integers: Used directly (subject_id=123)
Dates: ISO format (session_date=2025-01-15)
Timestamps: ISO format with safe separators (created=2025-01-15T10-30-00)
Simple strings: Used directly if path-safe (experiment=baseline)

Conversion to path-safe strings is applied only when necessary:

Strings containing /, \, or other path-unsafe characters
Very long strings (truncated with hash suffix)
Binary or complex types (hashed)

# Direct encoding (no conversion needed)
subject_id=123
session_date=2025-01-15
trial_type=control

# Converted encoding (path-unsafe characters)
filename=my%2Ffile.dat          # "/" encoded
description=a1b2c3d4_abc123     # long string truncated + hash

Filename Collision Avoidance

To prevent filename collisions, each stored object receives a random token suffix appended to the field name:

field: raw_data, source: recording.dat
stored: raw_data_Ax7bQ2kM.dat

field: image, source: scan.tiff
stored: image_pL9nR4wE.tiff

field: neural_data (staged with .zarr)
stored: neural_data_kM3nP2qR.zarr

Token Suffix Specification

Alphabet: URL-safe and filename-safe Base64 characters: A-Z, a-z, 0-9, -, _
Length: Configurable via object_storage.token_length (default: 8, range: 4-16)
Generation: Cryptographically random using secrets.token_urlsafe()

At 8 characters with 64 possible values per character: 64^8 = 281 trillion combinations.

Rationale

Avoids collisions without requiring existence checks
Field name visible in storage for easier debugging/auditing
URL-safe for web-based access to cloud storage
Filesystem-safe across all supported platforms

No Deduplication

Each insert stores a separate copy of the file, even if identical content was previously stored. This ensures:

Clear 1:1 relationship between records and files
Simplified delete behavior
No reference counting complexity

Insert Behavior

At insert time, the object attribute accepts:

File path (string or Path): Path to an existing file (extension extracted)
Folder path (string or Path): Path to an existing directory
Tuple of (ext, stream): File-like object with explicit extension

# From file path - extension (.dat) extracted from source
Recording.insert1({
    "subject_id": 123,
    "session_id": 45,
    "raw_data": "/local/path/to/recording.dat"
})
# Stored as: raw_data_Ax7bQ2kM.dat

# From folder path - no extension
Recording.insert1({
    "subject_id": 123,
    "session_id": 45,
    "raw_data": "/local/path/to/data_folder/"
})
# Stored as: raw_data_pL9nR4wE/

# From stream with explicit extension
with open("/local/path/data.bin", "rb") as f:
    Recording.insert1({
        "subject_id": 123,
        "session_id": 45,
        "raw_data": (".bin", f)
    })
# Stored as: raw_data_kM3nP2qR.bin

Insert Processing Steps

Validate input (file/folder exists, stream is readable)
Generate deterministic storage path with random token
Copy content to storage backend via fsspec
If copy fails: abort insert (no database operation attempted)
Compute content hash (SHA-256)
Build JSON metadata structure
Execute database INSERT

Copy-First Semantics

The file/folder is copied to storage before the database insert is attempted:

If the copy fails, the insert does not proceed
If the copy succeeds but the database insert fails, an orphaned file may remain
Orphaned files are acceptable due to the random token (no collision with future inserts)

Staged Insert (Direct Write Mode)

For large objects like Zarr arrays, copying from local storage is inefficient. Staged insert allows writing directly to the destination.

Why a Separate Method?

Staged insert uses a dedicated staged_insert1 method rather than co-opting insert1 because:

Explicit over implicit - Staged inserts have fundamentally different semantics (file creation happens during context, commit on exit). A separate method makes this explicit.
Backward compatibility - insert1 returns None and doesn't support context manager protocol. Changing this could break existing code.
Clear error handling - The context manager semantics (success = commit, exception = rollback) are obvious with staged_insert1.
Type safety - The staged context exposes .store() for object fields. A dedicated method can return a properly-typed StagedInsert object.

Staged inserts are limited to insert1 (one row at a time). Multi-row inserts are not supported for staged operations.

Basic Usage

# Stage an insert with direct object storage writes
with Recording.staged_insert1 as staged:
    # Set primary key values
    staged.rec['subject_id'] = 123
    staged.rec['session_id'] = 45

    # Create object storage directly using store()
    # Extension is optional - .zarr is conventional for Zarr arrays
    z = zarr.open(staged.store('raw_data', '.zarr'), mode='w', shape=(10000, 10000), dtype='f4')
    z[:] = compute_large_array()

    # Assign the created object to the record
    staged.rec['raw_data'] = z

# On successful exit: metadata computed, record inserted
# On exception: storage cleaned up, no record inserted
# Stored as: raw_data_Ax7bQ2kM.zarr

StagedInsert Interface

class StagedInsert:
    """Context manager for staged insert operations."""

    rec: dict[str, Any]  # Record dict for setting attribute values

    def store(self, field: str, ext: str = "") -> fsspec.FSMap:
        """
        Get an FSMap store for direct writes to an object field.

        Args:
            field: Name of the object attribute
            ext: Optional extension (e.g., ".zarr", ".hdf5")

        Returns:
            fsspec.FSMap suitable for Zarr/xarray
        """
        ...

    def open(self, field: str, ext: str = "", mode: str = "wb") -> IO:
        """
        Open a file for direct writes to an object field.

        Args:
            field: Name of the object attribute
            ext: Optional extension (e.g., ".bin", ".dat")
            mode: File mode (default: "wb")

        Returns:
            File-like object for writing
        """
        ...

    @property
    def fs(self) -> fsspec.AbstractFileSystem:
        """Return fsspec filesystem for advanced operations."""
        ...

Staged Insert Flow

┌─────────────────────────────────────────────────────────┐
│ 1. Enter context: create StagedInsert with empty rec    │
├─────────────────────────────────────────────────────────┤
│ 2. User sets primary key values in staged.rec           │
├─────────────────────────────────────────────────────────┤
│ 3. User calls store()/open() to get storage handles     │
│    - Path reserved with random token on first call      │
│    - User writes data directly via fsspec               │
├─────────────────────────────────────────────────────────┤
│ 4. User assigns object references to staged.rec         │
├─────────────────────────────────────────────────────────┤
│ 5. On context exit (success):                           │
│    - Compute metadata (size, hash, item_count)          │
│    - Execute database INSERT                            │
├─────────────────────────────────────────────────────────┤
│ 6. On context exit (exception):                         │
│    - Delete any written data                            │
│    - Re-raise exception                                 │
└─────────────────────────────────────────────────────────┘

Zarr Example

import zarr
import numpy as np

# Create a large Zarr array directly in object storage
with Recording.staged_insert1 as staged:
    staged.rec['subject_id'] = 123
    staged.rec['session_id'] = 45

    # Create Zarr hierarchy directly in object storage
    # .zarr extension is optional but conventional
    root = zarr.open(staged.store('neural_data', '.zarr'), mode='w')
    root.create_dataset('timestamps', data=np.arange(1000000))
    root.create_dataset('waveforms', shape=(1000000, 82), chunks=(10000, 82))

    # Write in chunks (streaming from acquisition)
    for i, chunk in enumerate(data_stream):
        root['waveforms'][i*10000:(i+1)*10000] = chunk

    # Assign to record
    staged.rec['neural_data'] = root

# Record automatically inserted with computed metadata
# Stored as: neural_data_kM3nP2qR.zarr

Multiple Object Fields

with Recording.staged_insert1 as staged:
    staged.rec['subject_id'] = 123
    staged.rec['session_id'] = 45

    # Write multiple object fields - extension optional
    raw = zarr.open(staged.store('raw_data', '.zarr'), mode='w', shape=(1000, 1000))
    raw[:] = raw_array

    processed = zarr.open(staged.store('processed', '.zarr'), mode='w', shape=(100, 100))
    processed[:] = processed_array

    staged.rec['raw_data'] = raw
    staged.rec['processed'] = processed

# Stored as: raw_data_Ax7bQ2kM.zarr, processed_pL9nR4wE.zarr

Comparison: Copy vs Staged Insert

Aspect	Copy Insert	Staged Insert
Data location	Must exist locally first	Written directly to storage
Efficiency	Copy overhead	No copy needed
Use case	Small files, existing data	Large arrays, streaming data
Cleanup on failure	Orphan possible	Cleaned up
API	`insert1({..., "field": path})`	`staged_insert1` context manager
Multi-row	Supported	Not supported (insert1 only)

Transaction Handling

Since storage backends don't support distributed transactions with MySQL, DataJoint uses a copy-first strategy.

Insert Transaction Flow

┌─────────────────────────────────────────────────────────┐
│ 1. Validate input and generate storage path with token  │
├─────────────────────────────────────────────────────────┤
│ 2. Copy file/folder to storage backend                  │
│    └─ On failure: raise error, INSERT not attempted     │
├─────────────────────────────────────────────────────────┤
│ 3. Compute hash and build JSON metadata                 │
├─────────────────────────────────────────────────────────┤
│ 4. Execute database INSERT                              │
│    └─ On failure: orphaned file remains (acceptable)    │
├─────────────────────────────────────────────────────────┤
│ 5. Commit database transaction                          │
│    └─ On failure: orphaned file remains (acceptable)    │
└─────────────────────────────────────────────────────────┘

Failure Scenarios

Scenario	Result	Orphaned File?
Copy fails	Clean failure, no INSERT	No
DB insert fails	Error raised	Yes (acceptable)
DB commit fails	Error raised	Yes (acceptable)

Orphaned Files

Orphaned files (files in storage without corresponding database records) may accumulate due to:

Failed database inserts after successful copy
Process crashes
Network failures

This is acceptable because:

Random tokens prevent collisions with future inserts
Orphaned files can be identified by comparing storage contents with database records
A separate cleanup procedure removes orphaned files during maintenance

Orphan Cleanup Procedure

Orphan cleanup is a separate maintenance operation that must be performed during maintenance windows to avoid race conditions with concurrent inserts.

# Maintenance utility methods
schema.file_storage.find_orphaned()     # List files not referenced in DB
schema.file_storage.cleanup_orphaned()  # Delete orphaned files

Important considerations:

Should be run during low-activity periods
Uses transactions or locking to avoid race conditions with concurrent inserts
Files recently uploaded (within a grace period) are excluded to handle in-flight inserts
Provides dry-run mode to preview deletions before execution

Fetch Behavior

On fetch, the object type returns a handle (ObjectRef object) to the stored content. The file is not copied - all operations access the storage backend directly.

record = Recording.fetch1()
file_ref = record["raw_data"]

# Access metadata (no I/O)
print(file_ref.path)           # Full storage path
print(file_ref.size)           # File size in bytes
print(file_ref.hash)           # Content hash (if computed) or None
print(file_ref.ext)            # File extension (e.g., ".dat") or None
print(file_ref.is_dir)         # True if stored content is a folder

# Read content directly from storage backend
content = file_ref.read()      # Returns bytes (files only)

# Open as fsspec file object (files only)
with file_ref.open() as f:
    data = f.read()

# List contents (folders only)
contents = file_ref.listdir()  # Returns list of relative paths

# Access specific file within folder
with file_ref.open("subdir/file.dat") as f:
    data = f.read()

No Automatic Download

Unlike attach@store, the object type does not automatically download content to a local path. Users access content directly through the ObjectRef handle, which streams from the storage backend.

For local copies, users explicitly download:

# Download file to local destination
local_path = file_ref.download("/local/destination/")

# Download specific file from folder
local_path = file_ref.download("/local/destination/", "subdir/file.dat")

Implementation Components

1. Settings Extension (`settings.py`)

New ObjectStorageSettings class:

class ObjectStorageSettings(BaseSettings):
    """Object storage configuration for object columns."""

    model_config = SettingsConfigDict(
        env_prefix="DJ_OBJECT_STORAGE_",
        extra="forbid",
        validate_assignment=True,
    )

    project_name: str | None = None  # Must match store metadata
    protocol: Literal["object", "s3", "gcs", "azure"] | None = None
    location: str | None = None
    bucket: str | None = None
    endpoint: str | None = None
    partition_pattern: str | None = None
    token_length: int = Field(default=8, ge=4, le=16)
    access_key: str | None = None
    secret_key: SecretStr | None = None

Add to main Config class:

object_storage: ObjectStorageSettings = Field(default_factory=ObjectStorageSettings)

2. Storage Backend (`storage.py` - new module)

StorageBackend class wrapping fsspec
Methods: upload(), download(), open(), exists(), delete()
Path generation with partition support

3. Type Declaration (`declare.py`)

Add OBJECT pattern: object$
Add to SPECIAL_TYPES
Substitute to JSON type in database

4. Schema Integration (`schemas.py`)

Associate storage backend with schema
Validate storage configuration on schema creation

5. Insert Processing (`table.py`)

New __process_file_attribute() method
Path generation using primary key and partition pattern
Upload via storage backend

6. Fetch Processing (`fetch.py`)

New ObjectRef class
Lazy loading from storage backend
Metadata access interface

7. ObjectRef Class (`objectref.py` - new module)

@dataclass
class ObjectRef:
    """Handle to a file or folder stored in the pipeline's storage backend."""

    path: str
    size: int
    hash: str | None           # content hash (if computed) or None
    ext: str | None            # file extension (e.g., ".dat") or None
    is_dir: bool
    timestamp: datetime
    mime_type: str | None      # files only, derived from ext
    item_count: int | None     # folders only
    _backend: StorageBackend   # internal reference

    # fsspec access (for Zarr, xarray, etc.)
    @property
    def fs(self) -> fsspec.AbstractFileSystem:
        """Return fsspec filesystem for direct access."""
        ...

    @property
    def store(self) -> fsspec.FSMap:
        """Return FSMap suitable for Zarr/xarray."""
        ...

    @property
    def full_path(self) -> str:
        """Return full URI (e.g., 's3://bucket/path')."""
        ...

    # File operations
    def read(self) -> bytes: ...
    def open(self, subpath: str | None = None, mode: str = "rb") -> IO: ...

    # Folder operations
    def listdir(self, subpath: str = "") -> list[str]: ...
    def walk(self) -> Iterator[tuple[str, list[str], list[str]]]: ...

    # Common operations
    def download(self, destination: Path | str, subpath: str | None = None) -> Path: ...
    def exists(self, subpath: str | None = None) -> bool: ...

    # Integrity verification
    def verify(self) -> bool:
        """
        Verify object integrity.

        For files: checks size matches, and hash if available.
        For folders: validates manifest (all files exist with correct sizes).

        Returns True if valid, raises IntegrityError with details if not.
        """
        ...

fsspec Integration

The ObjectRef provides direct fsspec access for integration with array libraries:

import zarr
import xarray as xr

record = Recording.fetch1()
obj_ref = record["raw_data"]

# Direct Zarr access
z = zarr.open(obj_ref.store, mode='r')
print(z.shape)

# Direct xarray access
ds = xr.open_zarr(obj_ref.store)

# Use fsspec filesystem directly
fs = obj_ref.fs
files = fs.ls(obj_ref.full_path)

Dependencies

New dependency: fsspec with optional backend-specific packages:

[project.dependencies]
fsspec = ">=2023.1.0"

[project.optional-dependencies]
s3 = ["s3fs"]
gcs = ["gcsfs"]
azure = ["adlfs"]

Comparison with Existing Types

Feature	`attach@store`	`filepath@store`	`object`
Store config	Per-attribute	Per-attribute	Per-pipeline
Path control	DataJoint	User-managed	DataJoint
DB column	binary(16) UUID	binary(16) UUID	JSON
Hidden tables	Yes (external)	Yes (external)	No
Backend	File/S3 only	File/S3 only	fsspec (any)
Partitioning	Hash-based	User path	Configurable
Metadata storage	External table	External table	Inline JSON
Deduplication	By content	By path	None

No Hidden Tables

A key architectural difference: the object type does not use hidden external tables.

The legacy attach@store and filepath@store types store a UUID in the table column and maintain a separate hidden ~external_* table containing:

File paths/keys
Checksums
Size information
Reference counts

The object type eliminates this complexity by storing all metadata inline in the JSON column. This provides:

Simpler schema - no hidden tables to manage or migrate
Self-contained records - all information in one place
Easier debugging - metadata visible directly in queries
No reference counting - each record owns its object exclusively

Legacy Type Deprecation

The existing attach@store and filepath@store types will be:

Maintained for backward compatibility with existing pipelines
Deprecated in future releases with migration warnings
Eventually removed after a transition period

New pipelines should use the object type exclusively.

Delete Behavior

When a record with a object attribute is deleted:

Database delete executes first (within transaction)
File delete is attempted after successful DB commit
File delete is best-effort - the delete transaction succeeds even if file deletion fails

Delete Transaction Flow

┌─────────────────────────────────────────────────────────┐
│ 1. Execute database DELETE                              │
├─────────────────────────────────────────────────────────┤
│ 2. Commit database transaction                          │
│    └─ On failure: rollback, files unchanged             │
├─────────────────────────────────────────────────────────┤
│ 3. Issue delete command to storage backend              │
│    └─ On failure: log warning, transaction still OK     │
└─────────────────────────────────────────────────────────┘

Stale Files

If file deletion fails (network error, permissions, etc.), stale files may remain in storage. This is acceptable because:

The database record is already deleted (authoritative source)
Random tokens prevent any collision with future inserts
Stale files can be identified and cleaned via orphan detection utilities

No Reference Counting

Each record owns its file exclusively. There is no deduplication or reference counting, simplifying delete logic.

Migration Path

Existing attach@store and filepath@store remain unchanged
object type is additive - new tables only
Future: Migration utilities to convert existing external storage

Future Extensions

Compression options (gzip, lz4, zstd)
Encryption at rest
Versioning support
Streaming upload for large files
Checksum verification on fetch
Cache layer for frequently accessed files
Parallel upload/download for large folders

This issue is stale because it has been open for 45 days with no activity.

this looks great! A few thoughts from a Zarr POV:

a nice thing about modern object storage like s3 is that there are many tools that can read / write to it. fsspec is 1 python library but there are alternatives, such as obstore, which in particular has some performance advantages over fsspec. Relying on fsspec today is fine, but I would explore making it an optional dependency and allow clients to provide their own interface to object storage. tldr: building around remote storage is a good idea, but building around fsspec is worthy of scrutiny. The proposed ObjectRef class depends on fsspec's filesystem abstraction. It might be more future-proof to separate the declaration of the object (the content in the db) from tools for accessing it (the connection to object storage).
a few thoughts on the JSON schema:
- size is a required field, but for large Zarr data this can take a very long time to calculate (minutes or hours). calculating this value might cause the client to be unresponsive long enough to break connections to external services like databases. so if size is required, then users should know that they have to calculate it prior to declaring the zarr data in the db. but it's also not clear why this field is necessary.
- hash is another required field, and this inherits the same concern as size for zarr data. without adding something like a merkle tree to zarr data, hashing it will be expensive. similarly, it's not clear why this field is necessary.
- mime_type is problematic for zarr, because zarr data is a mix of JSON documents (mime type application/json) and chunk files application/octet-stream). That's fine if the plan is to only use this field for data with a well-defined mime type.
- the utility of the ext field is unclear, especially in conjunction with the mime_type field. Is the goal to just track the literal suffix of a file? in this case, parsing the path field should be sufficient. But if the goal is to do more than string parsing, e.g. to encode a content type, then this field overlaps with mime_type (which can be OK), but the semantics should be made clearer.
- item_count can be expensive to calculate for zarr data. and it's not clear why this is necessary. A motivating example or use case would help illustrate what this field is for.
- is_dir: why is this field necessary? I think the use case should be stated. Object storage is generally just keys and objects, with no concept of directories.

some more comments about the storage schema:

project_name 	string 	Yes 	Unique project identifier
created 	string 	Yes 	ISO 8601 timestamp of store creation
format_version 	string 	Yes 	Store format version for compatibility
datajoint_version 	string 	Yes 	DataJoint version that created the store
database_host 	string 	No 	Database server hostname (for bidirectional mapping)
database_name 	string 	No 	Database name (for bidirectional mapping)

I don't think I understand the use case for this file. Something like this makes sense if multiple clients with different software versions are trying to consume the same data, and you need to guard naive clients against breaking changes to the data format. But my impression was that the object store is managed by a single database, in which case there's no need to guard against naive clients reading objects from the database. Maybe an explanation for the use case would help me understand.

a question about the design decision to use just 1 object storage backend -- how would the pipeline ingest data stored on a different cloud backend? for a janelia connectomics effort, some assets were stored locally, but other assets were generated by collaborators and stored on google cloud. I suspect multidisciplinary projects would benefit from referring to data stored on any cloud backend. But the proposed design makes this impossible, which seems like a shortcoming.

{
    "database.host": "localhost",
    "database.user": "datajoint",

    "object_storage.project_name": "my_project",
    "object_storage.protocol": "s3",
    "object_storage.endpoint": "s3.amazonaws.com",
    "object_storage.bucket": "my-bucket",
    "object_storage.location": "my_project",
    "object_storage.partition_pattern": "{subject_id}/{session_id}"
}

@d-v-b

@d-v-b Thank you for the thorough review from a Zarr perspective! Excellent points. Let's address each:

`fsspec` Abstraction

You're right that coupling tightly to fsspec limits flexibility. The current ObjectRef directly exposes fs and store properties that return fsspec types. A more future-proof design would:

Separate declaration from access: The JSON schema stored in the DB (path, size, metadata) is pure data. The ObjectRef class could be a thin wrapper that accepts a pluggable "accessor" interface.
Make fsspec optional: The accessor could default to fsspec but allow alternatives like obstore. Something like:

# Current (tightly coupled)
obj_ref.store  # Returns fsspec.FSMap

# Proposed (pluggable)
obj_ref.get_store(backend="fsspec")  # Default
obj_ref.get_store(backend="obstore")  # Alternative

We could define a simple protocol/interface that both fsspec and obstore adapters implement.

JSON Schema Field Requirements

Looking at the current spec, I realize several fields should be clarified or made optional for Zarr compatibility:

size should be optional. You're absolutely right - for large Zarr stores, calculating total size is expensive (requires listing all chunks and summing sizes).

I will make the following changes:

Make size optional (nullable)
For staged inserts, default to null unless user explicitly requests size calculation
Document that size calculation may be slow for large hierarchies

hash - Already Optional. Looking at the spec: hash is required in the schema but explicitly allows null: "Content hash with algorithm prefix, or null". The spec also states: "Staged inserts never compute hashes". So this is already addressed - the field exists but null is the expected value for Zarr.

mime_type - Already Optional.
This field is marked optional in the spec. For Zarr, simply omit it. Agreed that a single MIME type doesn't make sense for a mixed-content store.

ext - Clarify Purpose. The intent is purely for tooling hints (like displaying appropriate icons or suggesting applications). For Zarr, .zarr is useful because:

Tools recognize it as a Zarr store
It's conventional in the ecosystem
Doesn't claim anything about internal content types

Not meant to overlap with mime_type semantically - it's the literal suffix. I'll clarify this in the docs.

item_count - Should be Optional.. You're right that counting items in a Zarr store is expensive. Looking at the spec, this is already marked optional but the folder example includes it. I will document explicitly that item_count may be null for large hierarchies. For staged inserts, default to null. Only compute when explicitly requested.

is_dir - Semantic Distinction.
While object storage doesn't have true directories, the field serves a semantic purpose for the API:

is_dir=false: Single blob → use read(), open()
is_dir=true: Key prefix (Zarr store, folder structure) → use listdir(), walk(), store property
For Zarr, is_dir=true indicates "this is a collection of related keys" which affects how the API behaves. It's more about the logical structure than the storage model.

if you identify stored objects by a full URL (e.g., s3://<bucket>/<path>, file://<path>, etc) instead of just a path, then you can support multiple storage backends and remove the need to define the bucket in the configuration. If you give every datajoint project its own globally unique identifier, then you can have multiple projects use the same storage backend without conflicts. This would make things much simpler IMO.

Assuming support for multiple stores for object-augmened schemas, since datajoint decides on where the data goes in a deterministic way, then we need to be able to configure a formula that maps specific tables/schemas to various backends.

Object-augmented Schema

Store Configuration (client settings) MUST NOT be changed once a project has been instantiated!!
in Store Metadata Schema, why do we need database_name (is this schema name?) - for bidirectional mapping?
On the folder manifest , is there still a path, pointing to the path of the folder (and then all of the files can have their full paths build from there)?
query view shows if the object is a file (with extension) or a folder - e.g. instead of =BLOB=, we show =OBJ[.bin]= or =OBJ[folder]= (or something like that)
No Deduplication - this means storing nested subfolders as different "object" will explode the storage size (nested duplication)
stage_insert will compute the hash or not? the specs has conflicting info here
copy_insert will compute the hash upon successful insert - why is the hash needed? is this configurable?
- can we turn off or use a more efficient hashing approach (e.g. at most N bytes - at beginning, end and somewhere in the middle of the file) - we can call this partial_hash (so both hash and partial_hash are optional)
- valid for file only, not folder
- hash - full hash (for small files) - default True for copy_insert
- partial_hash - default False, if set to True, then hash is switched to False. This must be a decision at the attribute level (how should we set this? at table definition or a kwarg for insert? more to think about here)
Orphan Cleanup Procedure
a. love this!
b. schema.file_storage -> should it be schema.object_storage?
c. why Should be run during low-activity periods? what race condition? the orphaned files are all with random tokens, thus no likelihood of collisions or race condition, right?
d. what is this schema.file_storage ? is this another hidden table similar to schema.externals[store_name]?

@dimitri-yatsenko

@dimitri-yatsenko , I've been raising similar arguments for multiple stores support as @d-v-b has above. I can see so many use cases that would benefit from this. Of course this raises the complexity significantly, but that's why DataJoint can offer unique capability

Object-Augmented Schema - Response to Feedback

1. Store Configuration Immutability

Agreed. Once a project has been instantiated (i.e., datajoint_store.json has been created), the following settings MUST NOT be changed:

object_storage.project_name
object_storage.protocol
object_storage.bucket
object_storage.location
object_storage.partition_pattern

Changing these settings after objects have been stored will result in broken references—existing paths stored in the database will no longer resolve to valid storage locations.

We will add a Configuration Immutability section to the spec with this warning.

2. `database_name` in Store Metadata

The database_name field exists for DBMS platforms that support multiple databases on a single server (e.g., PostgreSQL). This field identifies which database the object store belongs to.

The object storage configuration is shared across all schemas comprising the pipeline—it's a pipeline-level setting, not a per-schema setting. The database_name in the store metadata enables bidirectional mapping:

Forward: Client settings → object store location
Reverse: Object store metadata → originating database

This is informational only (not enforced at runtime) but helps administrators trace orphaned stores back to their databases.

3. Folder Manifest Path Structure

Yes, the folder's root path is stored in the JSON metadata (path field), and the manifest contains relative paths within the folder:

JSON metadata (in database):

{
    "path": "schema/Table/objects/pk=123/field_token/",
    "is_dir": true,
    ...
}

Manifest file (alongside folder in object store):

{
    "files": [
        {"path": "file1.dat", "size": 1234},
        {"path": "subdir/file2.dat", "size": 5678}
    ]
}

Full paths are constructed by combining: {folder_path}/{relative_path}

The manifest is stored alongside the folder (not inside) to avoid polluting folder contents—important for formats like Zarr that have strict directory structure requirements.

4. Query View Display Format

We will implement informative display for object columns:

Stored Value	Display
File with extension	`=OBJ[.dat]=`
File without extension	`=OBJ[file]=`
Zarr directory	`=OBJ[.zarr]=`
Folder without extension	`=OBJ[folder]=`

For accessing the raw JSON metadata, the ObjectRef class will provide a method rather than a fetch kwarg (since fetch1 retrieves multiple fields, directing a flag to a specific field is complicated):

record = Recording.fetch1()
obj_ref = record['raw_data']

# Access raw JSON metadata via ObjectRef method
metadata = obj_ref.to_dict()
# Returns: {"path": "...", "size": 12345, "hash": null, ...}

5. No Deduplication

The no-deduplication design is intentional for simplicity. Users working with object storage already understand that each stored object is independent. No additional warning needed in the spec.

6. Staged Insert Hash Computation

Staged inserts do NOT compute hashes. The spec is clear on this (but we'll verify no conflicting text exists):

"Staged inserts never compute hashes — data is written directly to storage without a local copy to hash."

For staged inserts:

hash is always null
Data flows directly to storage via fsspec
Re-reading all data after write to compute hash would be prohibitively expensive (especially for Zarr stores with millions of chunks)

7 & 8. Hash Configuration

We will implement explicit control per insert rather than implicit size-based rules:

# Default: no hash computed (fast)
Recording.insert1({..., "raw_data": "/path/to/file.dat"})

# Explicit hash request
Recording.insert1({..., "raw_data": "/path/to/file.dat"}, hash="sha256")

# Other supported algorithms
Recording.insert1({..., "raw_data": "/path/to/file.dat"}, hash="md5")
Recording.insert1({..., "raw_data": "/path/to/file.dat"}, hash="xxhash")

Design decisions:

Default: hash=None (no hash computed)
Explicit opt-in: User specifies algorithm when hash is desired
Files only: Hash applies to files, not folders (folders use manifests)
Staged inserts: Hash is always null regardless of kwarg

Rationale for explicit-only:

No implicit rules based on file size
User controls performance vs. integrity tradeoff
Simple, predictable behavior

Future consideration: Partial hashing (sampling beginning/middle/end) is an interesting optimization but adds complexity. We'll defer this to a future version after observing real usage patterns.

9. Orphan Cleanup Procedure

a. Glad you like it!

b. Naming: `schema.object_storage`

Agreed. We'll use schema.object_storage to match the type name:

schema.object_storage.find_orphaned()
schema.object_storage.cleanup_orphaned()

c. Race Condition Clarification

You're right that random tokens prevent filename collisions. However, there's a race condition with in-flight inserts:

Timeline:
1. Insert starts: file copied to storage with token Ax7bQ2kM
2. Orphan cleanup runs: lists storage, queries DB for references
3. File Ax7bQ2kM not yet in DB (INSERT not committed)
4. Cleanup identifies Ax7bQ2kM as orphan → DELETES it
5. Insert commits: DB now references deleted file!

Solution: Grace period. Only consider files orphaned if older than N minutes:

def find_orphaned(self, grace_period_minutes: int = 30):
    """
    Find orphaned files, excluding files newer than grace period.
    
    Files created within the grace period are assumed to be in-flight
    inserts and are not considered orphans.
    """

The "low-activity period" recommendation is a secondary precaution—with a proper grace period, cleanup is safe anytime, but running during low activity reduces the number of in-flight operations to reason about.

d. What is `schema.object_storage`?

Not a hidden table. Unlike attach@store which uses ~external_* tables, the object type has no hidden tables.

schema.object_storage is a utility object (not a table) that provides administrative methods:

Method	Description
`find_orphaned(grace_period_minutes=30)`	List files in storage not referenced by any DB record
`cleanup_orphaned(dry_run=True)`	Delete orphaned files (dry_run previews deletions)
`verify_integrity()`	Check all referenced objects exist and match metadata
`stats()`	Storage usage statistics

Implementation: A helper class that queries the storage backend and compares against JSON metadata in table columns. No database tables are created.

Summary of Changes to Implement

Item	Change
1	Add "Configuration Immutability" section to spec
2	Clarify `database_name` is for multi-database DBMS; config is pipeline-level
3	No change needed (manifest structure is correct)
4	Implement `=OBJ[.ext]=` display; add `ObjectRef.to_dict()` method
5	No change needed
6	Verify no conflicting text about staged insert hashing
7-8	Document explicit `hash` kwarg; no implicit rules; defer partial_hash
9	Rename to `object_storage`; document grace period; clarify it's a utility not a table

@ttngu207

@ttngu207 Implemented here: d5439cf

@d-v-b

@d-v-b @ttngu207

Thanks for the thoughtful feedback. Let me explain the design rationale and address the access control concern.

What Object-Augmented Schema (AUS) Means

The object type implements a specific paradigm: the object store is a logical extension of the relational schema, not a reference to external data.

This means:

DataJoint owns the lifecycle — only DataJoint writes to the store; users have read access
Joint operations — insert/delete/fetch operate on database and object store as a unit
Browsable structure — paths reflect schema organization (project/schema/Table/objects/pk=val/...)
Unified integrity — we can verify the schema is complete by scanning one backend

This tight coupling is what distinguishes AUS from external references. The filepath@store type already exists for the multi-backend, user-managed-data use case.

Why Single Backend Matters for Integrity

If objects are distributed across multiple backends:

Concern	Single Backend	Multi-Backend
Integrity verification	Scan one backend	Query N backends, merge results
Orphan cleanup	One system	N systems, track table→backend mapping
Migration	Change config, paths unchanged	Mass URL updates in database
Availability	One system	Partial schema if one backend down

The schema's integrity becomes a distributed systems problem. With a single backend, we can answer "is this schema complete?" with one listing operation.

Access Control via Prefix-Based Policies

The private/public bucket use case can actually be solved with one bucket and prefix-based IAM policies:

s3://my-bucket/project/
├── lab_internal/              ← restricted IAM policy
│   └── InternalResults/
│       └── objects/...
└── publications/              ← public bucket policy
    └── PublishedDatasets/
        └── objects/...

Or table-level:

s3://my-bucket/project/myschema/
├── PrivateTable/objects/...   ← restricted policy on this prefix
└── PublicTable/objects/...    ← public policy on this prefix

Our deterministic path structure (schema/Table/objects/pk=val/...) naturally enables this. S3, GCS, and Azure all support prefix-based policies, so:

Schema-level access: Policy on project/schema_name/*
Table-level access: Policy on project/schema/TableName/*

This achieves the private/public separation without fragmenting the store.

Row-Level Access Control: A Future Direction

What object stores don't handle well today is row-level access (e.g., user can only access objects for records where user_id = their_id). This would require:

Per-object ACLs (doesn't scale)
Dynamically generated signed URLs based on DB permissions

This is a real gap, but it's better solved by projecting database permissions onto object access (e.g., DataJoint generates signed URLs based on query permissions) rather than fragmenting the store across backends.

We'll note this as a future enhancement direction.

Summary

AUS requires tight coupling — single backend ensures we can reason about integrity
Access control should happen at the storage layer — prefix-based policies enable schema/table-level control
Multi-backend is for external references — filepath@store already supports this
Row-level object access is a future enhancement, best solved by DB-permission-aware signed URLs

The full-URL approach would break migration (URLs baked into DB), lose the browsable structure benefit, and make integrity verification a distributed problem — trade-offs that don't align with AUS goals.

Does this address your concern, or is there a specific use case that prefix-based policies wouldn't cover?

I am not following the reasoning here. As long as each table that references storage defines a unique URL prefix for the stored rows, then integrity questions are resolved by inspecting only objects stored under keys relative to that prefix. This means you never need to list an entire bucket to validate the database schema. Instead, you need to list (or query some object relative to) exactly the unique URL prefixes defined for each table to validate the database schema.

This basically means there's no bucket-level semantics, which means it shouldn't matter if the objects described by rows in table A are stored in a different s3 bucket (or different cloud backend) from the objects described by rows in table B. The only thing that matters is that changes to table A and table B can be correctly translated to changes to backend A and backend B.

The points about reliability, using prefix-based bucket policies for managing public / private, etc, read like arguments in favor of using a single bucket. Maybe a single bucket is practically a good idea most of the time (although I don't really agree that prefix-based bucket policies are a good solution for partitioning public data from private data). That doesn't mean that datajoint should mandate that projects use a single bucket. From what I can tell the basic data model doesn't require it.

With a single backend, we can answer "is this schema complete?" with one listing operation.

Just to emphasize -- if people use Zarr in earnest, you may have buckets with hundreds of thousands or millions of objects. You do not want to execute a single listing operation on such a bucket. Instead, you want to list relative to specific prefixes, or avoid listing altogether and rely on other ways of indexing your bucket.

@d-v-b

@d-v-b Thanks for pushing on this—the design is now better for it.

We're adopting multi-store support. The spec now allows a default store plus named secondary stores:

@schema
class Recording(dj.Manual):
    definition = """
    subject_id : int
    ---
    raw_data : object           # default store
    published : object@public   # named 'public' store
    """

Key design points:

All stores follow OAS principles. Named stores aren't "second-class"—they get the same deterministic path structure, same lifecycle ownership, same cleanup procedures. This maintains the integrity guarantees while enabling separate buckets for different access policies.
Both URL and path are stored. The JSON metadata now includes:
- url: Full URI (e.g., s3://bucket/path/to/object.dat)
- path: Relative path within the store
- store: Store name (or null for default)
This gives robustness (URL is self-contained) plus flexibility for migrations.
Efficient Zarr listing via delimiters. For cleanup, we use Delimiter='/' in S3 list operations to get only root object names, not millions of chunks:
```
s3.list_objects_v2(Bucket=bucket, Prefix='...', Delimiter='/')
# Returns: ['data_Ax7b.zarr/'] not millions of chunk keys
```
Per-store orphan cleanup. Each store is cleaned independently. Grace period handles race conditions with in-flight inserts.
Credential management is platform-side. Stores are configured in datajoint.json; credentials are managed at the platform level (IAM roles, environment, credential files). Same pattern as database access control.

This should address the need for separate buckets (public vs private, compliance requirements, etc.) while maintaining the integrity guarantees that distinguish object from filepath@store.

Implemented in DataJoint 2.0

Object-augmented schemas are fully implemented via the <object> storage type:

Features implemented:

Automatic path generation: Based on schema/table/primary key
File and folder support: Store individual files or entire directories
Immutability contract: Objects immutable after finalization
Two insert modes: Copy (small files) and Staged (large objects, Zarr)
ObjectRef handle: .read(), .open(), .download(), .listdir(), .walk()
Garbage collection: dj.gc.collect() removes orphaned objects
fsspec backends: Local, S3, GCS, Azure supported

Usage:

@schema
class Recording(dj.Imported):
    definition = """
    -> Session
    ---
    data_file : <object>
    """

# Insert
self.insert1({**key, 'data_file': ObjectRef.from_file('data.nwb')})

# Fetch and access
rec = Recording.fetch1()
rec['data_file'].download('/local/path/')

See src/datajoint/objectref.py and src/datajoint/storage.py.

FEAT: Object-augmented schemas -- File Type #1259

Description