IMPR: split unit/integration test

Improvement Request

Problem

Currently, datajoint-python pytest runs unit and integration test by default, which requires a DB

pytest --cov-report term-missing --cov=datajoint tests
# trying to connect to a DB, otherwise integration test would fail

so it actually needs runs in docker that comes with a DB container

docker compose --profile test up --quiet-pull --build --exit-code-from djtest djtest

Requirements

N/A

Justification

We should add unit test check in pre-commit locally, and let the CI to run integration test with docker

Alternative Considerations

N/A

Related Errors

N/A

Please include steps to reproduce provided errors as follows:
N/A

Screenshots

N/A

Additional Research and Context

N/A

DataJoint 2.0 Implementation

This improvement is implemented in DataJoint 2.0. Tests are now split into separate directories with independent CI jobs.

Test Structure

tests/
├── unit/                    # No database required
│   ├── test_codecs.py
│   ├── test_condition.py
│   ├── test_hash.py
│   ├── test_pk_rules.py
│   └── test_settings.py
├── integration/             # Requires MySQL/MinIO (via testcontainers)
│   ├── test_blob.py
│   ├── test_fetch.py
│   └── ... (40+ test files)
└── conftest.py

CI Workflow

The GitHub Actions workflow (.github/workflows/test.yaml) now has two separate jobs:

jobs:
  # Full integration tests with testcontainers
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: prefix-dev/setup-pixi@v0.9.3
      - run: pixi run -e test test-cov

  # Unit tests only - faster feedback, no containers
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: prefix-dev/setup-pixi@v0.9.3
      - run: pixi run -e test pytest tests/unit -v

Local Development

Run unit tests locally without any database:

# With pixi (recommended)
pixi run -e test pytest tests/unit/

# With pip
pytest tests/unit/

Run full test suite (testcontainers auto-manages MySQL/MinIO):

pixi run test

Benefits

Aspect	Before	After
Local unit tests	Required DB connection	No dependencies
CI feedback	Single slow job	Fast unit-tests job + full integration
Pre-commit potential	Not feasible	Can add unit tests to pre-commit

The unit tests complete in ~30 seconds in CI, providing fast feedback before the full integration suite (~2-3 minutes).

Closing as resolved in DataJoint 2.0. Tests are now split into tests/unit/ and tests/integration/ with separate CI jobs.

@dimitri-yatsenko

@dimitri-yatsenko Might want to consider also adding it to pre-commit, so that the unit test can fail before the CI failed to save some time.

feat: add unit tests to pre-commit hooks

@drewyangdev

@drewyangdev Good suggestion! Added unit tests to pre-commit in commit f195110:

- repo: local
  hooks:
    - id: unit-tests
      name: unit tests
      entry: pytest tests/unit/ -v --tb=short
      language: system
      pass_filenames: false
      always_run: true
      stages: [pre-commit]

Now unit tests run automatically on git commit, catching issues before CI.

Navigation Menu

Repository navigation

IMPR: split unit/integration test #1211

Improvement Request

Problem

Requirements

Justification

Alternative Considerations

Related Errors

Screenshots

Additional Research and Context

DataJoint 2.0 Implementation

Test Structure

CI Workflow

Local Development

Benefits

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

IMPR: split unit/integration test #1211

Description

Improvement Request

Problem

Requirements

Justification

Alternative Considerations

Related Errors

Screenshots

Additional Research and Context

Activity

dimitri-yatsenko commented on Jan 8, 2026

DataJoint 2.0 Implementation

Test Structure

CI Workflow

Local Development

Benefits

dimitri-yatsenko commented on Jan 8, 2026

drewyangdev commented on Jan 8, 2026

dimitri-yatsenko commented on Jan 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions