Skip to content

IMPR: Modernize Project Configuration Management #1257

New issue
New issue
Closed
Closed
IMPR: Modernize Project Configuration Management#1257
Assignees
dimitri-yatsenko
Labels
enhancementIndicates new improvementsstaleIndicates issues, pull requests, or discussions are inactive
Milestone
DataJoint 2.0

Description

@dimitri-yatsenko
dimitri-yatsenko
opened on Aug 18, 2025

Improvement Request

Problem

The current configuration system is nonstandard and lacks common features such as validation. Secrets and regular settings are managed jointly, which is not a best practice.

Requirements

  • Load settings hieararchically from project specific locations (. and up), user specific, and system wide.
  • Use pydantic to strictly validate all loaded settings against a predefined schema on startup, providing clear errors for any misconfiguration.
  • Supports secure management of sensitive data through a .gitignore'd secrets file and environment variables.
  • Ensure that environment variables have highest precedence, overriding configuration files
  • Clearly documented, explaining loading order, file structure, and validation rules.

Activity

  1. dimitri-yatsenko
    added
    enhancementIndicates new improvements
    triageIndicates issues, pull requests, or discussions need to be reviewed for the first time
    on Aug 18, 2025
  2. dimitri-yatsenko
    self-assigned this
    on Aug 18, 2025
  3. dimitri-yatsenko
    removed
    triageIndicates issues, pull requests, or discussions need to be reviewed for the first time
    on Aug 18, 2025
  4. dimitri-yatsenko
    added this to the DataJoint 2.0 milestone on Aug 18, 2025

  5. github-actions commented on Oct 3, 2025

    @github-actions
    github-actionsBot
    on Oct 3, 2025 – with GitHub Actions

    This issue is stale because it has been open for 45 days with no activity.

  6. github-actions
    added
    staleIndicates issues, pull requests, or discussions are inactive
    on Oct 3, 2025

  7. dimitri-yatsenko commented on Jan 9, 2026

    @dimitri-yatsenko
    dimitri-yatsenko
    on Jan 9, 2026
    MemberAuthor

    Implemented in DataJoint 2.0

    Configuration management has been modernized with Pydantic:

    Features implemented:

    • Pydantic validation: Type-safe settings with automatic validation
    • Hierarchical loading: Project → User → System → Defaults
    • Secrets separation: .secrets/datajoint.json for credentials (gitignored)
    • Environment variables: DJ_HOST, DJ_USER, DJ_PASS, etc. with highest precedence
    • Context manager: dj.config.override() for temporary settings
    • Template generation: dj.config.save_template() creates starter config

    Configuration files:

    project/
├── datajoint.json          # Non-sensitive settings (commit to git)
└── .secrets/
    └── datajoint.json      # Credentials (gitignored)

    Precedence (highest to lowest):

    1. Programmatic settings
    2. Environment variables
    3. .secrets/datajoint.json
    4. datajoint.json
    5. Defaults

    See src/datajoint/settings.py for implementation.

  8. dimitri-yatsenko
    closed this as completedon Jan 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementIndicates new improvementsstaleIndicates issues, pull requests, or discussions are inactive

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions