Epic: configurable multi-agent workflows (declarative definitions, versioned store, audited publish, iteration loop) #80

Goal

Let a fork define multi-agent workflows (e.g. an interpreter agent → a worker agent → a reviewer agent that sends work back for fixes) declaratively, iterate on their prompts safely, and publish changes with a full audit trail — without writing LangGraph code for each new workflow.

This is template-level infrastructure: the building block every fork uses to compose its own enterprise agents, so auditability and change management are first-class, not bolt-ons.

Architecture decisions (from the design discussion)

Standard patterns, fixed primitives. Workflows are built from a small, code-defined set of primitives drawn from the established multi-agent patterns (prompt chaining, routing, evaluator–optimizer, parallelization): agent, sequence, route, review_loop, parallel. YAML composes primitives; it is never a general-purpose scripting language. A fork needing a new primitive adds it in code, with tests.

YAML compiles to LangGraph. Definitions are validated (Pydantic) at load/save time and compiled to a StateGraph, so the existing SqlAlchemyCheckpointSaver, Phoenix tracing, per-(user_id, thread_id) isolation, guardrails, and per-caller MCP token forwarding all apply unchanged.

Storage: Postgres is the system of record; git holds the built-in defaults. Options weighed: files baked into the image, ConfigMap, Postgres, files + DB overrides, MinIO, git (embedded repo or a git server with PRs).

• A running container's filesystem can't be the edit target (ephemeral, per-replica, no audit), so UI edits live in Postgres.
• Postgres wins for the fast iteration loop (instant drafts, run-a-draft, compare versions, FK from chat turns / eval runs / audit entries to a version) and keeps the publish + audit write in one transaction.
• MinIO was rejected as primary store (no transactions, no queryable metadata, author identity is only whatever the backend asserts) but is a good WORM archive target — see follow-up.
• Git was rejected as primary store for v1 (slow iteration loop, two-system atomicity, extra infra for air-gapped installs) but is the promotion target for proven workflows and the home of built-in defaults — see follow-ups.

Three roles, one new build:

Change management is configurable per deployment (#87), because forks run very different processes. Modes: direct (publish from the EAIStack UI; the default and the only v1 mode), git_advisory (publish in the UI and mirror to git for review), git_authoritative (the UI proposes a PR; a version is published only on merge), plus optional in-app two-person approval. Where the repo lives is configurable too: an existing external repo, or an optional in-stack Gitea/Forgejo. The mode is env/Helm deployment config, deliberately not a DB-editable admin setting: it is the change-control policy, so the admins it constrains must not be able to switch it off. The effective mode is shown read-only in the UI. Drafting and eval runs stay in EAIStack in every mode; only the publish step changes.

Invariants:

1. One source of truth per data type. Test cases and eval results live in EAIStack; Phoenix only displays eval runs as traces.
2. The workflow version ID travels everywhere: chat turn, Phoenix span, eval run, audit entry.
3. Nothing edited in Phoenix's playground is real until saved as an EAIStack draft — Phoenix cannot publish, so nothing bypasses the audit trail.
4. A newer built-in (git) default never silently overwrites a published admin change.

Iteration loop

 PHOENIX   1. Observe    bad answer in production traces (tagged with workflow version)
           2. Diagnose   which step failed
           3. Tinker     optional single-step replay in Phoenix playground
 EAISTACK  4. Capture    add the input as a test case (curated copy)
           5. Draft      edit prompt in a draft version (validated on save)
           6. Prove      run test cases on draft vs published; compare pass rates
           7. Publish    evidence-gated; audit_logs records who / diff / eval run
 GIT       8. Promote    (occasionally) export YAML -> PR -> becomes a built-in default

Slices

v1:

• Workflows 1/5: engine — primitives, YAML schema + compiler, tool registry, chat agent as built-in workflow #81 Engine: primitives, YAML schema + compiler, tool registry, chat agent as built-in workflow
• Workflows 2/5: reference workflows — review loop (evaluator–optimizer) and routing #82 Reference workflows: review loop + routing
• Workflows 3/5: versioned append-only workflow store + admin UI (draft/publish/rollback, audit, diff, version per chat turn) #83 Versioned append-only store + admin UI (auditability)
• Workflows 4/5: iteration loop — draft test chat, test cases, eval runs with pass rates, evidence-gated publish #84 Iteration loop: draft test chat, test cases, eval runs, evidence-gated publish
• Workflows 5/5: workflow selection in chat for end users #85 Workflow selection in chat (user feature)

Follow-ups:

• Workflows: export to YAML and 'newer built-in available' flow (git promotion path) #86 Export to YAML + "newer built-in available" (git promotion path)
• Workflows: configurable change management — direct / git-advisory / git-authoritative publish modes, two-person approval, external or in-stack git #87 Configurable change management: direct / git_advisory / git_authoritative + two-person approval; external repo or in-stack Gitea/Forgejo
• Workflows: create a test case from a production chat turn/trace (with provenance + retention handling) #88 Test case from a production turn/trace (provenance + retention)
• WORM archive of audit events and published configuration to an object-locked MinIO bucket #89 WORM archive of audit events + published config (MinIO object lock)
• Observability: put Phoenix behind authentication (Keycloak OIDC), restrict playground access #90 Phoenix behind Keycloak auth; restrict playground

Supersedes / relates to

• Supersedes Observability: Phoenix Prompt Hub for prompt storage/versioning and in-UI iteration #35 (Phoenix Prompt Hub as prompt storage) — decided: EAIStack's versioned store is authoritative; Phoenix is a viewer.
• Supersedes Observability: Phoenix Datasets/Experiments for prompt regression testing #36 (Phoenix Datasets/Experiments) — decided: test cases + eval runs live in EAIStack, traced to Phoenix for inspection.
• Relates to Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30 (evaluation hooks) — the eval-run model here is the substrate Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30's LLM-as-judge scorers plug into; judge-model design stays in Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30.
• Relates to Add a lightweight golden-set regression harness for agent/retrieval quality (CI-runnable, non-Phoenix) #41 (golden-set harness) — complementary; the chat workflow's test cases could later absorb Add a lightweight golden-set regression harness for agent/retrieval quality (CI-runnable, non-Phoenix) #41's golden set.
• Prerequisites for relying on Phoenix in this loop: Observability: configurable trace retention (session-only / N days / forever) #32 (trace retention), Observability: verify and enable TLS termination for the Phoenix Helm chart #33 (Phoenix TLS), plus Phoenix authentication (Observability: put Phoenix behind authentication (Keycloak OIDC), restrict playground access #90).

RBAC

• Defining/editing/publishing workflows: admin feature (require_admin). A separate workflow-admin role is a deferred option.
• Using a published workflow in chat: user feature.