Skip to content

Workflows: create a test case from a production chat turn/trace (with provenance + retention handling) #88

Open

Description

@prometheusgr

Follow-up to #80. Depends on slice 4 (test cases + eval runs).

Context

The most valuable test cases come from real failures: a bad answer seen in a Phoenix trace should become a test case in one step. But that copies real user content (potentially sensitive) out of conversation_threads, which has a retention window, into test cases, which are kept long-term and aren't covered by the retention sweep.

Scope

  1. "Create test case from this turn": from a chat turn (via its workflow version ID / trace link), an admin creates a test case pre-filled with the input, and edits it before saving.
  2. Explicit, audited action (workflow.test_case_created_from_trace), recording the source thread/turn and the admin.
  3. Provenance label on the test case (authored vs. from_production), shown in the UI.
  4. Retention decision, documented in docs/SECURITY.md's retention table: how long from_production test cases are kept, and whether they're purged when their source user is deleted.

User-visible signal

A "Create test case" action in the admin draft/eval screens, and a provenance badge on each test case.

RBAC

Admin feature. An admin can see other users' conversation content through this path, which is a real privacy expansion. State it explicitly in the PR and in docs/SECURITY.md.

Definition of Done

  • Action, audit event, provenance label: TDD.
  • Retention/privacy decision documented.
  • Standard merge gate from AGENTS.md's Definition of Done applies in full.

Activity

  1. added
    type:featureBrand-new capability that doesn't exist today
    effort:MMedium: one layer (backend or frontend), a few days
    area:retentionData retention windows and purge behavior
    area:agent-toolsLangGraph agents and MCP tool integrations
    on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:agent-toolsLangGraph agents and MCP tool integrationsarea:retentionData retention windows and purge behavioreffort:MMedium: one layer (backend or frontend), a few dayspriority:P2Valuable, not urgenttype:featureBrand-new capability that doesn't exist today

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions