Skip to content

feat(providers): track Jev typed AI evaluation and its data-egress, failure, and release contract #397

Description

@Akshat0694

Is your feature request related to a problem? Please describe.
Tirith's current providers extract values from input documents. Semantic questions about a change need an explicit inference provider. Draft PR [#396](#396) implements stackguardian/jev but has no tracking issue and introduces a new trust boundary: a locally invoked policy can send input to an external model API.

This issue tracks that proposed capability and the decisions needed before shipping it. It does not treat the draft as production-ready or suggest that AI authoring skills require remote inference.

Describe the solution you'd like
Review and finish [#396](#396) as an explicitly enabled provider with a documented contract:

  • Typed noul, choice, and score answers, their ranges, and confidence semantics.

  • Exactly which input slice is transmitted, what happens when state_path is missing/misspelled, and how users can restrict/redact sensitive input before transmission.

  • A consistent distinction between an answer violating policy, low confidence, absent input, and a service/configuration error. Specify how error_tolerance affects enforcement; do not silently market tolerated outages as successful validation.

  • Consistent local-only claims across the README, landing/demo pages, docs and AI-facing content.

  • Bounded total execution time and responsive explicit execution in the Playground.

Describe alternatives you've considered
Keep deterministic JSON/Terraform checks for facts present in the input; use AI skills solely to author those checks. Alternatively keep semantic inference advisory until confidence, reproducibility, and error-handling behavior are established. The documentation should explain which mode a user is choosing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions