Skip to content

Evals for the setkernel plugin with claude plugin eval #32

Open

Description

@realRaghavGupta

Why

Plugin changes ship without regression tests. claude plugin eval runs cases with and without the plugin, with free graders (regex, tool_used, tool_order, file_exists) and a --threshold (https://code.claude.com/docs/en/plugin-evals.md).

Do

  • claude plugin eval init; 5 cases from real failures for verify, ship-pr and review-blockers.
  • Check whether eval runs on a seat login (unverified; the CI docs name API keys). If it does, run it on the canary VM before a plugin change merges.
  • Add 3 scored past PRs as few-shot examples to the reviewer agent (Anthropic: a separate, calibrated evaluator).

Done when

A plugin change that breaks a case fails before merge.

Roadmap: wave 5 of the next-level roadmap (https://claude.ai/artifact/Rsg4uPgpwSp2DDTM2iVK78), tracked in setkernel/hq#273.
Released by hand when wave 4 (setkernel/hq#272) closes (the release watcher reads hq only).

— Claude Lead

Activity

  1. added
    model:opusRun this task's main step at this setting (sk-agent, hq#174)
    effort:highRun this task's main step at this setting (sk-agent, hq#174)
    on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort:highRun this task's main step at this setting (sk-agent, hq#174)model:opusRun this task's main step at this setting (sk-agent, hq#174)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions