You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evals for the setkernel plugin with claude plugin eval #32
Plugin changes ship without regression tests. claude plugin eval runs cases with and without the plugin, with free graders (regex, tool_used, tool_order, file_exists) and a --threshold (https://code.claude.com/docs/en/plugin-evals.md).
Do
claude plugin eval init; 5 cases from real failures for verify, ship-pr and review-blockers.
Check whether eval runs on a seat login (unverified; the CI docs name API keys). If it does, run it on the canary VM before a plugin change merges.
Add 3 scored past PRs as few-shot examples to the reviewer agent (Anthropic: a separate, calibrated evaluator).
Done when
A plugin change that breaks a case fails before merge.
Roadmap: wave 5 of the next-level roadmap (https://claude.ai/artifact/Rsg4uPgpwSp2DDTM2iVK78), tracked in setkernel/hq#273.
Released by hand when wave 4 (setkernel/hq#272) closes (the release watcher reads hq only).
Activity