You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The production ci-failure-fix and ci-failure-scan-feedback workflows require issue, pull request, and comment content to be read through the integrity-gated GitHub MCP tools. Their Vally eval stimuli and tool-call graders still instruct and reward shell gh reads instead.
This means /ci-fix eval and /ci-feedback eval exercise a different retrieval path than production and can pass without validating the workflows' MCP-only content-read policies. This mismatch surfaced while updating the shared MCP authentication plumbing in #132619; that PR intentionally aligns only the ci-failure-scan eval.
Proposed work
Update the fixer eval stimulus to discover KBEs with scoped search_issues calls and read issue bodies/comments with issue_read.
Update the feedback eval stimulus to use scoped search_issues and search_pull_requests, then read issue/PR bodies and comments through issue_read and pull_request_read.
Retain shell gh only for workflow metadata and repository-owned artifact enumeration explicitly permitted by each production prompt.
Update tool-call graders to require the canonical github-* MCP calls and reject shell-based GitHub content reads.
Run the Docker-backed live evals to confirm both workflows satisfy the updated contracts.
Update .github/workflows/evals/README.md to describe the MCP-backed behavior accurately.
Problem
The production
ci-failure-fixandci-failure-scan-feedbackworkflows require issue, pull request, and comment content to be read through the integrity-gated GitHub MCP tools. Their Vally eval stimuli and tool-call graders still instruct and reward shellghreads instead.This means
/ci-fix evaland/ci-feedback evalexercise a different retrieval path than production and can pass without validating the workflows' MCP-only content-read policies. This mismatch surfaced while updating the shared MCP authentication plumbing in #132619; that PR intentionally aligns only theci-failure-scaneval.Proposed work
search_issuescalls and read issue bodies/comments withissue_read.search_issuesandsearch_pull_requests, then read issue/PR bodies and comments throughissue_readandpull_request_read.ghonly for workflow metadata and repository-owned artifact enumeration explicitly permitted by each production prompt.github-*MCP calls and reject shell-based GitHub content reads..github/workflows/evals/README.mdto describe the MCP-backed behavior accurately.Note
This issue was created with GitHub Copilot.