Skip to content

Align CI fixer and feedback evals with GitHub MCP policies #132634

Description

@vitek-karas

Problem

The production ci-failure-fix and ci-failure-scan-feedback workflows require issue, pull request, and comment content to be read through the integrity-gated GitHub MCP tools. Their Vally eval stimuli and tool-call graders still instruct and reward shell gh reads instead.

This means /ci-fix eval and /ci-feedback eval exercise a different retrieval path than production and can pass without validating the workflows' MCP-only content-read policies. This mismatch surfaced while updating the shared MCP authentication plumbing in #132619; that PR intentionally aligns only the ci-failure-scan eval.

Proposed work

  • Update the fixer eval stimulus to discover KBEs with scoped search_issues calls and read issue bodies/comments with issue_read.
  • Update the feedback eval stimulus to use scoped search_issues and search_pull_requests, then read issue/PR bodies and comments through issue_read and pull_request_read.
  • Retain shell gh only for workflow metadata and repository-owned artifact enumeration explicitly permitted by each production prompt.
  • Update tool-call graders to require the canonical github-* MCP calls and reject shell-based GitHub content reads.
  • Run the Docker-backed live evals to confirm both workflows satisfy the updated contracts.
  • Update .github/workflows/evals/README.md to describe the MCP-backed behavior accurately.

Note

This issue was created with GitHub Copilot.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions