PR review `commit_id` is the head at post time, not the commit the agent reviewed

[dsfaccini]

This issue was posted by Claude Code using claude-opus-5 on behalf of David (@dsfaccini), a Pydantic AI maintainer. He asked for it to be filed and has reviewed the conclusion but not this text.

Summary

submit-pull-request-review posts with commit_id resolved from a live pulls.get in the safe_outputs job, not from the commit the agent actually reviewed. When the PR head moves during the agent run, the posted review is attributed to a commit whose contents the agent never saw.

Where

actions/setup/js/submit_pr_review.cjs → setReviewContextOnBuffer. Under workflow_run there is no context.payload.pull_request, so usePayloadPR is false and it takes the fetch branch:

const { data: fetchedPR } = await githubClient.rest.pulls.get({
  owner: repoParts.owner,
  repo: repoParts.repo,
  pull_number: prNum,
});

That result flows into actions/setup/js/pr_review_buffer.cjs:

commit_id: pullRequest.head.sha,

safe_outputs runs after agent and detection, so pulls.get returns whatever the head is at post time.

Observed

A workflow_run-triggered reviewer on pydantic/pydantic-ai (run 30368354427), all times UTC:

Time	Event
14:26:31	our eligibility job resolves the PR head: ac23489
14:27:12	agent checks out ac23489
14:28:28–14:34:08	agent job reads the code and writes its findings
14:33:58	a new commit 4f8f1bf is pushed — during the agent job
14:35:52–14:36:09	safe_outputs runs pulls.get → head is now 4f8f1bf → posts with commit_id: 4f8f1bf

The resulting comment quotes a source line that exists in ac23489 and not in 4f8f1bf. It reads as a fabrication. A maintainer's agent dismissed it as invalid on exactly that basis, and we had to retract that dismissal after reading the run's job timings.

Why the correct sha matters

The REST docs describe precisely the behavior we want, and it only works if the reviewed sha is passed:

commit_id — "The SHA of the commit that needs a review. Not using the latest commit SHA may render your review comment outdated if a subsequent commit modifies the line you specify as the position. Defaults to the most recent commit in the pull request when you do not specify a value."
— https://docs.github.com/en/rest/pulls/reviews

Pass the reviewed sha and GitHub marks the review outdated on its own when head has moved. Pass the live head and a stale review is indistinguishable from a hallucinated one.

Requested change

A config option on submit-pull-request-review and create-pull-request-review-comment to pin the commit, e.g.:

safe-outputs:
  submit-pull-request-review:
    commit-id: ${{ needs.eligibility.outputs.head_sha }}

gh aw compile currently rejects commit-id, commit_id, commit, sha, and head-sha on that output, so there is no way to express this today.

A knob rather than inference is what we need, because gh-aw cannot infer the sha in this setup: under workflow_run, gh-aw's own "Checkout PR branch" step is gated on github.event.pull_request / aw_context and no-ops, so the workflow does its own checkout in pre-agent-steps from a sha its own job resolved. Only the workflow author knows it.

Inferring it where gh-aw does own the checkout (record the checked-out sha in the agent job, thread it to safe_outputs) would be a good default on top, but it would not cover this case.

Workarounds considered and ruled out

• safe-outputs.if — not a supported key (gh aw compile: Unknown property: if. Did you mean 'env'?).
• Gating via a custom job in safe-outputs.needs — the generated safe_outputs.if is (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success'. !cancelled() still runs when a dependency fails or is skipped, so a gate job cannot block the post.
• Adding a step to safe_outputs — that job is entirely generated; pre-steps / pre-agent-steps only reach the agent job.

That leaves only prompt-level mitigation (have the agent write the sha it read into the review body), which does not fix the attribution or the line anchoring.

Downstream tracking issue: pydantic/pydantic-ai#6788