Skip to content

Commit a7651ff

Browse files
edburnsedburns
committed
shepherd-tasks-20260730-2115
1 parent 336e9fd commit a7651ff

3 files changed

Lines changed: 3876 additions & 0 deletions

File tree

‎1917-java-embed-rust-cli-runtime-remove-before-merge/shepherd-tasks-20260730-2115/20260730-2128-post-mortem.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Post-Mortem Report: Shepherd-Task Run for Issue [#2168](https://github.com/github/copilot-sdk/issues/2168)
2+
3+
**Campaign:** Java embedded runtime classifier work
4+
**Report generated:** 2026-07-30 21:28 UTC
5+
**Period covered:** 2026-07-30 21:15:53-21:28:30 UTC
6+
**Run directory:** `shepherd-tasks-20260730-2115`
7+
**Base branch:** `edburns/1917-java-embed-rust-cli-runtime-dd-3039924-agentic-run-02`
8+
**Caller exit code:** `1`
9+
10+
## Table of Contents
11+
12+
- [Section 1: Executive Summary](#section-1-executive-summary)
13+
- [Section 2: System Architecture](#section-2-system-architecture)
14+
- [2.1 Copilot Coding Agent (CCA)](#21-copilot-coding-agent-cca)
15+
- [2.2 Copilot Code Review Agent (CCRA)](#22-copilot-code-review-agent-ccra)
16+
- [2.3 Local Copilot CLI (Shepherd)](#23-local-copilot-cli-shepherd)
17+
- [Section 3: Per-Task Metrics](#section-3-per-task-metrics)
18+
- [3.1 — Issue #2168 / PR #2173](#31--issue-2168--pr-2173)
19+
- [Section 4: Aggregate Statistics](#section-4-aggregate-statistics)
20+
- [Section 5: AI Credits and Token Usage](#section-5-ai-credits-and-token-usage)
21+
- [Section 6: Wall-Clock Timeline](#section-6-wall-clock-timeline)
22+
- [Section 7: Failure Analysis](#section-7-failure-analysis)
23+
- [Section 8: Observations and Recommendations](#section-8-observations-and-recommendations)
24+
25+
---
26+
27+
## Section 1: Executive Summary
28+
29+
The single-task phase-2 shepherd run failed to merge [#2173](https://github.com/github/copilot-sdk/pull/2173), leaving [#2168](https://github.com/github/copilot-sdk/issues/2168) incomplete. The implementation fixes were already present in commit `bc0275f8`, both original Copilot review threads were resolved, and CI was green. The blocking event was operational: after the shepherd successfully requested a second Copilot review, no new review appeared within the configured 10-minute polling window.
30+
31+
| Metric | Value |
32+
|--------|-------|
33+
| Target tasks | 1 |
34+
| Completed and merged | 0/1 (0%) |
35+
| Failed or incomplete | 1/1 (100%) |
36+
| PRs touched | 1 ([#2173](https://github.com/github/copilot-sdk/pull/2173)) |
37+
| Run elapsed | 12m 37s |
38+
| Completed CCRA rounds observed | 1 |
39+
| CCRA comments observed | 2 |
40+
| Original comments resolved | 2/2 |
41+
| New reviews received during run | 0 |
42+
| Local CLI output tokens | 4,698 |
43+
| Local code changes during session | 0 files, 0 lines |
44+
| Final state | PR open and unmerged |
45+
46+
The caller reported exit code `1`, which correctly classifies the campaign outcome. The exported Copilot session's internal result record reports `exitCode: 0`; this reflects successful session export rather than successful shepherding and should not be used as the task outcome signal.
47+
48+
---
49+
50+
## Section 2: System Architecture
51+
52+
### 2.1 Copilot Coding Agent (CCA)
53+
54+
CCA produced the implementation PR [#2173](https://github.com/github/copilot-sdk/pull/2173), titled `[Java] Add PlatformDetector for runtime classifier selection`, targeting the requested base branch. No phase-1 artifact exists in this run directory, so CCA timing, token usage, and initial implementation metrics are unavailable.
55+
56+
### 2.2 Copilot Code Review Agent (CCRA)
57+
58+
CCRA completed one review before this run, at 20:12 UTC. It generated two comments:
59+
60+
1. A Linux classifier test depended on the host libc and could fail on musl.
61+
2. Malformed ELF program-header metadata could escape the documented fallback through an `ArrayIndexOutOfBoundsException`.
62+
63+
Both findings were addressed in `bc0275f8`, replied to, and marked resolved before the phase-2 session began. A required second review was requested during this run but did not arrive before the timeout.
64+
65+
### 2.3 Local Copilot CLI (Shepherd)
66+
67+
The local shepherd verified issue/PR linkage, confirmed the base branch and mergeability, inspected the existing CCRA review, confirmed all relevant CI checks were green, and verified that both review threads were resolved. It then attempted to request and await a second CCRA review. The session made no source changes because the fixes were already present.
68+
69+
---
70+
71+
## Section 3: Per-Task Metrics
72+
73+
### 3.1 — Issue [#2168](https://github.com/github/copilot-sdk/issues/2168) / PR [#2173](https://github.com/github/copilot-sdk/pull/2173)
74+
75+
| Metric | Value |
76+
|--------|-------|
77+
| Phase 1 duration | Not captured; no phase-1 artifact |
78+
| Phase 2 duration | 12m 37s |
79+
| PR state at start | Open, non-draft, mergeable |
80+
| Initial CCRA review | 1 round, 2 comments |
81+
| Initial review timestamp | 2026-07-30 20:12 UTC |
82+
| Comment fixes | Commit `bc0275f8`, present before session |
83+
| Threads resolved | 2/2 |
84+
| CI state | Green |
85+
| Re-review request | Successful with `gh` 2.96.0 and `@copilot` |
86+
| Re-review result | No response within 10 minutes |
87+
| Local modifications | None |
88+
| Outcome | Failed; PR remained open and unmerged |
89+
90+
The task demonstrated implementation convergence but not workflow completion. All observed engineering gates were satisfied except receipt of the second CCRA review and the subsequent merge.
91+
92+
---
93+
94+
## Section 4: Aggregate Statistics
95+
96+
Because this batch contained one task, aggregate and per-task values are identical.
97+
98+
| Metric | Total | Average per task |
99+
|--------|------:|-----------------:|
100+
| Tasks attempted | 1 | 1.00 |
101+
| Tasks merged | 0 | 0.00 |
102+
| PRs touched | 1 | 1.00 |
103+
| Completed CCRA rounds | 1 | 1.00 |
104+
| CCRA comments | 2 | 2.00 |
105+
| Resolved comments | 2 | 2.00 |
106+
| Phase-2 duration | 12m 37s | 12m 37s |
107+
| Output tokens | 4,698 | 4,698 |
108+
109+
**Convergence signal:** the implementation had converged against the first review: both comments were resolved and CI passed. The failure therefore measures review-service latency and orchestration timeout behavior, not unresolved code defects.
110+
111+
---
112+
113+
## Section 5: AI Credits and Token Usage
114+
115+
| Scope | Measured value |
116+
|-------|---------------:|
117+
| Local Copilot CLI assistant messages | 15 |
118+
| Local Copilot CLI output tokens | 4,698 |
119+
| Local Copilot CLI input tokens | Unavailable in exported message records |
120+
| Premium requests | 0 |
121+
| API duration | 76.942s |
122+
| Session duration | 758.923s |
123+
124+
CCA and CCRA token or billing-credit totals are not present in the local artifacts. The report therefore uses observed CCRA rounds and comments as the only measurable review-agent activity.
125+
126+
---
127+
128+
## Section 6: Wall-Clock Timeline
129+
130+
| Time (UTC) | Event |
131+
|------------|-------|
132+
| 20:12:04 | CCRA's first review completed with 2 comments. |
133+
| 20:14:38-20:14:40 | Replies recorded that both findings were fixed in `bc0275f8`; both threads were resolved. |
134+
| 21:15:53 | Phase-2 shepherd session started. |
135+
| 21:16:03 | Shepherd confirmed [#2173](https://github.com/github/copilot-sdk/pull/2173) was open, non-draft, mergeable, and linked to [#2168](https://github.com/github/copilot-sdk/issues/2168). |
136+
| 21:16:25 | Existing CI checks were confirmed green, including Java SDK tests on JDK 17 and JDK 25. |
137+
| 21:16:47 | Shepherd confirmed both comments were fixed, replied to, and resolved. |
138+
| 21:16:54 | Re-review request through the installed `gh` 2.45.0 failed on the deprecated Projects-classic GraphQL field. |
139+
| 21:17:00-21:17:19 | REST reviewer requests using `copilot-pull-request-reviewer` failed with HTTP 422 because that identity is not a collaborator. |
140+
| 21:18:05 | A standalone `gh` 2.96.0 successfully requested `@copilot` review. |
141+
| 21:18-21:28 | Shepherd polled for a new review or Copilot inline comment. |
142+
| 21:28:10 | Poll ended with `ERROR: Copilot review did not complete within 10 minutes`. |
143+
| 21:28:30 | Session reported failure; [#2173](https://github.com/github/copilot-sdk/pull/2173) remained unmerged. |
144+
145+
---
146+
147+
## Section 7: Failure Analysis
148+
149+
### 7.1 Primary Failure
150+
151+
The immediate failure was a bounded wait timeout. The re-review request succeeded, but CCRA did not post a review within 10 minutes. The script treated absence of a review by the deadline as fatal and exited before merge.
152+
153+
**Evidence:**
154+
155+
- Current `gh` reported a successful `pr edit ... --add-reviewer @copilot`.
156+
- The polling loop checked both PR reviews and line-level Copilot comments.
157+
- The loop terminated with `ERROR: Copilot review did not complete within 10 minutes`.
158+
- The final shepherd message explicitly stated that the PR remained unmerged.
159+
160+
### 7.2 Contributing Factors
161+
162+
1. **The configured timeout was shorter than the external service's observed latency.** A successful asynchronous request was converted into a hard campaign failure after exactly 10 minutes.
163+
2. **The installed GitHub CLI was too old for the supported `@copilot` reviewer flow.** Version 2.45.0 queried a deprecated Projects-classic field and failed before submitting the request.
164+
3. **The initial REST fallback used the wrong reviewer model.** `copilot-pull-request-reviewer` resolves as an organization, not a collaborator user, so the REST endpoint returned HTTP 422 on all three attempts.
165+
4. **Outcome signals differ across layers.** The caller exit code was `1`, while the exported session result carried `exitCode: 0`. Automation that reads only the latter could incorrectly classify the run as successful.
166+
167+
### 7.3 Root Cause Classification
168+
169+
| Category | Assessment |
170+
|----------|------------|
171+
| Code correctness | Not the observed blocker; known comments were fixed and CI was green |
172+
| CCRA availability/latency | Primary external blocker |
173+
| Shepherd timeout policy | Primary orchestration contributor |
174+
| Local `gh` compatibility | Recoverable setup delay |
175+
| Reviewer identity handling | Recoverable request-path error |
176+
177+
---
178+
179+
## Section 8: Observations and Recommendations
180+
181+
### 8.1 What Worked Well
182+
183+
- The shepherd correctly found and validated the issue/PR relationship and target base branch.
184+
- It verified the actual review-thread state rather than assuming replies meant resolution.
185+
- It confirmed broad CI success before attempting merge progression.
186+
- It recovered from the obsolete local `gh` version by installing a current CLI in session-local storage.
187+
- Polling remained blocking, avoiding an idle-session termination.
188+
189+
### 8.2 What Failed
190+
191+
- A successful review request was not durable across a timeout; the only result was campaign failure requiring manual intervention.
192+
- The skill tried unsupported reviewer identities before using the documented `@copilot` flow.
193+
- Tool compatibility was discovered during the critical path rather than checked at startup.
194+
- The session result's zero exit code does not encode the semantic shepherd failure.
195+
196+
### 8.3 Recommendations
197+
198+
1. **Persist a resumable state after requesting review.** Record PR number, request timestamp, baseline review ID, and resolved-thread count so a retry can continue waiting without repeating completed work.
199+
2. **Use a longer or configurable CCRA timeout.** Treat 10 minutes as an observation checkpoint, not an immediate terminal failure; use bounded backoff with a larger campaign-level deadline.
200+
3. **Preflight `gh` capability.** Require a version supporting `--add-reviewer @copilot` before entering phase 2, or invoke a known-compatible bundled binary from the start.
201+
4. **Remove the invalid REST fallback.** Do not submit `copilot-pull-request-reviewer` through the standard collaborator reviewer endpoint.
202+
5. **Separate request failure from review latency.** Emit distinct failure signatures such as `REVIEW_REQUEST_FAILED` and `REVIEW_PENDING_TIMEOUT` so retries and post-mortems can classify them automatically.
203+
6. **Propagate semantic failure into exported results.** Ensure a final `SHEPHERD FAILED` state produces a nonzero session result or an explicit machine-readable outcome field.
204+
7. **Avoid repeating already-completed engineering work.** On retry, revalidate that `bc0275f8`, thread resolution, and green CI remain current, then resume from the second-review wait.
205+
206+
The safest recovery for this task is a phase-2 retry that recognizes the existing fixes and resolved threads, checks whether the delayed CCRA review has since arrived, and merges only after the required review state is satisfied.

0 commit comments

Comments
 (0)