Repository navigation
fix(orchestrator): project-reconcile.sh times out over ~200 items — make the bounded pass the standard path #1408
Description
Problem
project-reconcile.sh (org Project #4 "Developing Today" sync, orchestrator is single writer per #910 Stage 4) computes ALL proposals over the full ~200-item Project before writing anything. The proposal computation does not finish inside the run's time budget, so passes die with 0 Project writes:
- 2026-10-27 07:41Z pass — ended incomplete at 540s timeout mid-list; which MOVE writes completed is unverified.
- 2026-10-27 08:12Z pass — exit 124 at 240s, 0 writes (report:
reconcile-2026-10-27-0812.txt, header + KEEP lines only). - 2026-10-27 08:43Z pass — interrupted during proposal computation, 0 writes (
reconcile-2026-10-27-0843.txt, header only). - 2026-10-27 11:14Z pass — dry-run timed out at 240s after proposing on only ~20 of 200 items;
--applynot attempted because it would time out identically.
Four consecutive incomplete passes. The Project board drifts behind actual issue/PR states until this lands.
What works today
The bounded inline pass (2026-10-27 06:23Z) applied 14 updates successfully by working a bounded slice instead of the whole list. The very first Stage 4 pass (2026-10-26) also completed when the Project had 27 items — the approach does not scale to 200.
Fix
Make the bounded pass the standard path in project-reconcile.sh:
- Process items in bounded batches (e.g. oldest / most-blocked first, within the existing ~25-update cap), writing each batch before computing the next.
- A pass that hits its time budget must leave the writes it completed applied and reported — never all-or-nothing.
- The run report must state exactly which writes completed, so the next pass can resume instead of re-deriving state.
- Keep the existing rules unchanged: label-gated auto-add (now/next/epic), status sync signals, single-writer discipline, no new Project fields.
Acceptance
- A full pass over the ~200-item Project completes within the orchestration run's budget, or completes a bounded batch with applied writes and a precise resume point.
- Two consecutive orchestration passes run with no incomplete-reconcile flag in the run log.
Activity
developing-today commentedon Oct 27, 2026
CODE-499
dezren39 commentedon Oct 27, 2026
Intake review
🤖 Agent note · filed
## AcceptanceRelated: Related: filed from the orchestrator's own run logs after four consecutive incomplete passes (2026-10-27 07:41Z–11:14Z); the orchestrator/single-writer context is #910 Stage 4. The body already carries an
## Acceptancesection, so no criteria were appended — this pass is labels + this note only.Risks / notes: The bounded inline pass at 06:23Z (14 updates applied) is the existence proof. Risk: batching must preserve the single-writer discipline and the ~25-update cap; a pass that reports writes it did not actually apply would be worse than today's honest zero — the report's write list should be verified against the Project after the first landed run.
Suggested approach: Implement the bounded batch loop inside
project-reconcile.shitself (write each batch before computing the next), then watch two consecutive orchestration passes for the incomplete flag.Liveness: searched merged PRs and recent fleet records this run — no merged PR explicitly resolves this issue, so it stays open. Marked
meta/enriched.dezren39 commentedon Oct 28, 2026
Reconcile timeout fixes implemented (all three approved by Drew 2026-10-28 ~06:05 CDT). The hourly orchestration pass remains the single writer — no
--applywas run outside it; the pass picks the new script up on its next reconcile step, and the incremental state file is already primed.1. Incremental proposals —
project-reconcile.shnow persistsproject-reconcile-state.jsonbeside itself: per item, the last proposal plus the signal snapshot it came from (issue state +updatedAt, and a hash of the repo's open-PR/branch signals). An item whose snapshot is unchanged reuses its stored proposal and skips the timeline fetch entirely. Cross-reference events bump the issue'supdatedAt, and any open-PR or branch change flips the repo signal hash, so a stale reuse cannot survive a signal change. Missing/corrupt state, or a new--fullflag, forces a full recompute.2. Batched signal reads — issue state/
updatedAtand timeline cross-references for all ~200 items are now fetched with a few GraphQL queries per repo (aliased issue fields: one light query per repo for every item, one heavy timeline query covering only new/changed items). The per-repo open-PR and branch reads were already batched. If a repo's batched query fails, that repo falls back to the legacy per-item REST path — a GraphQL outage degrades to slow, not wrong. Proposal semantics are unchanged: same signal→status mapping, same ~25-write cap, same Done → In Review → In Progress → Todo apply order. The PR/branch matching logic was lifted verbatim into a shared function used by both paths.3. Step timeout 600s → 1500s — the 600s was not stored anywhere: not in the script, and not in the orchestration-pass job definition (job timeout is 1800s requested / 3600s effective). The pass worker was applying it ad hoc per run (240s in one early run, 600s later). It is now pinned durably in the job body's step-3 HOW line: run the script under
timeout 1500. Verified by diffing the saved job definition before/after — that line is the only body change.Measured (dry-run, same 200 items)
dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
3 remaining items
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7242 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: picked_up
task: 7242 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151)
state: claimed; first checkpoint records origin=openchamber-scheduled-lane.
verified: #1408 is OPEN; no open PR search result matches it. Prior triage comment #6590720979 reports the script is outside this repo and recommends an operational confirmation of two clean passes. I will recheck current main before deciding whether this task is already done or needs an owner handoff.
next: fetch main safely and verify the script path and issue acceptance against current main.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: done
outcome: already_done; task 7242 was a duplicate coverage-loop triage of work already completed by task 2579 (#6590720979).
verified: fetched origin/main SHA a51be0e contains neither a project-reconcile script path nor any tracked project-reconcile reference (git ls-tree + git grep). #1408 remains OPEN; no open PR references it. The prior triage records the out-of-repository scope and the remaining external check: two orchestration passes with no incomplete-reconcile flag.
disposition: no code fix task can be routed in developing-today/code; do not close #1408 without the two-pass evidence. Claim labels cleared; no files changed.