Success outro claims all changes even when some steps didn't complete #951

Activity

1. gewenyu99 commented on Jul 22, 2026

   

   gewenyu99

   on Jul 22, 2026

   Collaborator

   More actions

   

   This will be fixed with our new orchestrator agent.

   👍React with 👍1Reacted by Peter Trost
2. gewenyu99 commented on Sep 23, 2026

   

   gewenyu99

   on Sep 23, 2026

   Collaborator

   More actions

   The orchestrator did not fix this. A real-TUI sweep finds the same problem in three more programs. The sweep ran on 200961f (the head #1303 merged), on the older base beccc79 and on the refactor stack head d938a43 (#1307), and all three behave the same. Main (d8486dc) has the same code. In each run the PostHog MCP connection lacked a scope the step needed. The agent said so in its report, but the outro and the task list still reported success.

   Problem

   warehouse-source (app revenue/stripe/stripe-saas-demo). The connection had no external_data_source:read or external_data_source:write. On 200961f the report starts with "Created 0 of 1 detected sources in PostHog." and ends with "No source was connected or verified during this run." On all three heads the outro shows ✔ Data warehouse source connected! and every task shows as completed.

   replay-vision (app replay-vision/react-vite/court-booking, orchestrator). The connection had no product_enablement:write and no replay_scanner:read or replay_scanner:write. The report says session replay could not be turned on server-side and that no scanner was created. On d938a43 the task list still shows "Enable session replay" as completed. The outro says ✔ PostHog set up: 4/4 steps completed (3 skipped as not required). The three scanners did not skip because they were not required. A missing scope blocked them. The agent's status choice also changes between runs. On 200961f, "Scan for broken experiences" showed as completed and the outro said 5/5 steps, but the report said no scanner was created. On beccc79, two scanner tasks showed as completed and none were created.

   self-driving (app self-driving/react-router/expense-splitter). The connection had the same three gaps. The report says Support could not be enabled without product_enablement:write, and that both Replay Vision monitors were skipped without replay_scanner:read and replay_scanner:write. On d938a43 all nine tasks show as completed, including "Enable products (replay, error tracking, support)" and "Set up Replay Vision scanners". The outro says ✔ Self-driving is on. PostHog is scanning your project; first findings hit your inbox within ~30 minutes.

   The same "skipped as not required" count also hides steps that were needed but not done. In the error-tracking run on express-todo (d938a43), "Collect upload credentials" and "Wire source-map uploads" were skipped, and the TUI showed only "(2 skipped as not required)". The report itself lists the missing upload key as a step the user still has to do.

   Evidence

   Paths are on main, d8486dc.

   warehouse-source:

   • src/lib/programs/warehouse-source/index.ts:58: the success message is the fixed string 'Data warehouse source connected!'. The program has no buildOutroData.
   • src/agent/runner/sequence/linear.ts:293-304: when a program has no buildOutroData, the linear sequence builds an OutroKind.Success outro from run.successMessage. What the agent actually created has no effect on it.
   • The skill already reports the true outcome. In context-mill, context/skills/data-warehouse-source/description.md:101-111 makes the report's first line Created N of M detected sources in PostHog. and tells the agent not to call a run with zero sources complete. On the linear path, the wizard never reads that count.

   replay-vision:

   • In context-mill, context/agents/replay-vision/enable-replay.md:24-29 says that on a permission rejection the agent should record a follow-up "and finish". So the agent reports done.
   • In context-mill, context/skills/replay-vision-scanners-core/description.md:24-33 says the same for a 403 or a missing tool but does not name a status. So the agent picks done on some runs and not needed on others.
   • src/agent/runner/sequence/orchestrator/queue.ts:75 has NotNeededReason.Blocked, added in fix(orchestrator): record why an agent reported a task 'not needed' #1195. Nothing shown to the user reads it:
     • src/agent/runner/sequence/orchestrator/orchestrator-runner.ts:1398 counts every Skipped task as not required. Lines 1460-1474 then print "N skipped as not required" in the outro.
     • src/agent/runner/sequence/orchestrator/orchestrator-runner.ts:849-858 (renderQueue) sends only content, status and activeForm to the UI. The skip reason never reaches it.
     • src/ui/tui/primitives/ProgressList.tsx:23-27 drops skipped rows from the list. Lines 74-76 print "(N skipped as not required)".
   • A related case, found by reading the code and not seen in the sweep: src/agent/runner/sequence/orchestrator/orchestrator-runner.ts:133-145 (toTodoStatus) maps TaskStatus.Failed to 'completed'. A failed optional task therefore shows the same filled square as a finished one. Open draft feat: synchronize Wizard CLI executions with WizardRun #1323 maps Failed to a new 'failed' status with a warning icon in ProgressList, so that part is in progress there. feat: synchronize Wizard CLI executions with WizardRun #1323 is on the old src/lib/agent paths.

   self-driving:

   • src/lib/programs/self-driving/index.ts:25-27 defines a fixed SUCCESS_MESSAGE, and buildOutroData at :91-96 always returns it as OutroKind.Success. What the agent could not enable has no effect on it.

   On d938a43 the code has moved (src/programs/resolve-run-definition.ts:214, src/agent/runner/sequence/orchestrator/orchestrator-runner.ts:1340).

   Suggested fix

   1. warehouse-source: add a buildOutroData that reads the Created N of M line from the report. Read it from the published handoff text first (RunSnapshot.handoffText in src/agent/runner/shared/types.ts:290, which the TUI store also keeps), and from posthog-warehouse-report.md only when that file exists. On main the publish_handoff description tells the agent not to write the file, and on d938a43 it did not, so a file-only reader would miss the line. If N is at least 1, the headline says how many sources were connected, for example "Connected 1 of 2 data warehouse sources". If N is 0, the outro is not a success. It says no source was connected and points to the report and the browser setup link. If the line is missing, fall back to a neutral "Data warehouse setup finished" instead of "connected!".
   2. Orchestrator: split Skipped by notNeededReason. Leave not-applicable as "skipped as not required". Count blocked and user-declined separately in the outro message, for example "1 step blocked, see the report". Send the reason with the tasks event, so ProgressList can keep blocked rows in the list with their own icon. Map Failed to its own display status instead of 'completed', or land that part of feat: synchronize Wizard CLI executions with WizardRun #1323.
   3. self-driving: build the outro from what the run did. When the report or the task notes say a product or scanner was blocked, say "Self-driving is on, with N steps still to do" and list them, instead of the fixed message.
   4. context-mill: in enable-replay.md and replay-vision-scanners-core/description.md, tell the agent to call complete_task with not needed and notNeededReason: blocked when a permission is rejected or a tool is missing, and never with done. fix(orchestrator): record why an agent reported a task 'not needed' #1195 left the status choice unchanged on purpose to keep telemetry comparable. This is the follow-up that changes it, so any before-and-after comparison should start at that release.