Repository navigation
OpenClaw: finish setup, begin passing jobs, and review results against OpenCode #2171
Description
What this is
OpenClaw (gateway on the laptop, :18789) is running but the fleet barely uses it: ACP agents are configured on paper (PR #1728), the acpx wrappers exist, and Drew directed 2026-10-29 that we begin passing real jobs to OpenClaw and review the results against OpenCode directly.
Work
- Finish the setup. Verify gateway auth/binding and the agent roster once PR openclaw: enable claude, codex, agy as ACP agents alongside opencode #1728 has merged and the NixOS switch has run (both pending). Repoint the acpx adapters (Nix-package the Claude/Codex ACP adapters — repoint OpenClaw acpx off tmp/plugin-captures #1711) and land the muse-serve adapter (Build the acpx 'muse' agent — ACP adapter for the Muse CLI over muse serve (implements #1721) #2012). Confirm which agents OpenClaw can actually drive (opencode, claude, codex, agy) with a live smoke run each.
- Define the job-pass contract. A job goes to OpenClaw as a packet: scope, repo/worktree, evidence links, definition of done. A result comes back as: outcome, artifacts (PR/branch/files), tokens used, wall time. A result without those fields does not count as done.
- Begin passing jobs. First classes, in order: read-only investigations; design reviews; small, well-scoped implementations in a worktree. Grow volume only as the review below justifies it.
- Review every OpenClaw job against OpenCode. For each job record: outcome (accepted / needed rework / failed), tokens used, time to complete. Compare against OpenCode runs of the same job class, using queue completions and the Supabase mirror tables (agent.mirror_*) as the OpenCode baseline. The comparison lives on this issue until it earns a dashboard (External-system observability: unified PM + queue + quota attribution (webhooks, hot list, session IDs, projections) #2072).
Acceptance
- First 5 jobs passed to OpenClaw with complete result packets.
- A written comparison of those 5 against matched OpenCode runs: outcomes, tokens, time.
- A recommendation on which job classes move to OpenClaw by default.
Related
#1711 (acpx repoint), #2012 (muse-serve adapter), PR #1728 (ACP agents), #2072 (observability).
Activity
developing-today commentedon Oct 29, 2026
CODE-1063
dezren39 commentedon Oct 29, 2026
First job passed + first review row (2026-10-29, from the scheduled-tasks study — full findings in ~/workspace/research_notes/openclaw-scheduled-tasks-20261029/findings.md; scheduling plan filed separately):
Smoke job — read-only self-report via
openclaw agent --agent main --session-key smoke-20261029 --json:opencode).Blockers found while passing it: (1) gateway token SecretRef does not resolve from a bare SSH shell — OPENCLAW_GATEWAY_TOKEN must be exported from ~/.secrets/openclaw/gateway-token (env only); (2) the agent primary commandcode/deepseek/deepseek-v4-flash returns 401 (pending Command Code sign-in) — the run succeeded on the design-code fallback. Both are recorded as gates on the scheduling issue.
dezren39 commentedon Oct 29, 2026
DREW (2026-10-29): Aster's proposed first OpenClaw-vs-OpenCode comparison is APPROVED AS DESIGNED and ORDERED RUN — five frozen fixtures (issue-delta triage, CI-log diagnosis, repo/config audit, design review, completed-run evidence review), one run per executor per fixture, 10 runs cap, identical specs and source snapshots for both sides. Results land here as review rows under the shared review contract (quality / time split / usage where measurable, supervision time noted; missing metrics = unavailable, never zero). Dispatched 2026-10-29. Also recorded from Drew: OpenClaw is placed ambitiously, not cautiously — being less proven is what using it fixes.
dezren39 commentedon Oct 29, 2026
OpenClaw vs OpenCode — comparison results (review rows)
Five frozen fixtures, one run per executor per fixture (10 runs, cap respected), identical prompt + source snapshot to both, all read-only. Full findings:
~/workspace/research_notes/openclaw-vs-opencode-20261029/findings.md(artifacts, frozen prompts, and scoring ground truth alongside).Routes that actually served: OpenCode leg = OpenChamber-managed OpenCode instance on the laptop, agent
build, routeopencode-route/workhorse→ deepseek-v4.1-flash#high (door varied per run — see table). OpenClaw leg = gateway agentmain, primarycommandcode/deepseek/deepseek-v4-flashon all five runs, first attempt, zero fallbacks — the key replaced tonight works; the smoke test's 401/fallback behaviour is gone.Token caveat: the systems count differently (OpenClaw input includes its full harness prompt uncached; OpenCode's rides in cache reads) — compare cost-relatively, not as absolute totals.
Per-fixture verdicts
--fail-on-changeexit 1), exact evidence lines incl. the changed file.github/ci-policy.md, formatting-gate classification, fix command, and "unchanged re-run fails identically"; both ignored the decoy flake warnings. OpenCode added a generator-suspicion catch (file grew 5,341→7,863 bytes mid-check).deepseek-flash-planreferenced at 6190/6525/6575/6579/6710, undefined in-slice, correctly handled as QUESTION) and shared five further catches. OpenCode went deeper (17 findings, incl. summit's reserve rule defeated by sol-plan's own round-robin) with a clean-checks section; OpenClaw had cleaner classification discipline and two unique catches. OpenClaw spent 23,903 output tokens for a 4,188-char deliverable vs OpenCode's 3,028 for 11,232 chars.opencode-godoor and died withprovider.invalid-output("OpenAI Chat stream ended without finish_reason"); its partial reasoning was on-track. Platform reliability failure, not an answer-quality judgment.Headline: answer quality is at parity (both DeepSeek Flash-class; OpenCode deeper on judgment-heavy work, OpenClaw cleaner in classification and equal on triage/diagnosis). The difference is platform reliability: OpenClaw dispatched 5/5 first-try on its primary route; the OpenCode leg completed 4/5, and only after routing around two non-viable surfaces (
opencode runblocks without creating a session; the standing 13420 server hangs on prompt POST, session DB stale ~Oct 6). Also surfaced: five OpenCode runs drew four different provider doors from the workhorse composition — door-level variance inside the pool is real, and the single failure came on the base door.dezren39 commentedon Oct 29, 2026
Bug issues filed from the 1v1 findings (Drew's ask; evidence: research_notes/openclaw-vs-opencode-20261029/findings.md):
opencode runblocks indefinitely without creating a session — already tracked: opencode#193 ("opencode run (CLI) hangs after init on headless Linux", open since 2026-10-20). No duplicate filed; the comparison's fresh reproduction (two attempts killed, no session created at all) was added as evidence: opencode run (CLI) hangs after init on headless Linux — investigate developing-today-forks/opencode#193 (comment)provider.invalid-output — stream ended without finish_reason, 374s, no deliverable) — per-door reliability evidence; it falls inside opencode#6's adopted scope (per-credential health state + escalating cooldown per the task-3287 design pass), so it was filed as evidence on Bump build/buck/preludes/buck2-prelude from3a02ea3to4371899#6 rather than a new issue: feat(routing): automatic multi-credential failover on 429 quota exhaustion via credential.activate developing-today-forks/opencode#6 (comment)dezren39 commentedon Oct 29, 2026
Round 2 (OpenClaw on workhorse) — BLOCKED before any run; config change needed
At Drew's order I attempted the round-2 retest: same five frozen fixtures through OpenClaw, but carried by
opencode-route/workhorseinstead of Command Code, to isolate platform vs door. No fixture ran — under the current rendered config there is no per-run way to do it, and the task barred config edits for a test hack (~/.openclaw/openclaw.jsonis a read-only Nix symlink regardless). Full record appended to the findings file ("Round 2 — OpenClaw on workhorse").Routes tested, in order:
openclaw agent --model opencode-route/workhorse→Unknown model. OpenClaw's providers are only agy / codex-proxy / commandcode / design-code (local proxies :18743/:18744/:18748/:18746); none fronts the OpenCode server or exposes route ids./acp spawn opencodein a CLI session → treated as an ordinary turn; no ACP session created, and a follow-up probe in the same session was answered by main itself on Command Code (envelope-verified). ACP bindings are a chat-channel feature.sessions_spawn({runtime:"acp", agentId:"opencode", mode:"run", model:"opencode-route/workhorse"})via the gateway main agent → the agent reports there is nosessions_spawntool in its environment (searched its full catalog).openclaw agent exec→ same answer: tool not exposed on the embedded headless surface either.The ACP plumbing itself is healthy (acpx backend, dispatch enabled,
opencodeallowed and mapped toopencode acp; OpenCode's default model is already the workhorse route after the defaults sweep). What's missing is a dispatch surface. Any one of these unblocks round 2 (all arehome/openclaw/default.nixchanges riding the switch):agents.list[](runtime.type="acp",runtime.acp.agent="opencode") — smallest; matches the Add OpenClaw to #alignment — decide one shared channel vs point-to-point #2170 topology.sessions_spawnto headless/CLI sessions (tool-profile change).opencode-routeprovider tomodels.providers— largest; also needs an OpenAI-compatible front for the OpenCode server, which doesn't exist today.Note: PR #1728 widens
acp.allowedAgents, which is necessary but not sufficient — it governs which harnesses may be spawned, not whether a headless session can spawn at all. This gap survives it.Door-vs-platform verdict: not yet decidable. Round 1 (quality parity, OpenClaw reliability win, all five OpenClaw runs on the Command Code door) stands as recorded; the confound round 2 was meant to remove is still in place until one option above lands. The re-run is then mechanical: frozen fixtures, ACP opencode harness, model
opencode-route/workhorse, door verified per run from the OpenCode v2 session tables.dezren39 commentedon Oct 29, 2026
Round-2 unblock — direction update (Drew, 2026-10-29): build all three mechanisms, no menu. Tracked on #2229:
home/openclaw/default.nix(agents.entries.*.runtime.acp.agent="opencode") — standing OpenClaw→OpenCode dispatch.sessions_spawnfor headless/CLI sessions — determined from the installed OpenClaw 2026.10.7 dist: NOT a config/tool-profile knob. The tool is created only when!embedded || allowGatewaySubagentBinding(upstreamcreateOpenClawTools); embedded mode is a process-local flag the CLI entry point sets, and the binding flag is passed only by gateway/channel entry points. So this one is a fleet packaging patch on the pinned OpenClaw build, documented as such on OpenClaw cannot reach the OpenCode server: build all three dispatch mechanisms (ACP-runtime agent, headless sessions_spawn, opencode-route provider front) #2229.models.providersentry so--model opencode-route/workhorseworks directly.Each mechanism: own PR, proven live (an OpenClaw run actually carried by
opencode-route/workhorse, verified in the OpenCode v2 session tables) where the mechanism allows.dezren39 commentedon Oct 29, 2026
Method 3 of #2229 (the priority target) is built and live-proven: PR #2250 adds the opencode-route provider proxy (port 18750). An OpenClaw run with
--model opencode-route/workhorsecompleted carried by the workhorse pool (OpenCode sessionses_e5e78b6f4f4ffeVyPSZquFZkwV, opencode-go/deepseek-v4.1-flash). Details on #2229. Methods 1 and 2 continue next.dezren39 commentedon Oct 29, 2026
Scope final on #2229 (Drew, 2026-10-29): method 3 (opencode-route provider front) was the priority build — done, PR #2250, live-proven. Methods 1 and 2 are descoped to follow-ups: method 1 (ACP-runtime agent) → #2256, method 2 (sessions_spawn for headless/CLI) → #2257, each with its determination/starting point recorded and the trigger "sometime soon / when an OpenClaw-origin workload needs it".
dezren39 commentedon Nov 3, 2026
agent-note:v1 — task 7856 picked up by lane code-k (repo-worker-code). Execution lane assigned: opencode — handed off to the opencode claimant, which will launch a bound session to triage issue #2171 at current main.
developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/none). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 3, 2026
dezren39 commentedon Nov 3, 2026
Triage (task 7856, coverage pass 2026-11-03, verified at origin/main 8e151e8)
Verdict: still live, not already-done. Setup is part-finished, the 5-fixture comparison is recorded, but the acceptance recommendation is missing and several setup pieces are still open.
What is done at current main:
home/openclaw/default.nixat main has all four inacp.allowedAgents, dispatch enabled, defaultAgent opencode.opencode-route/workhorse. OpenClaw cannot reach the OpenCode server: build all three dispatch mechanisms (ACP-runtime agent, headless sessions_spawn, opencode-route provider front) #2229 itself CLOSED 2026-11-01.What remains (fix path + ordering):
Suggested ordering: #1711 + #2257 → round-2 retest → recommendation → phased job classes; #2012 and #2256 proceed on their own blockers (Drew decision; #2509 schema) without gating the recommendation.
Execution note: laptop SSH unreachable from this lane session (tunnel proxy CONNECT failed), so verification was read-only via local clone + gh API at origin/main 8e151e8; no worktree created, no edits made (triage-only task).
developing-today commentedon Nov 6, 2026
🤖 Agent note · escalated
developing-today commentedon Nov 6, 2026
Correction to the fleet monitor escalation note above (its label names were stripped by a shell substitution error when posting): the stale labels on this issue are
queue/claimed+work/assigned; the requeue path, if the reviewer finds the work is not done, is to swap them toqueue/queued+status/needs-triageand enqueue a fresh triage packet;user/reviewwas added by that note.