Repository navigation
Feature: verifiable dreamer run progress (pending count + per-task status) #272
Description
Short description
Surface verifiable progress for on-demand /ctx-dream runs so users can track backlog draining (e.g. unclassified memories) instead of relying on the end-of-run completion summary alone.
What problem does this solve?
The Dreamer runs a suite of tasks as child sessions, but the only feedback is a summary notification after the run completes. During long-running tasks there is no way to observe:
- how many items (memories/notes) remain to be processed in the current task;
- whether the run is making progress or stuck;
- which task is currently active.
Concretely, on a project with ~6.3k memories, map-memories and classify-memories abort with incomplete: N selected memories remain after the 20-minute default timeout_minutes. The message says N remains, but there is no increasing/decreasing signal across repeated on-demand runs, no per-task breakdown, and no percentage — so a user cannot tell whether repeated /ctx-dream invocations are converging or stuck on a prompt/issue (see closed bug #232 where classify-memories was silently using the wrong system prompt; a progress counter would have made that visible immediately).
Proposed solution
- Persist per-task progress in the shared SQLite DB (a
dream_progresstable or columns ondream_runs): current task name, items processed, items remaining, started/finished timestamps, and status (pending/running/done/failed) for each task of the active run. - Expose it via a read-only surface: either a
magic-context statusCLI subcommand, a TUI sidebar section, or a Dashboard "Dream runs" progress view. The CLI is the most portable across OpenCode/Pi and trivially scriptable. - Emit progress during the run: the runner writes rows as each task iterates its selected items (e.g. every batch or every N items), so a watcher can poll and see the number decrease.
Alternatives considered
- Parsing the completion summary: insufficient — it only arrives after the run, carries no delta, and no partial info if a task aborts.
- Log scraping (
/tmp/opencode/magic-context/magic-context.log): works but is fragile, harness-specific, and untyped. - A
statuscallback hook: viable for dashboards but leaves CLI users (who trigger/ctx-dreammost) without a way to poll.
Area
Memory / historian / dreamer
Additional context
- The
dreamer.tasksconfig already supports per-tasktimeout_minutes(raised to 90 formap-memories/classify-memorieson our side); a progress view would let users right-size these timeouts with evidence. - Related closed bug: "classify-memories" task uses my system prompt #232 — "classify-memories task uses my system prompt" (a progress counter surfaced the anomaly).
- Inspiration: CI runners show live job progress; the Dreamer is effectively a background job runner and would benefit from the same observability.
- We run a small monitoring script that queries
memories WHERE classified_at IS NULLbefore/after each run to approximate progress; a first-class counter would replace this workaround.
Activity
feat(dreamer): verifiable run progress (#272)
alfonso-magic-context commentedon Aug 7, 2026
Implemented on master (44155b4), shipping in the next release. What you get:
/ctx-dream <task>prints the task's pending/total count before starting, so run → observe the drop → run again works as you described.incomplete: N remain (was M at run start; processed M-N this run)— a stuck run and a slow run are finally distinguishable.No scheduling or timeout behavior changed — partial progress already persisted for the chunked tasks (that's why your repeated runs were draining the backlog even when each run reported incomplete); it just had no visible signal. Thanks for the concrete 6.3k-memory numbers — they made the right scope obvious.