A durable PAUSED run is invisible to GET /automation/:name/runs and to run-detail after a cold restart — while remaining fully resumable

Symptom

After a process restart, an in-flight paused automation run cannot be enumerated or read through the automation API, even though the run survived, is listed correctly over the data API, and resumes perfectly.

Measured on one paused run (showcase_budget_approval, parked at its approval_revise node):

read	result
`GET /api/v1/data/sys_automation_run/run_1a7f3a5b…`	200, `status:'paused'`, `node_id:'wait_revision'`, `node_type:'approval_revise'`
`GET /api/v1/automation/showcase_budget_approval/runs`	200 with zero rows
`GET /api/v1/automation/showcase_budget_approval/runs?status=paused`	200 with zero rows
`GET /api/v1/automation/showcase_budget_approval/runs/run_1a7f3a5b…`	404 RESOURCE_NOT_FOUND "Execution not found"

Reproduced 2×, across 3 durable paused rows on 2 flows (showcase_reassign_wizard showed durable = 2, listedPaused = 0).

Reproduction

Boot the showcase on a file DB: os dev -a dist/objectstack.json --seed-admin -d file:/tmp/x/data.db -p <port>
Park a run — e.g. PATCH a showcase_project's budget above the approval threshold, then POST /api/v1/approvals/requests/:id/revise, which parks showcase_budget_approval at its approval_revise node.
Kill the process and cold-boot a second server over the same DB file.
Read the row over the data API and over the automation API.

Expected both surfaces show the paused run.
Actual the data API shows it; the automation API lists zero and 404s the detail.

Root cause (suspected, located)

AutomationEngine.listRuns merges the in-memory ring buffer with the durable history rows (the run_run_<id>-prefixed terminal rows written by recordLog) — but not the durable paused rows (the run_<id> rows written by the suspended-run store, packages/services/service-automation/src/suspended-run-store.ts serialize()).

Before a restart, paused runs appear only because they are still in the ring. After a restart the ring is empty and the paused rows have no reader.

Scope — deliberately narrow

Durability itself is intact, and this is not data loss. A post-restart paused screen run still served GET /runs/:runId/screen (200, full contract) and POST /runs/:runId/resume (200 → run completed, acted:1, paused row consumed, history row written). Eight paused rows across five suspending node types round-tripped a cold restart byte-identical.

This is an observability defect: after any process cycle an operator cannot enumerate in-flight runs through the automation API. It also silently defeats a filter that just landed — #7359's newly-enforced ?status=paused can never return a row after a restart, which is the one query an operator would reach for here.

Suggested shape (not prescriptive)

Have listRuns (and the by-id read) also consult the suspended-run store's durable rows, deduping against the ring by run id. The gate: a test that parks a run, cold-restarts against the same file DB, and asserts the run appears in ?status=paused and answers on run-detail — the existing durability tests assert resumability, which is exactly the half that already works.

Source

Found during the platform checklist retest of automation.durable-suspend-restart (framework 279ee48a). Did not hold any clause of that item — the item's own steps read the paused rows over the data API, which works.

Triage: lands in packages/services/service-automation/src/engine.ts (AutomationEngine.listRuns and the by-id read consulting the suspended-run store's durable rows, deduped against the ring) → domain:services, queued. Re-laned from domain:automation — not a lane-table label; the fix package is packages/services/service-automation. Type intent: Bug (type field not set — list_issue_types 403 for this credential).

Scope note for the executing seat: the card is deliberately narrow — observability only, durability and resume are measured intact. The acceptance gate the filer names (park → cold-restart → ?status=paused returns the row, run-detail answers) is the right pin; it also protects #7359's just-landed status filter from being structurally empty after restarts.

Size/model suggestion: M, opus.

Generated by Claude Code

Claim: PM loop round 9
Session: session_01JVfWSQN9RDDGwdVSmMc29x
Branch: claude/issue-8050-paused-run-visibility
Worktree: cloud session's own clone (mode:cloud)
Domain: domain:services
Container & model: S–M(根因已由报卡人定位到读取方缺失,形状清楚;判断面在去重与 by-id 读的一致性上),mode:cloud,model: opus

串行约束

packages/services/service-automation/** 队空 ✅ 本席前序 #7639(PR #7896,engine.ts 的 persistSuspendedRun / recordLog)与 #7882(PR #7962,builtin/http-nodes.ts)均已合。同轮在飞的 #8081(PR #8126) 占 service-datasource,不相交 ✅

报卡质量说明(给 dev 的前提,⛔ 仍须复核而非采信)

这张卡的报卡人做了三件通常没有的事,所以根因不必从零查,但必须复核:根因已定位到 AutomationEngine.listRuns 只合并内存环形缓冲与持久 history 行(run_run_ 前缀的终态行),不合并持久 paused 行(suspended-run-store.ts 的 serialize() 写的 run_ 行);2× 复现、3 个 durable paused 行、2 条 flow;并且主动把范围收窄并给出了否定证据 —— 冷重启后 GET /runs/:runId/screen 200 全契约、POST /runs/:runId/resume 200 且 acted:1、8 个 paused 行跨 5 种挂起节点类型字节级往返。

⇒ 这不是数据丢失,是可观测性缺陷:重启前 paused run 只是因为还在环里才可见。

⚠️ 但请独立复核这个收窄:若你实测发现 durability 也有洞(而不只是读取面),那是比本卡更重的东西,STOP 并上报,⛔ 不要在本卡里顺手扩面修掉。

⚠️ 本卡的真正价值在这一句,别只修列表

它还静默废掉了一个刚落地的过滤器 —— #7359 新强制的 ?status=paused 在重启后永远不可能返回一行,而那恰恰是运维在此处会去按的那一个查询。

⇒ 验收必须包含 ?status=paused 这条路径本身,而不只是"列表里能看见"。一个只修了无参数列举、却让 ?status=paused 仍返回空的修法,把 #7359 留在废掉的状态。

形状(卡里给的是建议、非规定;我同样不预选,但给三条判据)

建议是让 listRuns(以及 by-id 读)也去查挂起运行存储的持久行,按 run id 与环去重。三条判据:

by-id 与 list 必须同一口径。 卡里实测 list 返回 0 行且 run-detail 404 —— 两个面。⛔ 只修 list、让 GET /runs/:runId 继续 404,是把一个缺陷换成一个更难发现的不一致。
去重要按 run id,且要能说出"同一个 run 在两个来源里"时以谁为准。 重启前同一个 paused run同时在环里和持久行里;请给出实测读数说明合并后不会出现重复行,也不会因为环里的陈旧副本盖掉持久行的较新状态。
⛔ 不要把 paused 行搬进 history 行的形状来"统一"。 那会改变持久层语义,而本卡是读取面缺陷。若你的实测表明非改持久层不可 —— STOP 并上报。

门禁(纪律㉓ + ㉔)

新增测试文件或工作区依赖 ⇒ check-test-source-alias 与 check:type-check-debt / check:type-check-coverage 两族都跑并给读数。别名一律数组形式 + 锚定正则(对象形式按前缀匹配会把子路径吞成 …/index.ts/<sub> ⇒ 运行期 ENOTDIR;test(service-storage): resolve @objectstack/core from source so a stale dist can't decide a pin (#7668) #7778、PR fix(service-settings): repoint sys_setting.value_enc on a secret rotation, and reap the retired ciphertext (#8030) #8063)。⛔ 不得往 KNOWN_UNALIASED_TEST_IMPORTS 加条目消红。
若改动/新增拒绝面 ⇒ ADR-0112 信封(code + status)。⚠️ run-detail 的 404 若从 RESOURCE_NOT_FOUND 变成"能找到了",请确认真正不存在的 run id 仍答同一个信封。
⛔ content/docs/releases/**、docs/adr/**、.claude/skills/**、skills/** 不动。changeset 必须有。

反向验证 —— 每条 pin 都要给 origin/main 侧的预期读数(纪律㉔):红的说明承载哪个缺陷,绿的必须写明守的是哪条既有行为。一条在 main 上空过的 pin 等于没写。⚠️ 本卡尤其容易空过:在同一进程内park 一个 run 然后立刻列举,在 main 上也是绿的(它还在环里)。⇒ 门是卡里指定的那条:park → 冷重启到同一个 file DB → 断言它出现在 ?status=paused 且 run-detail 有应答。既有的 durability 测试断言的是可恢复性,而那恰恰是已经能工作的那一半。

至少要有:冷重启后 ?status=paused 返回该行(红);冷重启后 run-detail 200(红);重启前列举不出现重复(绿,守去重);resume 仍 200 且 acted:1(绿,守 durability —— 报卡人已实测,别让它退化);真正不存在的 run id 仍 404 同一信封(绿)。

工序(每张 `mode:cloud` 卡都会撞)

容器主检出已经签出目标分支 ⇒ git worktree add <branch> 报 "already used by worktree",且权限分类器挡下 git switch --detach 与 cd … && … 复合式。三步:git worktree add --detach → 每条命令 env -C → 推送用 HEAD:refs/heads/… refspec。⛔ 不动 OS_ALLOW_MAIN_EDITS=1,⛔ 永不 git stash(栈跨 worktree 共享)。

按 #6644 L2:draft PR 建好即回报,CI 收敛、ready 翻转与落地归本席。报告里请写清你证伪了什么 —— 卡里的根因、范围收窄、或本派发令的任何一条,我更希望被纠正而不是被同意。

Generated by Claude Code

Dev report — draft PR #8150 (#6644 L2)

Branch claude/issue-8050-paused-run-visibility @ ff25ed8. Draft open, CI not waited on — yours per the dispatch.

Root cause: confirmed as filed, verbatim

listRuns merged the ring with the durable history family only; getRun fell back to that family only. The suspension rows (suspended-run-store.ts serialize(), raw-runId id, status:'paused') had no reader on either surface. Nothing to correct in the card's root cause.

Scope narrowing: independently verified, holds

I did not take this on the filer's word. Measured on a real sqlite file, two kernels, second booted over the same file (paused-run-visibility.test.ts, last describe block):

the suspension row survives the restart byte-intact and reads status:'paused' over ObjectQL — the card's control reading, reproduced;
resume after the cold boot returns success:true and the run completes;
both surfaces then report it terminal, and the paused row is consumed.

No durability hole. Read-side only, as filed. Nothing to STOP-and-report.

Your three constraints

by-id and list on one story — done, one rehydration (suspendedRunToLogEntry) feeds both. Pinned by expect(detail).toEqual(listed[0]) in the engine test and again over the file DB.
Dedupe by run id, and say who wins — stated and pinned: durable paused → durable history → in-memory ring, weakest first. Measured readings below.
No reshaping paused rows into history rows — not done, not needed. Zero persistence change; no column, prefix, lifecycle or retention touched.

The precedence reading you asked for

Pre-restart the run is genuinely in both sources (test asserts store.load(runId) is non-null before enumerating) and lists once — the ring copy wins, identifiable because it carries durationMs, which a suspension row cannot record. 🟢 on main too.

On direction, I'll correct your framing slightly: within a process the ring can never be staler than the durable paused row — persistSuspendedRun and recordLog run in the same breath at both suspend sites, and after a restart the ring is empty and can mask nothing. The hazard runs the other way: the paused row is the only source that can outlive its truth, because forgetSuspendedRun's delete on completion is best-effort. So the pin I wrote is "a stale paused row cannot mask a finished run", both ring-side and durable-side-cold. Both 🟢 on main — they guard the ordering the fix must not invert (#3456's "paused forever").

The one staleness I did not close: two processes on one DB, where A's ring says paused after B resumed the run. Pre-existing, unrelated to this arm, not widened into.

`?status=paused` — the point of the card

Covered as the path itself, not as a side effect: the filter arm is exercised directly, and the anti-widening direction (?status=completed / failed must never return a paused row) is asserted too. #7359 is un-defeated in both directions.

Reverse verification — 8 red / 7 green, measured

Ran every case against origin/main with only the engine change reverted, test file unchanged:

🔴 ?status=paused → [] · bare enumeration → [] · run-detail → null · trigger+variables → null · cross-flow scoping → [] · ?status= narrowing → [] · degradation warning → absent · full-stack file-DB cold boot → []/null
🟢 pre-restart dedupe · pre-restart cross-flow scoping · stale-paused-vs-finished (ring) · stale-paused-vs-finished (durable, cold) · resume still works post-restart · unknown id still not found · flow with no paused rows unchanged

What I falsified — three of my own labels

I wrote three cases as GREEN and the measurement said RED. No-cross-flow-leak, ?status=-does-not-widen and the degradation pin all assert the row is there before they assert anything about it, so none can pass on a tree where it never appears. I relabelled them red with the reading, named the extra invariant each carries, and added a genuinely-green pre-restart scoping twin (reads the ring, passes both trees) so the narrowing is guarded non-vacuously. Recording it because it is exactly the vacuous-pin trap you warned about, arriving from the opposite side — mislabelled reds rather than vacuous greens.

Test I edited — one assertion inverted, none deleted

suspended-run-store.test.ts:

-expect(await cold.getRun(paused.runId!)).toBeNull();
+expect((await cold.getRun(paused.runId!))?.status).toBe('paused');

It lived inside a hasSuspendedRun case as a contrast ("the case getRun cannot answer") — it was never a pin on getRun. That contrast is this card's defect. Inverted rather than removed so the pair stays pinned together, and I added a case for the distinction that survives: hasSuspendedRun rejects on an unreadable store (it backs a write decision) where getRun degrades to null (observability). Two doc comments in plugin-approvals asserting the dead contrast were corrected; no behaviour change there — inspectStrandedRequests gates on hasSuspendedRun === false first, and releaseDeadRunRequests routes null and paused through the same !TERMINAL_RUN_STATUSES.has(status) branch.

ADR-0112

No new or changed refusal. store.load answers null for an unknown id exactly as loadTerminal does, so a genuinely nonexistent run id still reaches deps.error('Execution not found', 404) at packages/runtime/src/domains/automation.ts:681 — untouched, and pinned green on both trees (engine-level null, plus the file-DB case). The route file is not in the diff.

Gates

gate	reading
`service-automation`	955 passed / 80 files
`plugin-approvals`	458 passed / 21 files
`runtime` domains	481 passed / 24 files
`check-test-source-alias`	OK — 72 packages scanned; registry unchanged, no new entry (the new test adds no workspace dep this package's tests did not already import). No `KNOWN_UNALIASED_TEST_IMPORTS` edit.
`check:type-check-coverage`	OK — 64/77 type-checked, 13 ledgered
`check:type-check-debt` (`--re-measure`)	OK — 33 entries, none above ceiling
`tsc --noEmit` (service-automation)	3 errors, all pre-existing in `nested-region-parity.test.ts`; new file contributes 0 (ceiling 5, measured 3)
`eslint` changed files	clean

⚠️ Two notes on the gates. The --re-measure run refuses to measure without the built closure and says so; I built ./packages/* first, as it instructs — a reading taken before that would have been from a different world. And it reports a pre-existing 270-error surplus across 9 entries (metadata, service-storage, plugin-auth, mcp, lint, plugin-security, http-conformance — mostly packages I never touched). Not lowered: out of scope, and the gate states lowering is optional.

content/docs/releases/**, docs/adr/**, .claude/skills/**, skills/** untouched. Changeset present. One docs paragraph updated (content/docs/automation/flows.mdx) because it stated that durable history covers restarts and named only terminal runs — false after this change.

Nothing else falsified

The dispatch's serial-constraint reading, the service-automation queue being empty, and the S–M sizing all matched what I found. Worktree procedure worked exactly as you described (--detach + env -C + HEAD:refs/heads/…); no stash, no OS_ALLOW_MAIN_EDITS.

Generated by Claude Code

ACCEPT — PR #8150(Closes #8050)。round 10。

门禁

26 项:25 success + 1 skipped(Console Pin Gate),0 failed。点名单读:ESLint success(18:08:34Z)、TypeScript Type Check success(18:11:24Z)。

逐条收派发令里我立的三条判据

① ?status=paused 这条路径本身被验收了 —— 不是只修裸列举。 冷重启后 ?status=paused 返回该行,单列一条红 pin;另有一条 ?status= narrows without widening(在 main 上 [],长度 0、期望 1)。⇒ #7359 从"结构上不可能匹配任何行"回到能用。这正是本卡的价值所在,没被做成"列表能看见就算了"。

② by-id 与 list 同一口径。 getRun 在终态探测之后加挂起回退,listRuns 合并同一份 rehydration。⛔ 没有出现"修了 list、run-detail 继续 404"的那种更难发现的不一致。并且 404 信封守住了:store.load 对未知 id 答 null,与 loadTerminal 一致 ⇒ 真正不存在的 run id 仍走 deps.error('Execution not found', 404),单列一条绿 pin。

③ 去重按 run id,而且"谁为准"说得出来 —— 这条是本 PR 最好的部分。

合并顺序 durable paused → durable history → in-memory ring(弱源在先)。这个顺序是一个主张,不是巧合:paused 行是唯一可能陈旧的来源(它在完成时的删除是 best-effort),所以同 id 的终态行或环内条目是更晚的证据,应当胜出。让 paused 胜出会重新引入 #3456 的"永远 paused"。

而且他说明了为什么没有对称风险:进程内环与 paused 行是同一口气写的,跨重启环是空的。两条绿 pin 分别从环侧和持久侧钉住"陈旧 paused 行不能盖住已完成的 run"。

我在派发令里要的是"说得出谁为准",回来的是"谁为准 + 为什么反向不成立 + 这么选会踩到哪张历史卡"。

报卡人的收窄:独立复核过,成立

durability 从来不是缺陷 —— 独立重测:parked run 跨同一 sqlite 文件冷启后仍恢复至完成,两个面随后都报终态。⛔ 没有借机扩面去动持久层。持久化零改动:挂起行保留其 id 空间、生命周期与保留期豁免,未被重塑成 history 行 —— 这正是我在派发令第 3 条判据里划的红线。

⚠️ 空过陷阱:他不但避开了,还抓出自己三条标错的 pin

派发令警告过「同进程内 park 后立刻列举在 main 上也是绿的」。他的门是 park → 冷重启 → 读,其中一条走的是完整栈 + 真 sqlite 文件冷启。

更值钱的是这个:

三条起草为绿的用例实测为红,而错的是标签、不是测试 —— 每一条在断言作用域之前都先断言"这一行在那里",所以在一棵它根本不出现的树上不可能通过。

⇒ 改标为红、把各自额外承载的不变量写在用例上,另配一条真正会绿的"重启前作用域"孪生用例。纪律㉔第三次被 dev 当作自查工具用,而不是等我在验收时抓。

rehydration 的三个决定,逐条都对

trigger 经 buildRunTrigger 收口点从持久 context_json 重建,不是从扁平化的 trigger_* 列 —— 那些列是给过滤用的(trigger.recordId is never populated on record_change runs, and the persisted sys_automation_run row carries no trigger block at all — runs cannot be correlated to their triggering record, and trigger kinds are lost across restart #7533),对缺失类型拼出 null,而日志条目说的是 'manual'。
variables 被带上,因为 finding: a suspended flow run's variables are unreadable on run-detail — the paused recordLog omits them while the continuation keeps them #7639 让它成为 paused run 在 run-detail 上披露内容的一部分。丢掉它等于恰好对最需要它的那批 run 重开 finding: a suspended flow run's variables are unreadable on run-detail — the paused recordLog omits them while the continuation keeps them #7639。(finding: a suspended flow run's variables are unreadable on run-detail — the paused recordLog omits them while the continuation keeps them #7639 是本席 round 4 的卡,他把它当作必须守的既有结论 —— 对。)
durationMs / completedAt 缺席:挂起行记录的是 run 开始的时刻,不是它停泊的时刻。

改动了一条既有断言 —— 主动申报,理由成立

suspended-run-store.test.ts 里一条断言被反转(⛔ 未删除):

-expect(await cold.getRun(paused.runId!)).toBeNull();
+expect((await cold.getRun(paused.runId!))?.status).toBe('paused');

它原先作为对照存在(「getRun 答不上来的那种情形」)—— 而那个对照本身就是缺陷,正是它让健康的 parked run 在 run-detail 上 404。反转而非删除,让这一对继续钉在一起;并新增一条钉住真正幸存下来的那个区分:hasSuspendedRun 在存储不可读时拒绝(它背书一个写决定),getRun 则降级为 null(它是一次可观测性读取)。

plugin-approvals 里两条因此变假的文档注释也一并更正,并给出了行为不变的论证(inspectStrandedRequests 先门控 hasSuspendedRun === false;releaseDeadRunRequests 把 null 与 paused 走同一 !TERMINAL_RUN_STATUSES.has(status) 分支)。三个套件读数:service-automation 955、plugin-approvals 458、runtime 481,全绿。

门禁账

check-test-source-alias OK(注册表未变);check-type-check-coverage OK,--re-measure 33 条无一超顶;tsc --noEmit 3 个错误全部是 nested-region-parity.test.ts 里既有的,新文件加 0(上限 5)。⛔ 未 --lower 那 270 个既有富余 —— 与 #7986 / #8081 两位 dev 的判断一致:在别人的卡里悄悄改写九个包的账本条目,会让 diff 不可复核。三次独立作出同一判断,这条可以收作车道口径了。

Ready 翻转 + merge queue 布防作为同一个动作。本卡由 PR 关闭关键词闭环,⛔ 无残留半边。

Generated by Claude Code

fix(service-automation): a durable PAUSED run is visible after a cold…

A durable PAUSED run is invisible to GET /automation/:name/runs and to run-detail after a cold restart — while remaining fully resumable #8050

Description

Symptom

Reproduction

Root cause (suspected, located)

Scope — deliberately narrow

Suggested shape (not prescriptive)

Source

Activity

hotlong commented on Aug 12, 2026

huangyiirene commented on Aug 12, 2026

串行约束

报卡质量说明(给 dev 的前提,⛔ 仍须复核而非采信)

⚠️ 本卡的真正价值在这一句,别只修列表

形状(卡里给的是建议、非规定;我同样不预选,但给三条判据)

门禁(纪律 ㉓ + ㉔)

工序(每张 mode:cloud 卡都会撞)

huangyiirene commented on Aug 12, 2026

Dev report — draft PR #8150 (#6644 L2)

Root cause: confirmed as filed, verbatim

Scope narrowing: independently verified, holds

Your three constraints

The precedence reading you asked for

?status=paused — the point of the card

Reverse verification — 8 red / 7 green, measured

What I falsified — three of my own labels

Test I edited — one assertion inverted, none deleted

ADR-0112

Gates

Nothing else falsified

huangyiirene commented on Aug 12, 2026

门禁

逐条收派发令里我立的三条判据

报卡人的收窄:独立复核过,成立

⚠️ 空过陷阱:他不但避开了,还抓出自己三条标错的 pin

rehydration 的三个决定,逐条都对

改动了一条既有断言 —— 主动申报,理由成立

门禁账

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A durable PAUSED run is invisible to `GET /automation/:name/runs` and to run-detail after a cold restart — while remaining fully resumable #8050

门禁(纪律㉓ + ㉔)

工序(每张 `mode:cloud` 卡都会撞)

`?status=paused` — the point of the card