Skip to content

feat(core,platform-objects): implement the ADR-0119 D2 migration-journal runner (sys_migration_journal + runMigrationJournal) #4617

Closed

Description

@os-zhuang

Implements ADR-0119 D2, which specifies this runner in implementable detail. Surfaced by #4612 — four migration-class consumers (ADR-0105 D13 promotion, ADR-0117 D8 ownership backfill, org lifecycle 新设/分立/合并, D10 master-data distribution #4585) each independently converge on dry-run preflight → undo journal → LIFO compensation → re-entrant forward recovery. One copy is engineering; four is platform debt.

ADR renumbered. This issue originally cited ADR-0118. That record is now ADR-0119 (docs/adr/0119-plugin-reachable-transactions-and-honest-atomic-batch.md) — it merged the same day as an unrelated ADR-0118 (非用户 actor 的平台契约), and the earlier merge holds the number. Every "D1/D2/D3/D4" below means the renumbered record.

Why a journal at all, given ADR-0034 transactions

ADR-0119 D1 makes engine.transaction() contract-reachable, but a transaction cannot be the whole answer for migration-class work:

  • a million-row backfill cannot hold one SQLite write-lock for its duration;
  • driver-memory's beginTransaction deep-clones the entire database (memory-driver.ts:595-630) — O(db) per begin;
  • ObjectQL.transaction() binds only the default driver (engine.ts:4950), so a multi-datasource migration silently commits part of its work outside the transaction;
  • a process crash (as distinct from a thrown error) defeats in-process rollback entirely — the case the issue calls out.

So the unit of atomicity is the chunk, and durability across chunks is the journal.

Deliverables

1. sys_migration_journal platform object (packages/platform-objects). Rows keyed (run_id, seq); event kinds run_started (carries the plan hash + chunk plan), chunk_started(i), chunk_done(i), compensated(i), run_done, run_failed.

2. The runner — packages/core/src/utils/migration-journal.ts, next to bulk-write.ts, typed against IObjectQLEngine (core already re-exports it):

  • Preflight dry-run. Every plan step declares a read-only validator; the runner runs all validators before any write and refuses to start on any failure. Fail-closed, matching ADR-0117 D8's enable-gate posture.
  • Chunked writes. Rows re-chunked per the bulk-write.ts discipline; each chunk's writes run inside engine.transaction(), and chunk_done(i) is written inside that same transaction so done ⇔ committed is not a race. chunk_started(i) is written autonomously before it, so started ∧ ¬done precisely means "outcome unknown".
  • LIFO compensation. On failure, walk completed chunks newest-first, running each step's declared compensate in its own transaction plus a compensated(i) marker. A compensation failure journals and halts loudly — it must never be swallowed.
  • Re-entrant forward recovery. On restart (boot reconciliation and/or os migrate resume), scan for run_started ∧ ¬run_done ∧ ¬fully-compensated and resume forward from the first chunk lacking chunk_done, per-plan policy onCrash: 'resume' | 'compensate'.
  • At-least-once + idempotency. Every forward and compensate callback receives an attempt counter; attempt > 1 means the prior outcome is unknown and the callback must recheck by natural key before re-writing. This is verbatim the contract packages/core/src/utils/bulk-write.ts already documents in its header — reuse the wording, don't re-derive it.
  • Capability gate. Refuses to start when the runtime cannot provide real transactions (same probe as ADR-0119 D4's batchData gate).

3. Boot reconciliation + os migrate resume so an interrupted run is discoverable without the operator knowing it happened.

Why an engine-persisted object, not a caller-supplied JournalStore

Recorded in ADR-0119 D2 as a decision, not an implementation detail:

  • Recovery must be re-entrant and discoverable with zero host wiring — a boot scanner needs one authoritative place to find half-run migrations. A pluggable store fragments that authority per caller, which is the ADR-0078 silently-inert failure mode: a journal nobody re-reads is a journal that does not exist.
  • The journal is data-plane state about the data plane, so it belongs inside the same backup/restore/transaction boundary as the rows it describes. A side-file journal (the ADR-0008 JSONL shape) desyncs from the database on restore — ADR-0008 solved audit, which is a different problem from recovery authority.
  • Framework-owned sys_* tables with a registered schema are the established ledger pattern (ADR-0060).

Acceptance

A migration killed mid-run is either resumable to completion or compensable to clean, with journal rows proving which — asserted by conformance tests covering: crash mid-chunk → resume completes exactly once (idempotency honored on the re-attempted chunk); crash mid-compensation → halts loudly, no silent partial; plan-hash mismatch on resume → refuses rather than resuming a changed plan against an old journal.

Once this lands, the four queued consumers drop their hand-rolled copies. Existing hand-rolled prior art to fold in or retire: ImportUndoLog (packages/rest/src/import-runner.ts:48-59), the publish revert plan (packages/metadata-protocol/src/protocol.ts:7400-7424, revertCommit at :8210).

Refs: ADR-0119 D2, #4612, ADR-0034, ADR-0087 D3, ADR-0060, ADR-0117 D8, #4585.

Activity

  1. changed the title [-]feat(core,platform-objects): implement the ADR-0118 D2 migration-journal runner (sys_migration_journal + runMigrationJournal)[/-] [+]feat(core,platform-objects): implement the ADR-0119 D2 migration-journal runner (sys_migration_journal + runMigrationJournal)[/+] on Aug 2, 2026
  2. os-zhuang commented on Aug 2, 2026

    @os-zhuang
    ContributorAuthor

    Deliverables 1 and 2 landed in #4668 (merged). Staying open for deliverable 3.

    Done:

    • sys_migration_journal — (run_id, seq) under a unique index, registered unconditionally beside sys_migration.
    • runMigrationJournal (packages/core/src/utils/migration-journal.ts) — preflight, chunked transactional writes with chunk_done written inside the chunk transaction, LIFO compensation, re-entrant recovery, attempt counter, capability gate.
    • 19 conformance tests covering all three acceptance cases, run against a fake engine implementing real rollback — without that, the assertions would also pass against a runner that never opened a transaction.
    • engineCanRollBack extracted to @objectstack/core and shared with batchData's D4 gate, so the two copies of "can this runtime actually roll back?" cannot drift.

    Remaining — deliverable 3: boot reconciliation + os migrate resume. findInterruptedRuns is the discovery primitive they consume; it is exported and tested in #4668, so the follow-up is the boot-hook wiring and the CLI command rather than new recovery logic.

    One implementation note for whoever picks that up: findInterruptedRuns treats a run as concluded on run_done, or on run_failed with every committed chunk compensated. A run_failed whose compensation did not finish is still reported — that case needs an operator, not an automatic retry, and the boot scanner should surface it rather than resume it.


    Generated by Claude Code

  3. os-zhuang commented on Aug 2, 2026

    @os-zhuang
    ContributorAuthor

    已落地,三块交付齐了

    交付项 PR 内容
    1 · sys_migration_journal #4668 平台对象,(run_id, seq) 键,六种事件
    2 · runner #4668 packages/core/src/utils/migration-journal.ts — 预检 dry-run、分块写、LIFO 补偿、可重入前向恢复、能力闸
    3 · 发现通道 #4687 MigrationRecoveryPlugin + os migrate resume + MigrationPlanRegistry

    交付项 3 是让「操作者不知道发生过」的中断运行可被发现的那一环。

    一处设计判断:启动负责发现,CLI 负责行动

    恢复是一次不可逆、可能长达数小时的生产写入。把它做成进程启动的副作用,操作者往往是从监控图上才发现的;而且启动时未必做得到——恢复需要 plan 的活回调,而拥有它的包未必在恰好先重启的那个进程里。

    所以启动只把运行报出来并给出确切命令,由命令在明确的操作者意图下行动。D2 的每-plan onCrash 策略仍然决定做什么(前向恢复还是回滚),只是不决定何时,而「何时」该由人拿。

    推迟是安全的,而且安全性正来自 runner 的可重入性:started ∧ ¬done 是持久的,一小时后这个运行的可恢复性和启动那刻完全一样。没有东西在操作者思考期间衰减。

    为什么需要 plan 注册表

    日志装不下 plan。forward/compensate 是函数,load() 读活库,都过不了进程边界——这正是日志只记 plan 哈希的原因。恢复需要由拥有这段代码的包把 plan 交回来,migration-plans 就是这个接缝。

    plan 未被任何已加载包注册的运行会被报出来,而非静默跳过:「没什么可恢复」和「拥有这段代码的包不在这里」是两个不同的事实,只有前者可以忽略。

    验收

    11 个 runtime 测试钉住了职责切分(启动不向日志写入任何东西)、操作者必须区分的三种状态(干净 / 中断 / 半回滚)与两条降级路径;2 个 core 测试覆盖注册表,含「同 id 重复注册是替换而非累积」,并以日志的 plan-hash 校验作为「拒绝对变更过的 plan 执行恢复」的兜底。


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions