Repository navigation
id: design: native-spill write lease protocol for hot worlds (from #2059) #2819
Description
From the #2059 investigation (hot-world sharding)
The landed investigation doc for #2059 (doc/2026-10-30T19-35-00Z_investigation_hot_world_sharding/, §3 rung 3) designs native-node spill for an indivisible hot world: a native id node runs the world's program turns while the hosted Durable Object keeps the journal of record. The doc is explicit that the journaled single-writer lease is the whole safety design — split-brain (DO and native node both accepting writes) is the catastrophic failure. The lease protocol itself was not designed. This issue designs it, so the #2046 synthesis adopts a designed protocol instead of a placeholder.
Design to produce
- Lease object: grant/revoke/handoff as journaled entries — who grants (DO? coordinator? operator?), lease term and renewal, and how the grant is recorded so both sides and any auditor can prove who held the write lease at any journal position.
- Fencing: how a partitioned or crashed ex-holder is fenced out (monotonic lease epoch checked on journal append is the obvious shape — verify or replace), and what happens to writes an ex-holder attempts after losing the lease.
- Handoff both ways: DO → native at spill start, native → DO at de-escalation (investigation §2: spill ends when the warning band clears for ~1 hour), including in-flight command treatment at the boundary.
- Failure walkthroughs: native node crashes holding the lease; DO and node partitioned while both alive; lease expires mid-turn; journal mirror lag at handoff.
Acceptance criteria
- Design recorded in a doc under
doc/and linked from this issue - Lease grant/revoke/handoff specified precisely enough to implement against
pkgs/id/src/world/'s journaled-entry patterns - Split-brain impossibility argued from the protocol mechanics, not asserted
- Anything buildable that falls out is filed as its own issue(s)
Program: ID platform expansion, epic #2056. Parent investigation: #2059. Consumer: #2046 (serverless edition v1 synthesis).
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Large: multi-day or open-ended; expect staged check-insA design proposal (design-architect): multi-step, phased, decision-bearingIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Filed by the new-issue-creator jobQueue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Oct 31, 2026
Intake review
Related: parent #2059; sibling #2818.
Notes: design-stage issue spun out of #2059; body already carries an acceptance section, so none was appended. No merged PR or commit found this run that implements this design — verified still live.
Risks: design drift if the sibling design lands first and changes shared assumptions (hot-world sharding seams) — the two designs should be read together before either is implemented.
Suggested approach: keep as design; when picked up, the design output should land as a linked implementation issue per the standing "a design is not finished work" rule.
🤖 Agent note · filed
dezren39 commentedon Nov 1, 2026
🤖 Agent note · state: picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · state: triaged — design record (queue task 5745)
Design record: native-spill write lease protocol (#2059 rung 3)
Context. The #2059 investigation (
doc/2026-10-30T19-35-00Z_investigation_hot_world_sharding, §3 rung 3) moves a hot indivisible world's execution to a nativeidnode while the hosted Durable Object keeps the journal of record — "execution moved, journal and identity did not." Consumer: the #2046 synthesis. Sibling #2818 (region-seam handoff format) shares only the journaled-entry pattern; it shuttles journal segments between shards and does not interact with the lease.The decision
The lease lives in the journal itself, the DO is the only grantor, and fencing is an epoch check on journal append. No side channel, no separate lease service: the journal is already the single ordered record both sides and any auditor read (pkgs/id: one
JournalEntryper line injournal.jsonl, fsynced per append —world_store.rs:6), so lease state derived from it can never disagree with the history it protects.The protocol
LeaseGranted { holder, epoch, term, granted_at_position },LeaseRenewed { epoch, term },LeaseRevoked { epoch, reason },LeaseHandedOff { from, to, epoch }. Epoch is monotonic: every grant/handoff increments it; it never resets across the journal's life. Holder identity is the node's world identity (the same key that signs its journal-visible actions).LeaseGranted. An operator can forceLeaseRevokedthrough the DO, never by editing the native node.LeaseHandedOff { epoch+1 }as its last act as writer, then stops accepting writes. The native node begins appending only after it has replayed the journal through the handoff position. Handoff native → DO (de-escalation, warning band clear ~1 h per id: investigate: hot-world sharding — when one world outgrows one Durable Object #2059 §2**):** symmetric — the node appends its final entries, requests handback, the DO appendsLeaseHandedOff { epoch+1 }and resumes.Split-brain impossibility (argued from the mechanics)
Split-brain requires two writers appending accepted entries concurrently. An append is accepted only if (1) it lands in the one journal and (2) its epoch equals the journal's latest lease epoch. The journal has a single append order (one JSONL file, fsynced per append, on the DO side of record). A handoff changes the current epoch exactly once, at one journal position. Before that position the old holder's epoch is current and the new holder's is not yet granted; after it, the reverse. There is no position at which two epochs are both current, and no append path that skips the journal — so two accepted writers at once would require the journal itself to fork, which is the one thing the DO-of-record construction excludes. A partitioned ex-holder can attempt appends; each is rejected-and-journaled (or cannot reach the journal at all), which is an availability event, not a consistency one.
Failure walkthroughs
LeaseRevoked { reason: expiry }thenLeaseGranted { holder: DO, epoch+1 }and resumes writing. The dead node's unjournaled work is lost by definition (it was never in the record); its journaled work stands.Invariants
Phased implementation packets (enqueued, in order)
repo-health:v1:fix:developing-today/code:issue:3803repo-health:v1:fix:developing-today/code:issue:3804repo-health:v1:fix:developing-today/code:issue:3805Question for Drew
None that blocks: grantor authority (DO-only) and the default lease term are the two judgement calls; the record fixes DO-only as a safety property (a second grantor reintroduces the split-brain the protocol exists to exclude), and the term is an operational default for phase 1 to set (recommended: 30 s term, renewal at 10 s — long enough to ride out a GC/network stall, short enough that a crashed holder stalls writes for under a minute). Flagging them here rather than stalling on them.
🤖 Agent note · triage task 5745 · lane B · 2026-11-01
6 remaining items
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 6822 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 3, 2026
🤖 Agent note · state: picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · state: triaged — design gate dispositioned
doc/2026-11-01_design_id2819.mdis present on main. id: design: native-spill write lease protocol for hot worlds (from #2059) #2819 remains OPEN to track implementation phases id: lease protocol phase 2 — journaled lease entries + epoch fencing in the world journal (#2819) #3804 and id: lease protocol phase 3 — spill handoff both ways + failure-walkthrough tests (#2819) #3805; closing the parent would be premature.