id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency

Source: the TigerBeetle investigation (on #2040). Cluster size cannot be changed after formatting, so this decision gates any deployment: 3 replicas tolerate ONE failure; the vendor production recommendation is 6 across 3 sites (2 per site, sites within a few ms); cross-cloud commits pay inter-site RTT on every transfer. Compare the three-machine idea (maybe one per cloud) against the recommendation with failure cases and latency numbers; end with a recommendation. Specialist-grade: the ramifications are the point.

Program: Drew's 2026-10-28 directive (full text: hatch-personal memory/topics/id-program-2026-10-28.md). Parent epic: #2056.
Working pattern: parallel investigations, then the named synthesis issue merges them into a full design. Route guidance: Workhorse by default; Specialist/Summit for synthesis and ramification-heavy work. A result that says "done" is not the endpoint — the orchestrator reviews it and may commission another round. Deliverables: a markdown design/investigation doc (mermaid/SVG diagrams welcome) linked from this issue, plus follow-up issues for anything buildable. Build a proof of concept where the design calls for one, but do NOT build the permanent feature on top of an unreviewed PoC.

Acceptance criteria

A markdown investigation doc is linked from this issue comparing 3-replica vs 6-replica TigerBeetle topologies, including site placement options.
Failure cases are enumerated for each option (failures tolerated: 3 replicas = 1, 6 across 3 sites = vendor production recommendation) and commit-latency numbers are given, including cross-cloud inter-site RTT cost per transfer.
The doc ends with a single recommendation, since cluster size cannot be changed after formatting and the decision gates any deployment.
Anything buildable that falls out is filed as follow-up issues linked here.

Parent epic: #2056 — ID platform expansion (2026-10-28 directive). Filed as batch 2 from PREPARED-ACTIONS.md.

CODE-985

Intake review

Related: parent epic #2056; source investigation #2040; gates any TigerBeetle deployment and therefore #2062/#2063/#2065.

Notes: flagged Specialist-grade in the body ('the ramifications are the point') — this is the one irreversible decision in the currency track, so depth beats speed here.

Risks: latency numbers need a stated measurement basis (vendor figures vs measured RTT between candidate sites); an ungrounded comparison would not settle the decision.

Suggested approach: Specialist-grade investigation per program route guidance; end with one recommendation, not a menu.

— Fleet monitor intake pass (protocol §5): issue verified still live this run (filed within the last hour; no merged PR or duplicate resolves it), labels normalized, meta/enriched applied.

Top priority — ID program wave-1. Drew directed this be picked up in the next runs (2026-10-29): enqueued directly as triage task 3366 in repo-health-code-pilot at queue priority 20 (only breakage-class work at priority 10 outranks it). Wave-1 is the ungated investigations and designs; syntheses and PoCs stay gated on their named inputs. Investigation route: Workhorse.

🤖 Agent note · parked

Agent: queue worker (repo-worker-code), lane C
Session / ID: queue task 3366
Found via: queue claim (resume)
Task: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 verified ready — CI green on head e3ac636, all 4 Codex P2 threads resolved/outdated, follow-up id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292 filed — and merged into the merge queue (position 6, AWAITING_CHECKS at 13:32Z). Parking instead of holding the claim on the queue.
Expectation: resume: confirm PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 merged, tick the acceptance boxes on id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064, then complete task 3366 as triaged.
Links: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 · follow-up id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292

🤖 Agent note · parked

Agent: queue worker (repo-worker-code), openchamber scheduled lane
Session / ID: queue task 3366
Found via: queue claim (resume from merge_ready)
Task: Resumed at merge_ready: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 is still open, MERGEABLE/CLEAN, and in the merge queue at position 5 (AWAITING_CHECKS). It has not moved, because repo CI is stalled: 0 runs in progress, 41 queued since 12:00Z, and no merge to main since 10:57Z. The merge_group run for this PR (40883023085) has sat queued with no runner pickup since 13:33Z. Nothing is wrong with this PR — it is waiting on the repo-wide outage tracked in ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599, where I posted the fresh evidence.
Expectation: resume when main is merging again: confirm PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 merged, tick the acceptance boxes on id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064, then complete task 3366 as triaged. Follow-up id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292 (provisioning, RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal) is already filed and open.
Links: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 · follow-up id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292 · CI blocker ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599

🤖 Agent note · parked (merge queue stall)

Agent: queue worker (repo-worker-code), lane C
Session / ID: queue task 3366 (triage)
Task: Resumed to confirm PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 merged. Fresh state: docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 was REMOVED from the merge queue at 14:11Z (during the repo-wide CI stall, ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599 — no main merge since 10:57Z, 29 runs queued at check time) while still OPEN, MERGEABLE/CLEAN, all its checks pass/skip, Codex review completed with no findings. Direct merge is refused (merge strategy is set by the merge queue), so I re-enqueued it: now position 6, AWAITING_CHECKS.
Resume: when PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 merges, tick the four acceptance boxes on this issue (the investigation doc lands with the PR) and complete task 3366 as triaged. If it is removed from the queue again while the stall persists, re-enqueue rather than re-investigating.
Links: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291

🤖 Agent note · triaged / done

Agent: queue worker (repo-worker-code), openchamber scheduled lane
Session / ID: queue task 3366, repo-health-code-pilot (resume from awaiting_ci)
Found via: queue claim (5th resume on this task — all five were the same wait)
Task: Closing out triage. Everything the issue asked for exists: findings + single recommendation are posted above (6 replicas, 2 per site × 3 near sites), the doc is in PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291, follow-on build work is id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292, and the four acceptance boxes are ticked. The doc PR has been review-clean since head e3ac636 — required check green, all four Codex P2 threads resolved, Codex re-review with no findings.
Why it is not merged: the repo-wide Actions stall (ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599) — no merge to main since 10:57:51Z, every merge_group run stays queued with no runner pickup (this PR's run 40891482818, created 14:26:29Z, still queued at 14:45Z). The main: merge queue pilot ruleset has no bypass actors, so direct/admin merge is refused. Re-enqueued and at position 5 at 14:45Z; entries get evicted ~30 min after enqueue while the stall lasts.
Expectation: the doc lands via the new shepherd task (repo-health:v1:pr:developing-today/code:pr:2291, enqueued 14:42Z), which re-enqueues docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 if the queue dropped it and merges it under the standing criteria. Leaving this issue open only for the program review round the directive describes; nothing queue-side is outstanding.
Links: PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 · follow-on id: build: TigerBeetle cluster provisioning — RTT mesh gate, 6-replica format, dead-replica runbook + rehearsal #2292 · CI blocker ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599

🤖 Agent note · done

Agent: repo-worker-code, openchamber scheduled lane
Session / ID: queue task 3506
Found via: queue claim; pickup and completion recorded together because the outcome already existed.
Task: Verified PR docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291 merged on 2026-10-30 at commit ec0573c; issue id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064 is closed. Completed the shepherd task as already done.
Links: docs(id): TigerBeetle cluster topology investigation — 6 replicas / 3 near sites (#2064) #2291

Navigation Menu

Repository navigation

id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064

Acceptance criteria

Intake review

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

id: investigate: TigerBeetle cluster topology decision — 3 vs 6 replicas, site placement, commit latency #2064

Description

Acceptance criteria

Activity

dezren39 commented on Oct 29, 2026

developing-today commented on Oct 29, 2026

dezren39 commented on Oct 29, 2026

Intake review

dezren39 commented on Oct 29, 2026

19 remaining items

dezren39 commented on Oct 29, 2026

dezren39 commented on Oct 29, 2026

dezren39 commented on Oct 29, 2026

dezren39 commented on Oct 29, 2026

dezren39 commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions