Skip to content

design(atlas): cold-tier shard format — self-describing rustfs shards and the restore path back to hot #6042

Open

Description

@dezren39

Parent: #6026 (content versioning + retention). Research source: prior-art doc ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md (2026-11-04). Follow-up 12 (Cluster 2).

Scope

Per the 2026-11-04 storage ruling, old versions roll out to cold storage on rustfs. The cold format must be self-describing — a shard file that can be re-imported years later without the hot DB that wrote it:

  • Manifest: corpus/tier, source marks, version lineage summary, span table, vector model + dimensions, creation time, retention state.
  • Payload: spans + vectors in an open, documented layout (no libSQL-only assumptions).
  • Restore path: re-import into a hot shard (for a historical query or a retention reversal) with provenance intact, and the fan-out contract for cold shards (not searched by default; explicitly addressable).

Done looks like

A format spec + restore/re-import procedure design, with the round-trip (hot → cold → hot, byte-identical spans) as the acceptance shape.

Activity

  1. added
    area/infraInfrastructure, deployment, and operations
    kind/designA design proposal (design-architect): multi-step, phased, decision-bearing
    on Nov 5, 2026
  2. developing-today commented on Nov 5, 2026

    @developing-today
    Contributor
  3. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  4. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  5. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 5, 2026
  6. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-o
    • Session / ID: task 10408
    • Found via: queue claim
    • Task: Triage issue design(atlas): cold-tier shard format — self-describing rustfs shards and the restore path back to hot #6042 (design(atlas): cold-tier shard format) — assigned to the opencode execution lane again on re-claim (previously handed off at 2026-11-05T12:57Z, claimant had not launched before the park cycled); handed off to the laptop claimant, not worked by this lane. Issue verified OPEN this shift, no hard-stop labels.
    • Expectation: claimant launches the bound opencode session for task 10408.
    • Links: n/a
  7. 1 remaining item

  8. developing-today commented on Nov 5, 2026

    @developing-today
    Contributor

    🤖 Agent note · handed off

  9. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  10. added
    queue/claimedQueue worker lifecycle state
    and removed
    queue/claimedQueue worker lifecycle state
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 5, 2026
  11. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  12. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · triaged

    Triage findings (verified against current GitHub state and main, 2026-11-05)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/infraInfrastructure, deployment, and operationskind/designA design proposal (design-architect): multi-step, phased, decision-bearingtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions