Skip to content

design(atlas): storage growth model — churn × retention simulation calibrating v1 retention defaults #6061

Open

Description

@dezren39

Parent: #6019 (semi-automated ingestion operations). Research source: prior-art doc ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md (2026-11-04). Follow-up 43 (Cluster 7). Consumer: the retention rule engine (sibling under #6026).

Finding (research, in substance)

Versioned-corpus growth ≈ Σ over sources of (churn bytes/day × avg versions retained). With span reuse, a version's marginal storage is only its new spans + manifest — so the model's inputs (per-source churn, reuse ratio, version counts) are all measurable from the ingest ledger (sibling cost-ledger design under #6019) after a few weeks of operation.

Scope

Done looks like

A growth model + the calibration procedure that turns ledger data into retention defaults for #6026's rule engine.

Activity

  1. added
    area/infraInfrastructure, deployment, and operations
    kind/designA design proposal (design-architect): multi-step, phased, decision-bearing
    on Nov 5, 2026
  2. developing-today commented on Nov 5, 2026

    @developing-today
    Contributor
  3. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  4. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-y
    • Session / ID: task 10426
    • Found via: queue claim
    • Task: Triage of issue design(atlas): storage growth model — churn × retention simulation calibrating v1 retention defaults #6061 (design: storage growth model — churn × retention simulation) was claimed by lane code-y; the execution-lane assignment routed it to the opencode lane, so it is handed off — the laptop claimant (opencode-claimant) launches the bound opencode session that drives task 10426 to its terminal state.
    • Expectation: claimant binds a session to task 10426; triage per the task packet.
    • Links: task 10426 (queue repo-health-code-pilot)
  5. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 5, 2026
  6. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-o
    • Session / ID: task 10426
    • Found via: queue claim
    • Task: Triage issue design(atlas): storage growth model — churn × retention simulation calibrating v1 retention defaults #6061 (design(atlas): storage growth model) — assigned to the opencode execution lane again on re-claim (previously handed off at 2026-11-05T12:58Z, claimant had not launched before the park cycled); handed off to the laptop claimant, not worked by this lane. Issue verified OPEN this shift, no hard-stop labels.
    • Expectation: claimant launches the bound opencode session for task 10426.
    • Links: n/a
  7. 14 remaining items

  8. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    on Nov 5, 2026
  9. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  10. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  11. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  12. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  13. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 5, 2026
  14. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  15. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Triage result — task 10426 / issue #6061

    Verdict: design-only; ready for the design stage. Revalidated on 2026-11-05: #6061 is OPEN with no hard-stop labels. Parent design #6019 and consumer design #6026 are OPEN; #6043 is CLOSED by merged PR #6269. GitHub search returned no matching open PR for the storage-growth/retention design. This is not a runtime defect: no reproduction, code change, or code tests apply. #6061 itself is the design handoff, so no duplicate queue task is needed.

    Evidence and bounded route: the current Atlas sizing document is a point-in-time baseline, not the requested churn/retention model: 107,639 chunks, 3,072 raw bytes/vector, 4.8 GB total, and 1.15 GB in messages (baseline). The same inventory measures a 160,576 B/row vector-index shadow (~17.3 GB), with the projected database at ~22.1 GB against 24 GB included (~1.9 GB headroom); validate that unusual amplification rather than assuming it applies to future versions (C1). The staging design permits unlimited detail before promotion and archives closed staging databases, but does not specify archive expiry; model this lifecycle separately from retained corpus versions (staging).

    Design #6061 around measured per-source new/changed bytes per period × retained-version duration, adjusted for span reuse, manifests/metadata, index amplification, and hot/cold storage. Use the per-batch ledger in #6058 as the calibration source; it will take weeks of operation to provide representative rates, so define the model and calibration procedure now but keep initial defaults conservative (retain more) until data matures. Project against #6019's plan envelope and include staging archives separately. Coordinate the retention inputs with #6026 and its #6041 dry-run rule-engine work; keep cold-tier movement ahead of any deletion by at least one full retention cycle. The merged #6043 decision record confirms retention requires both a named trigger and a class rule; PITR is not a substitute. Coordinate delta/reuse assumptions with #6028.

    The remaining blocker is only measured recalibration: representative ledger observations need to accrue. No destructive retention action should be enabled as part of this design triage.

    🤖 Agent note · done (triaged)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/infraInfrastructure, deployment, and operationskind/designA design proposal (design-architect): multi-step, phased, decision-bearingtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions