design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers

Parent: #6018 (Atlas ingestion program). Statements S16–S20, S29, S30 of the 2026-11-04 brief.

Requirement (Drew, in substance)

Atlas must understand content versioning well enough that:

New versions of files are first-class: a changed file lands as a new version of the same object, never an overwrite and never an unrelated new object.
Renames are understood: a renamed file's versions continue the same lineage. For git sources this is simple if commits are done in the right way (renames recorded as renames instead of delete+insert) — the design may set commit-hygiene expectations for source repos, and must handle non-git sources (session stores, doc trees) where lineage has to be inferred (segment association, sibling design under #3125).
Old and new are both kept, but resolution defaults to "the max" — the newest version of every file. Search/retrieval asks for a file and gets its latest version unless it explicitly addresses history.
Old versions roll out to cold storage over time (2026-11-04 storage ruling: cold tiers → rustfs) — a lifecycle with hot → cold movement, not permanent hot residence.
Old versions are eventually removed under an explicit policy with three named triggers, any of which can retire a version:
- by date — older than a threshold;
- by increments — N newer versions exist beyond it;
- by access — it has never (or no longer) been retrieved (this trigger consumes the retrieval statistics from the sibling stats/feedback design under #6018).

Tensions the design must face honestly

#3125 carries Drew's 2026-10-31 rule: keep ALL data; nothing stripped to save space; only byte-identical dupes cleaned. The deletion policy above is a deliberate evolution of that rule (2026-11-04 brief) — the design must state which rule governs which tier/class, and default to keep unless a trigger AND a class rule both say retire. Deletion of a version must never orphan chunks whose occurrences point at it (occurrence registry semantics per atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125).
Newest-resolution interacts with fan-out search (#5755) and the parameter surface (#5774): as-of / version pinning becomes a query parameter, and full-version logging per run must record WHICH version of each file a result came from (feeds the query-trace design).
Cold rollout must preserve resolvability: a cold version still resolves (slower) until a deletion trigger fires; deletion is the only hard end.

Design must answer

The version object model: identity (file id vs version id), lineage edges (rename, edit, split/merge of files), and how chunks map to versions (shared chunks across versions recorded as occurrences, not copies).
Per-class retention defaults (code, docs, session logs, chat logs, generated reports likely differ) and who can override them.
The retrieval-side contract: default-latest semantics, explicit history addressing, and how merge/rescore treats multi-version hits for the same file (one file, best version, unless the query is historical).

Done looks like

A design doc in code (doc/): version model, lineage rules per source type, retention policy table with the three triggers parameterized per class, cold-tier mechanics, and the query-side version contract — with implementation split into follow-ons.

CODE-1692

Research findings that change this design: git records NO renames; newest-resolution is a view

Two Cluster-2 findings reshape the versioning design:

Git rename detection is a per-diff re-guess, not recorded lineage. Git stores snapshots; rename detection runs heuristically at diff time (exact matches → same-basename pre-pass → pairwise similarity, default threshold 50%, -M50%; copies need -C). Documented failure modes: rename + heavy edit in one commit drops under threshold (reads as delete+create); file splits/merges are invisible to whole-file similarity; large diffs silently hit the rename limit. Drew's instinct — rename-only commits, git mv, no simultaneous edit — makes git lineage a convention-dependent hint, and this design may set that commit discipline for our repos, but Atlas must compute and store its own lineage edges (via segment search / the dedup-edge pipeline) as first-class data, with git's -M output as one input signal. Non-git sources (chats, logs, session stores) have no git signal at all — another reason the lineage graph lives in Atlas's tables.
"Take the max" must be a query-time view, not a destructive replace. Delta/Iceberg time travel works because resolution happens at read. Newest-version resolution = a view over the lineage graph (per logical file: the version reachable from the current head edge), with explicit handling for deleted files (head = tombstone — the previous version does not resurface) and renamed paths (identity follows the lineage edge, not the path string). Destructive replacement would break the query-trace design (#6027): a served result must forever resolve to the exact embed state that was served. Related dead end, recorded in a child below: Turso PITR is whole-DB disaster recovery (≤90-day windows) — never a corpus-retention mechanism.

Wave-2 children filed under this issue

#6041 — composite age/supersede/last-surfaced policy with a dry-run reporter (no silent vacuum)
#6042 — self-describing rustfs shards + restore path
#6043 — the layer-separation record

— Wave-2 research fold-in, 2026-11-04 (prior-art doc: ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)

🤖 Agent note · picked up

Agent: queue worker (repo-worker-code), OpenChamber scheduled lane 262eb403-3347-4a7c-ba6b-e7bc263ca853
Task / ID: triage queue task 10393
Found via: queue claim
Task: triage design issue design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026 under Atlas tracker atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018.
Verified fresh: design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026 is OPEN, no hard-stop labels, and no related PR. Parent tracker atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018 is OPEN/parked. Existing research comments record git rename limitations, newest-resolution as a query-time view, and wave-2 children design(atlas): retention rule engine — composite age / supersede-count / last-surfaced policy with dry-run reporter #6041–decision(atlas): layer separation — Turso PITR is DB disaster recovery, corpus retention lives in Atlas #6043; no corresponding design doc is present on current main.
Expectation: scope the design and ordering; do not take over the parked tracker or modify data-retention policy outside the design.

Triage result — #6026

Scope: this is a design issue, not a current runtime defect. No reproduction or code change is applicable. The issue remains OPEN; no related PR or existing Atlas versioning design doc was found on current main.
Research already on the issue: Git rename detection is heuristic, so Atlas needs first-class lineage edges; newest-resolution should be a query-time view; wave-2 children design(atlas): retention rule engine — composite age / supersede-count / last-surfaced policy with dry-run reporter #6041 (retention rule engine/dry-run), design(atlas): cold-tier shard format — self-describing rustfs shards and the restore path back to hot #6042 (cold-tier shard/restore), and decision(atlas): layer separation — Turso PITR is DB disaster recovery, corpus retention lives in Atlas #6043 (layer separation) are already filed.
Ordering: use atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125 occurrence/provenance semantics for version-to-chunk lineage; coordinate query-time latest/as-of behavior with atlas core consumer wiring: fleet atlas-search fan-out over the TursoDB shard core (tier ceilings, merge, --expand hydration) #5755/atlas search: query-time parameter surface — every retrieval knob as a CLI flag / raw-query field, no hardcoded constants (extends #5741, #5668) #5774 and retrieval statistics/query traces in design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027; fold design(atlas): retention rule engine — composite age / supersede-count / last-surfaced policy with dry-run reporter #6041–decision(atlas): layer separation — Turso PITR is DB disaster recovery, corpus retention lives in Atlas #6043 into the parent design rather than opening duplicate tickets. The parent tracker atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018 is OPEN/parked and remains the source for broader Atlas scope.
Policy gate: reconcile this design with atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125’s keep-all rule. Default to retain unless both a class rule and explicit trigger allow retirement; deletion must not orphan chunk occurrences. Cold versions remain resolvable until deletion. The atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018 spend ceiling remains in force; any design option exceeding it needs Drew’s explicit approval, but no spend change is needed to complete the design.
Routing: ready for the design stage; no separate queue packet was created because design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026 is already the design handoff and the research children are linked.

🤖 Agent note · done

Agent: repo-worker-code, OpenChamber scheduled lane 262eb403-3347-4a7c-ba6b-e7bc263ca853
Task: queue task 10393 — triage design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026
Outcome: triaged and routed to the design stage; findings and sequencing are in the preceding comment. design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026 remains open; no separate queue task was needed.
Links: design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026 · parent tracker atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018

Decision record for the PITR/Atlas-retention boundary: read the record. It records Turso PITR as whole-database disaster recovery and keeps per-version lineage, cold rollout, and policy deletion in Atlas. PR #6269 will close follow-on issue #6043 when merged.

Navigation Menu

Repository navigation

design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026

Requirement (Drew, in substance)

Tensions the design must face honestly

Design must answer

Done looks like

Research findings that change this design: git records NO renames; newest-resolution is a view

Wave-2 children filed under this issue

Triage result — #6026

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

design(atlas): content versioning + retention — versions, rename lineage, newest-resolution, cold rollout, deletion triggers #6026

Description

Requirement (Drew, in substance)

Tensions the design must face honestly

Design must answer

Done looks like

Activity

developing-today commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

Research findings that change this design: git records NO renames; newest-resolution is a view

Wave-2 children filed under this issue

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

Triage result — #6026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions