Repository navigation
task(atlas): classification revalidation scan — verify nothing sensitive sits at the wrong tier #6021
Description
Parent: #6018 (Atlas ingestion program). Statement S6 of the 2026-11-04 brief. This is a task to execute, not only a design.
Task (Drew, in substance)
Revalidate that there is nothing sensitive in the wrong classification level. Atlas is tiered — internal (28 shards / 107,001 chunks) vs secret (exactly 1 shard / 638 chunks), with the standing rule that secret material exists ONLY in the 2 secret DBs. Corpus growth since the tiers were established (shard restoration 2026-11-04: 31 new TursoDB shards, all counts verified) makes a fresh, deliberate revalidation pass necessary: ingestion has many sources, and a single mis-tiered secret in an internal shard is the failure this task exists to catch.
Scope
- Sweep the internal tier for secret-class content: credential-shaped strings, private keys/tokens, personal/financial/medical details, anything matching the secret tier's inclusion profile. Combine pattern scanning with a classifier pass over samples + over chunks nearest the secret tier's content profile (vector-neighborhood check: internal chunks embedding-close to secret chunks get human/agent review).
- Verify the converse boundary: secret-tier contents are actually secret-class (no large volumes of mundane content inflating the most-protected tier), and confirm secret material exists in exactly the 2 secret DBs and nowhere else (including the production libSQL monolith and local replicas/caches).
- Check the staging path (#5669) and recent ingestion runs for tier-assignment mistakes: which source classes fed each tier, and does any source class straddle tiers without a rule?
- Relate findings to the classification manifest work (#5410) — the manifest is the natural home for the per-source tier rules this scan validates against.
Handling rules
- Findings are handled at the SECRET tier's sensitivity: the report names counts, classes, and locations (DB + chunk IDs), never reproduces secret content in this issue, chat, or logs.
- Any confirmed mis-tiered chunk gets a remediation path (move + re-embed at the correct tier, purge from the wrong tier incl. replicas) filed as a follow-on before this task closes.
Done looks like
A dated revalidation report (counts scanned per tier, methods, findings by class, remediations filed/done) linked from this issue, and a recommendation on recurrence (one-off vs folded into the ingestion-operations design under #6018 as a standing gate).
Activity
developing-today commentedon Nov 5, 2026
CODE-1687
dezren39 commentedon Nov 5, 2026
Research design for this scan: three layers, quarantine-upward, enforced at fan-out
Cluster 5 gives this task its method. The scan layers by cost:
Plus a sampled human audit per round to estimate the scanners' miss rate instead of assuming it.
trufflehog warning (decision record filed as #6050): trufflehog is AGPL-3.0 — it collides with Drew's 2026-10-24 adoption hard filter ("NOT AGPL, NOT GPL") — and its verification mode is on by default and sends candidate secrets to vendor APIs, which inside a secret-tier scan would exfiltrate the very material being hunted. Gitleaks-first; trufflehog only
--no-verification, if ever.Handling: findings go to a quarantine state — tier-locked upward (treated as secret until cleared), never silently deleted — with rule + context attached for review; allowlists carry reasons and expiry (a stale allowlist is how leaks get grandfathered). Redact-and-forget is a dead end: a confirmed live credential also emits a rotation task for its owner. And the boundary this task verifies must be enforced at fan-out, pre-merge — a tier label in a metadata column is advisory; a secret-tier chunk must be structurally unreturnable to an internal-tier query, including via rescore and via span-join reconstruction (conservative rule for spans reused across versions: span tier = max tier of its source contexts — design decision flagged for the follow-ups). This corpus is known to contain at least one real instance of the failure class (a pasted third-party API key found in the conversation corpus by an earlier gap pass).
Wave-2 children filed under this issue
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)dezren39 commentedon Nov 5, 2026
Triage outcome — task 10388 (2026-11-05, lane code-p)
Verdict: CONFIRMED as an execution task — method already designed (Drew's 2026-11-05 research fold-in); the executable first wave is routed now, the dependent layers stay sequenced behind their wave-2 children.
Fresh verification (2026-11-05): issue #6021 OPEN, labels task/triage + queue/queued + area/infra + kind/enhancement — no hard-stop labels. Parent programme #6018 OPEN; staging-path #5669 OPEN; manifest relation #5410 was triaged earlier today (task 10035) with its design task enqueued. Wave-2 children all OPEN: #6048 (Presidio NER pass design), #6049 (personal-inventory pattern pack), #6050 (trufflehog AGPL/verification ruling), #6051 (revalidation cadence design). No scan has been executed yet and no revalidation report exists — searches return only this issue family.
What can execute now vs. what is sequenced:
--no-verification, if ever — its verification mode must never run against this corpus).Handling rules restated for the executing worker (from the issue + fold-in, non-negotiable):
Routing: execution follow-up enqueued for Layer 1 + boundary verification, ending in the dated revalidation report (counts scanned per tier, methods, findings by class, remediations filed) linked from this issue. Layers 2–3 execute under #6048/#6049 when those settle; this issue stays open as the tracker until the full done-criteria are met.
agent-note:v1
{"kind": "triage", "task": 10388, "issue": 6021, "state": "done", "outcome": "confirmed_execution_routed_layer1", "evidence": "parent #6018 and #5669 OPEN; wave-2 children #6048-#6051 all OPEN; no scan executed or report exists; method fixed by 2026-11-05 research fold-in on the issue"}
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
97 remaining items
dezren39 commentedon Nov 6, 2026
🤖 Agent note · parked
atlasmonolith contains 107,639 chunks, including 638 rows labeledsecret-bearing; this is a boundary discrepancy to validate with the required content scan, not yet a content-level finding.reproduced, branchqueue/10513-6021-atlas-tier-scan, worktree/home/user/git/fleet/10513. Run the full local deterministic scan across both tiers, production monolith, internal-content store, and local Atlas replica/cache DBs; output only counts/classes/DB+chunk IDs, then check staging source-to-tier rules and file required follow-ons before completion. Gitleaks 8.30.1 is available locally; no scan has run yet.dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
Handed off: task 10513 (issue #6021) is assigned to the opencode execution lane. Lane code-c claimed it, revalidated issue #6021 as OPEN with no hard-stop labels and no open PR, confirmed assigned_lane=opencode, and checkpointed stage handed_off. The laptop claimant launches the bound opencode session; this lane does not work the task directly. Prior progress on task 10513: metadata revalidation complete, branch queue/10513-6021-atlas-tier-scan and worktree on the laptop recorded, scanner setup in progress, no scan run yet. Attempts 9/16.