Repository navigation
Phase 5: persisted incremental tag index #183
Description
The tag fold and index are rebuilt on every call by scanning all events with a signature check (src/tag_claims.rs, src/tag_index.rs). Needed before P5.5 search. Persist the fold and update it incrementally as events are kept.
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Fully doable remotely: design, config, code, cloudQueue or user escalation stateHas a queued task in the Supabase agent queueFiled by the new-issue-creator jobQueue or user escalation stateQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
dezren39 commentedon Oct 24, 2026
id train triage (2026-10-24): too-big-needs-design — persisted incremental tag index; the fold/index is rebuilt each call by full scan + signature check (src/tag_claims.rs, src/tag_index.rs).
dezren39 commentedon Oct 29, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 29, 2026
🤖 Agent note · done
Verdict: CONFIRMED — the fold and index are rebuilt on every call at HEAD
a3cdf6c9a35df1ff8b88b7a02d62e5c04ca4c94a, and nothing is persisted.Evidence (fresh
gh apireads of current main, not the issue text):pkgs/id/src/event_store/claims.rs—tag_claims()iterates the wholeEVENTStable, decodes every stored event, builds the parents map for every event, and foldscompacted_claims;tag_index()is exactlyTagIndex::new(&self.tag_claims()?)— a full rebuild per call.pkgs/id/src/search.rsmodule header states it outright: "The index is rebuilt on each call, as [EventStore::tag_index] rebuilds it (Phase 5: persisted incremental tag index #183). Nothing is persisted here." — so P5.5 search, the consumer this issue said it was needed before, has already landed on top of the rebuild behaviour; the cost now grows with every event the store keeps.pkgs/id/src/tag_index.rsmodule header: "The index is local and rebuilt from the store's live claims."TagIndexis two in-memoryBTreeMaps built byTagIndex::new(claims).No competing work: no open PR implements a persisted tag index (open PR #1806 is the unrelated author-index search).
Routing: the 2026-10-24 triage comment on this issue called this "too-big-needs-design", so the follow-up packet pins the design instead of leaving it open: persisted fold state in the store's existing redb database (parents / live claims / α+Ω index tables + a meta row with schema version and an EVENTS-position watermark), an incremental
fold_new_events()path that folds only rows past the watermark while preserving the causal-past clear semantics exactly, and the existing full scan kept as the rebuild/reference fallback on version mismatch or corruption. Enqueued as queue task 2974 (small_fix, deduperepo-health:v1:fix:developing-today/code:issue:183, packetqueue-packets/packet-183-fix.json). This issue stays open and is labelledqueue/queuedfor that task.dezren39 commentedon Oct 29, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 29, 2026
🤖 Agent note · blocked
packet_defect— the packet's pinned design cannot satisfy its own acceptance criteria at current HEAD (ce5c72b2, laptop primary, fetched 2026-10-29). Not a verdict on the issue: Phase 5: persisted incremental tag index #183 itself stays CONFIRMED and open.Defect: the single scalar EVENTS-position watermark is unsound.
pkgs/id/src/event_store/mod.rs:1102-1106—position_key(author, seq)isauthor[32] || seq_be64, and the module header states the same. EVENTS rows sort by author, then seq. A watermark = "highest EVENTS position folded" therefore orders authors, not time: fold author0xFF…'s events, set the watermark there, then author0x00…ingests atag.set— its position is below the watermark, a scan of "rows with position > watermark" never sees it, andtag_claims()permanently diverges from the full fold. That breaks the packet's own acceptance criterion 2 (incremental result == full-rebuild result after any interleaved sequence).pkgs/id/src/event_store/ingest.rsrefresh()re-resolves the whole store and rewrites theAPPLIEDtable wholesale on ingest — a fork or a satisfied dependency flips applied status for events already stored (a previously-unappliedtag.clearbecoming applied changes live claims with zero new tag rows).PROOF_HEADERSrows likewise extend the parents map for clears already folded, andEventStore::prune_behind_cut(mod.rs:540) deletes EVENTS rows, which no>scan detects.TagIndexat HEAD has a third map,by_author(id phase-5: search by author/kind and multi-term AND are designed but not built or tracked #800, seepkgs/id/src/tag_index.rsheader). Persisting live claims and building the index in memory covers it — worth stating explicitly in the corrected packet.What a corrected packet needs to pin: per-author (author → highest folded seq) watermarks plus a way to catch applied-status flips, proof-header additions, and prunes — or drop the read-time scan entirely and maintain the persisted fold inside the ingest/
refresh()write transaction, where every mutation already passes. The worker instruction (§2/§7) forbids improvising a different design under a pinned packet, so this is blocked aspacket_defectfor re-packeting rather than silently redesigned in-flight.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 2974 (small_fix, blocked, priority 30) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03