Repository navigation
Define code/config embeddings: unit, metadata, usage — and whether they share the prose embedding dump #1518
Description
Define "code embeddings": what they are, how they're used, same dump as prose or not
Problem
The fleet's semantic index covers prose memory only. Code and infrastructure-defining config files are invisible to it. Concrete miss (2026-10-27): infra/dns/dns_config.yaml — the YAML config behind the legacy OpenTofu DNS project — was never embedded; memory held only prose about the project, so a semantic lookup for the config itself found nothing (see dezren39/darwin-nix#974).
Drew's rule: configs of this class (infrastructure-defining YAML and peers) qualify for embeddings by default.
Design work this issue owns
-
Definition. What is a code/config embedding entry?
- Unit of embedding: whole file, symbol/block, config section, or chunked windows — and how that differs per artifact type (YAML config, Terraform, source code).
- Required metadata: repo, path, commit SHA, language/kind, embedding timestamp, source-of-truth pointer.
- Freshness: what triggers re-embedding (commit, deploy, manual), and how stale entries are marked or evicted.
-
Usage. How the fleet actually consumes them:
- Semantic retrieval over configs+code during agent work ("where is X defined", infra and routing reviews).
- Cross-referencing prose memory against the artifacts it describes (the prose said a DNS project exists; retrieval should reach the config itself).
- Drift support: prose claims vs current config content.
-
The same-dump question (Drew, posed directly): why can't code embeddings live in the same dump as the prose ones?
- Default hypothesis: ONE dump/index, entries discriminated by a
kindtag (prose | code | config). Separation must be argued, not assumed. - Weigh concretely: embedding-model fit (prose-tuned vs code-aware), chunk-granularity mismatch, retrieval pollution both directions, differing update cadences (append-only memory vs git commits), dump portability/export, and access scope.
- Return a verdict with reasons; if same-dump loses, specify exactly what forces the split.
- Default hypothesis: ONE dump/index, entries discriminated by a
-
Qualifying criteria + first ingestion. Codify which artifacts qualify (starting rule: infrastructure-defining configs qualify by default) and land a minimal ingestion path whose first entry is
infra/dns/dns_config.yaml, verified retrievable by a semantic query afterwards.
Done when
A written definition exists (unit, metadata, freshness), the same-dump question has a reasoned verdict, qualifying criteria are recorded, and at least the DNS config is embedded and provably retrievable.
Acceptance criteria
- A decision is recorded on this issue for each of the three questions: (1) the embedding unit (whole file vs declaration), (2) whether retrieved config code is re-ranked as config or as prose, and (3) whether the config and prose embedding budgets are shared or separate.
- Each decision names the failure it accepts (e.g. semantic neighbours dropped when one declaration changes) and the evidence behind it.
- If the decision requires implementation, a follow-up implementation issue is filed and linked from this issue in the same pass — the design does not sit without a next action.
Activity
developing-today commentedon Oct 27, 2026
CODE-579
dezren39 commentedon Oct 28, 2026
Intake review
Related: No directly related open issue found; this sits next to the code/config corpus work the body cites (Semble
chunk_idmetadata) and the memory/embedding conventions recorded in the repo's design docs. No merged PR decides these questions — they are genuinely open design choices, so this item is routed touser/reviewfor a Muse/Drew decision rather than the queue.Notes: The issue arrived with no labels at all; a full intake set was applied. The three questions are well-posed and each names the risk that makes it load-bearing (re-embedding blast radius, re-rank domain, shared-budget starvation). It is a decision item, not an implementation task — effort/m reflects the design pass, not a build.
Risks: Deciding the unit question implicitly (by whatever the first implementation happens to do) locks in the re-embedding cost profile without the trade-off ever being weighed.
Suggested approach: Decide (1) first — it constrains the other two. Record each decision on the issue with its accepted failure, then file the implementation follow-up in the same pass if one is needed. An acceptance-criteria section was appended to the body (original preserved verbatim above it).
Labels normalized by the fleet monitor (intake enrichment): kind/question, priority/normal, effort/m, status/needs-triage, source/human, user/review, meta/enriched.
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
4 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
queue/2602-1518-embedding-design, worktree/home/user/git/fleet/2602; prior checkpoint wasverified.dezren39 commentedon Nov 1, 2026
🤖 Agent note · triage verdict — escalated for Drew decision
3eafbe66eb460a42322f5d61e0b1457423b9245. Repo-owned mcpx semantic search builds/ranks MCP tool-schema candidates (pkgs/mcpx/internal/daemon/registry.go:632-675,694-794); its vector store is onlykey+vector(pkgs/mcpx/internal/catalog/vector.go:126-156). The embeddings API imports vectors but does not index source text (pkgs/mcpx/internal/daemon/routes_embeddings.go:37-95,111-127). DNS config is generated and consumed as Terraform input (infra/dns/generate_dns_config.sh:14-35,infra/dns/main.tf:22-24), with no in-repo semantic-file ingestion or retrieval path..mcpx.json:70-94configurescodebase-memoryand disablescodedb; their ingestion rules are external, andpkgs/mcpx/ASSESSMENT.md:114-118records a codebase-memory startup failure. This does not prove the DNS file is indexed elsewhere.kindmetadata and source identity; use whole-file units for small configs and declaration-sized chunks for large code/config; keep retrieval quotas kind-aware to limit cross-domain pollution. Please confirm or revise those defaults and identify the authoritative index owner.