Skip to content

research(atlas): boundary-drift measurement — how much a paragraph's vector/rank moves with different embedding neighbors #6038

Open

Description

@dezren39

Parent: #6028 (delta-only ingestion + offset embeds). Research source: prior-art doc ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md (2026-11-04). Follow-up 5 (Cluster 1).

Question

Span reuse assumes a span's stored vector stays valid when the span is reused in a new version. But an embedding is a function of the chunk's exact text including its window neighbors: the same paragraph embedded with different surrounding text is a different vector, and its rank for a given query can move. How much?

Scope

  • Take a sample of real corpus paragraphs; embed each in its original window and in perturbed windows (shifted boundaries, changed neighbors); measure cosine drift + rank movement on a fixed query set.
  • Deliverable is a number with a distribution: median/p95 drift, and the reuse-safety rule it implies (e.g. reuse only byte-identical spans whose recorded context matches; re-embed when context class changes).

Done looks like

A short measurement report + the reuse rule it justifies, linked from the span-schema design (sibling under #6028).

Activity

  1. added
    area/infraInfrastructure, deployment, and operations
    kind/designA design proposal (design-architect): multi-step, phased, decision-bearing
    on Nov 5, 2026
  2. developing-today commented on Nov 5, 2026

    @developing-today
    Contributor
  3. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  4. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  5. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 5, 2026
  6. 35 remaining items

  7. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 5, 2026
  8. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  9. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 5, 2026
  10. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  11. added
    status/triagedTriaged and routed (see agent note)
    and removed
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 5, 2026
  12. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Triage — task 10404 (2026-11-05)

    Verdict: valid, distinct research gap; keep #6038 open as the scoped research follow-up. The request is not implemented or superseded. Fresh checks: #6038 OPEN; no hard-stop labels; no linked PR (the only search hit, #4708, is unrelated and merged). Parent #6028 research notes list #6038 as a parallel measurement gate; the qualitative discussion does not provide the requested measurements.

    Current-main evidence: laptop main was fetched and fast-forwarded (already up to date); local HEAD and GitHub main both equal 50237a7debf6e9114122c593a15a467858d2bf9a. pkgs/atlas-search/crates/atlas-embed/src/lib.rs:149-170 embeds input text; crates/atlas-embed/tests/gate_a.rs:57-139 tests fixed probes/model parity, not changed-neighbor drift. crates/atlas-core/eval.rs:118-125 consumes precomputed document rankings; eval/README.md:61-70 says its smoke fixture is synthetic and no real held-out set is in the harness. No boundary-drift report or experiment was found.

    Scoped route: run the bounded offline experiment in #6038: sample real paragraph-bearing chunks; keep model and query set fixed; compare each target embedded in original vs perturbed windows; report cosine drift and vector-rank/top-K movement (median/p95, split by perturbation or source type), then state a reuse rule and link results from #6035. Keep corpus text private; synthetic fixtures are not evidence. This can proceed in parallel with #6035 schema design, whose byte-identical reuse/context-reference contract should consume the result. No code changes or duplicate queue packet: #6038 itself is the existing scoped follow-up issue. No tests/builds run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/infraInfrastructure, deployment, and operationskind/designA design proposal (design-architect): multi-step, phased, decision-bearingstatus/triagedTriaged and routed (see agent note)task/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions