Repository navigation
Query ranking: local variables win seeds, answers tunnel into one community, and confidence/weight never affect retrieval #2384
Description
Version: graphify 0.9.7 (pip graphifyy), macOS, Python 3.11
Reproduction (real graph: 2,682 nodes / 3,218 links, code + docs corpus)
graphify query "how does reconcile decide contradictions"
Expected: the answer should reach src/reconciliation.ts (the module that
actually decides contradictions in this repo).
Actual: the top seed is a local variable literally named reconcile
inside a benchmark helper (docs/research/context-benchmark/runner/ inject-stale.mjs), and all 25 returned nodes come from that one file /
community. src/reconciliation.ts never appears. The whole answer is a
tunnel into a test utility.
Root causes, from reading serve.py:
- Seed scoring has no notion of node kind — an exact-label match on a local
variable outranks substring matches on files/classes, so any incidental
local with a lucky name hijacks the query (_score_nodestiers, tie-break
by degree). - Seeds and expansion have no diversity constraint — when the top seeds land
in one community, BFS depth-2 never escapes it, so one file supplies the
entire budget. - Edge
confidence(EXTRACTED/INFERRED/AMBIGUOUS) and the storedweight
field are display-only: nothing in seeding, expansion, or output ordering
consumes them, so an AMBIGUOUS edge ranks identically to an EXTRACTED one.
Proposals (independent; any subset helps)
- Node-kind priors in seed selection. Sort seeds by
(match_tier, kind_prior, degree)with a prior like
file > class/function > variable. A local variable is almost never the
right entry point for a natural-language question. - Community-diversity caps. Require top-K seeds to span more than one
community when matches exist elsewhere, and cap output nodes per
file/community, so a single lucky file cannot consume the whole budget. - Confidence/weight-aware ranking. Use weighted degree in expansion and
output ordering, with a factor per confidence class (e.g. EXTRACTED 1.0,
INFERRED 0.5, AMBIGUOUS 0.25) multiplied by the existingweightfield.
This makes the audit-trail labels actually shape retrieval, and it gives
external graph producers (we inject an intent layer from an architecture
model, with evidence-verified links) a sanctioned way to mark
higher-trust edges instead of relying on raw degree.
Happy to provide the full graph.json that reproduces this, or test a branch.
Activity
electrumnz commentedon Sep 21, 2026
Corroborating reproduction from a different corpus, posted by Devin (Cognition's coding agent) on behalf of the electrumnz/KeeceTakeOffApp maintainer, who asked me to report it.
Version: graphifyy 0.9.64, Windows 11 and Linux (Claude Code web container), Python 3.14 / 3.11.
Graph: ~50k nodes / ~118k links, code + markdown headings (tree-sitter only:
extract --code-onlythenupdate).graphify query "RLS"picked start nodes['rlStore', 'rls_enabled_no_policy (INFO)', 'RLS implication — the hard prerequisite']—rlStoreis a TypeScript local in an unrelated route. The graph contains a document node whose label is literallyBusiness Units / multi-company + company-scoped RLS (migration 108+)(iddocs_rules_91_business_units_multi_company_company_scoped_rls_…); it never appeared in the top 10. Same shape as your item 1: a substring/prefix hit on a code identifier outranks a whole-token hit on a heading.Second case, same run:
query "_watch_dead_roles _heartbeat_age_minutes last_sweep_at sweep_roles"split the snake_case terms and seeded on the fragments — start nodes includedAT(fromlast_sweep_at) andROLES, pulling in three unrelated apps.query "watchdog"alone seeded on a degree-1 test stub_watchdog()intests/test_sweep_route.py.A local workaround that fixed both for us, in case it's useful as a data point for proposal 1: whole-token-only label matching, +2 for
file_type == "document", ×0.5 for paths matching test/e2e patterns, degree as tie-break. With that, "RLS business unit" ranks the rule-91 heading in the top 3 and "stale watchdog" ranksrun_stale_watchdog()first with no test files above it. Happy to share the graph.json or a redacted node/link sample.