Activity

1. Fion9506 commented on Jun 11, 2026

   

   Fion9506

   on Jun 11, 2026

   Author

   More actions

   Our working solution: After hitting this gap in our project, we built a post-processing alignment pipeline that bridges the two subgraphs with claude code. Here's the full writeup in case it helps others or informs a future graphify feature:

   Our Solution: Structured ID Traceability for Cross-Domain Alignment

   Status: Our working solution — validated on a real-world project
   graphify version: ≥0.5 (tested with pip install graphifyy)
   Applicable to: Any project using graphify with mixed code + documentation

   ---

   Table of Contents

   1. Problem Analysis
   2. Solution Overview
   3. Step 1: Structured ID Annotations
   4. Step 2: Post-Processing Alignment Script
   5. Step 3: Quality Audit
   6. Results
   7. Design Decisions
   8. Reference Implementation

   ---

   1. Problem Analysis

   The Gap

   graphify builds knowledge graphs from mixed codebases using two extraction pipelines:

   Pipeline	Input	Output	Edge Types
   AST extraction	Source code files	Code nodes with deterministic IDs	import, call, contains
   Semantic extraction	Docs, papers, images	Concept/document nodes with LLM-generated IDs	references, implements

   These two pipelines produce structurally isolated subgraphs. The root cause chain:

   1. extract.py:65-78  → _make_id() generates path-based IDs (e.g., auth_service_authservice)
   2. Semantic subagents → LLM generates free-form IDs (e.g., concept_Authentication_Module)
   3. build.py:107-273  → Merge requires ID collision or (basename, label) exact match
   4. Result: AST IDs ≠ Semantic IDs → zero cross-domain edges
   

   (Line numbers based on graphify v0.5.x)

   Evidence

   Metric	Typical Result
   Total nodes	1,000-3,000
   Cross-domain edges (native)	0
   Community bridges (doc↔code)	0
   Spec→Code coverage	0%

   This is not a bug in extraction quality — both subgraphs are rich and accurate within their domain. The gap is purely in the bridging mechanism.

   ---

   2. Solution Overview

   A three-component architecture that bridges the two subgraphs without modifying graphify:

   ┌─────────────────────────────────────────────────────────────┐
   │                     graphify native                          │
   │  AST extraction ──→ code subgraph                            │
   │  Semantic extraction ──→ doc subgraph                        │
   │  build_from_json() ──→ merged graph (still disconnected)     │
   └────────────────────────┬────────────────────────────────────┘
                            │
                            ▼
   ┌─────────────────────────────────────────────────────────────┐
   │  Layer 1: ID Annotations (metadata source)                   │
   │    Code: # IMPORTANT: M-01 auth | E-01,E-02 | Scene-1       │
   │    Docs: already reference M-01, Scene-1 in content          │
   │    → Captured by graphify's rationale extractor              │
   │    → NOT for natural merge, but as structured input for L3   │
   ├─────────────────────────────────────────────────────────────┤
   │  Layer 2: graphify Native (relationship network)             │
   │    Code↔Code: excellent (dense import/call graph)            │
   │    Doc↔Doc: good (semantic references)                       │
   │    Doc↔Code: ZERO — the gap Layer 3 fills                    │
   ├─────────────────────────────────────────────────────────────┤
   │  Layer 3: Post-Processing Alignment (bridge)                 │
   │    Tier 0: Deterministic → auto-write (confidence ≥0.95)     │
   │    Tier 1: Controlled fuzzy → cross-validation (0.80-0.90)   │
   │    Tier 2: Free text → report only (<0.80)                   │
   │    + TrustScore audit + auto-delete D-grade edges            │
   └─────────────────────────────────────────────────────────────┘
   

   ---

   3. Step 1: Structured ID Annotations

   Why # IMPORTANT: prefix?

   graphify's AST rationale extractor (extract.py:3483) only captures comments with specific prefixes (line numbers based on graphify v0.5.x):

   _RATIONALE_PREFIXES = ("# NOTE:", "# IMPORTANT:", "# HACK:", "# WHY:", ...)

   Using # IMPORTANT: ensures the annotation is captured as a rationale node in the graph, making it available for post-processing.

   Annotation format

   Place in each module's __init__.py, after the module docstring, before the first import:

   """Authentication module — public interface exports."""
   
   # IMPORTANT: M-01 auth | E-01,E-02,E-03 | Scene-1
   
   from .service import AuthService
   from .models import User, Session
   
   __all__ = ["AuthService", "User", "Session"]

   Format: # IMPORTANT: <module-id> <module-name> | <entity-ids> | <scene-ids>

   Component	Purpose	Example
   M-01	Module ID — links docs to code module	M-01 through M-NN
   auth	Module name — human-readable	matches src/auth/ directory
   E-01,E-02	Entity IDs — links to doc entities	E-01 through E-NN
   Scene-1	Scenario/feature IDs — links to requirements	Scene-1 through Scene-N

   Critical constraint: 80-character label truncation

   graphify truncates rationale labels to 80 characters (extract.py:3544: label = text[:80], based on graphify v0.5.x). Your annotation MUST fit within 80 characters.

   Good (37 chars):

   # IMPORTANT: M-01 auth | E-01,E-02 | Scene-1

   Bad (>80 chars, will be truncated):

   # IMPORTANT: M-01 authentication-module | E-01-User,E-02-Session,E-03-Token | Scene-1-login-flow

   Annotation schedule

   When to annotate	What to do
   New module created	Add `# IMPORTANT: M-XX module_name
   Entity added/removed	Update the E-XX list in the annotation
   Module renamed	Update module name in annotation
   Function signature change	No action needed — AST captures automatically
   Documentation content change	No action needed — semantic extraction handles it

   ---

   4. Step 2: Post-Processing Alignment Script

   Three-tier safety model

   The alignment script runs 8 strategies organized into 3 tiers by risk level:

   Tier 0 — Deterministic (confidence: 1.0)
     S1: ID annotation match    → rationale "M-06" matches doc containing "M-06"
     S2: Exact class name       → doc label "AuthService" matches AST node AuthService
     S3: Path pattern mapping   → "docs/spec/01-auth.md" maps to src/auth/ module
   
   Tier 1 — Controlled fuzzy (confidence: 0.80-0.90)
     S4: Keyword → service      → within-module keyword matching
     S5: Scenario mapping        → requirement scenario IDs → module
     S6: Domain keyword          → curated keyword → module (locale-specific)
     S7: Issue cross-validation  → filename keyword + label confirmation
   
   Tier 2 — Free text (confidence: <0.80)
     S8: Token overlap           → Jaccard similarity, report-only
   

   Module isolation: the critical safety measure

   All fuzzy strategies (Tier 1-2) search WITHIN the identified module only.

   Without module isolation, a concept like "storage module" in a document about the auth system could match StorageService in the storage module — a cross-module false positive.

   With module isolation:

   1. First determine which module the document refers to (from path pattern or ID annotation)
   2. Only search within that module's 5-15 service classes
   3. Eliminates cross-module false positives entirely

   Lesson learned: Before adding module isolation, we observed false positives where storage documentation matched unrelated service classes. After isolation: zero false positives in 218 aligned edges.

   Cross-validation for Tier 1

   When a single fuzzy strategy produces a match, the confidence is 0.80 (CROSSVAL threshold). When two independent strategies agree on the same match:

   combined_confidence = min(strategy_a_confidence, strategy_b_confidence) + 0.10

   This promotes cross-validated matches to the AUTO threshold (≥0.90).

   Idempotency

   The script is safe to run multiple times:

   1. Each aligned edge carries _aligned: true tag
   2. Before each run, all _aligned edges are stripped
   3. This prevents duplicate edges on re-runs

   Configuration tables

   The following tables are project-specific examples. Adapt the mapping rules to your own project's directory structure and naming conventions.

   # Document path pattern → code module name
   DOC_MODULE_MAP = {
       "spec/01-": "auth",
       "spec/02-": "storage",
       "spec/03-": "api",
       # ...
   }
   
   # Module ID → module name (from your project's numbering)
   MODULE_ID_MAP = {
       "M-01": "auth",
       "M-02": "storage",
       "M-03": "api",
       # ...
   }
   
   # Module name → default service class ID (from AST extraction)
   MODULE_DEFAULT_SERVICE = {
       "auth": "auth_service_authservice",
       "storage": "storage_database_database",
       # ...
   }
   
   # Module → keyword → specific service class (module-isolated)
   MODULE_SERVICE_KEYWORDS = {
       "auth": {
           "token": "auth_token_manager_tokenmanager",
           "session": "auth_session_sessionhandler",
       },
       # ...
   }

   ---

   5. Step 3: Quality Audit

   TrustScore (0-100)

   Every aligned edge receives a composite trust score:

   Dimension	Weight	Measures
   Match Score	50%	Strategy confidence (1.0 → 100, 0.80 → 80)
   Topology Score	20%	Module consistency between source and target
   Context Score	15%	Path pattern alignment with module mapping
   Stability Score	15%	Both nodes exist in the graph

   Grade thresholds

   Grade	Score	Action
   A	≥85	Keep — high confidence
   B	≥70	Keep — acceptable confidence
   C	50-69	Review manually
   D	<50	Auto-deleted

   Write thresholds by tier

   Tier	Auto-write	Cross-validate	Report only
   Tier 0	≥0.90	—	—
   Tier 1	≥0.90	0.70-0.89	<0.70
   Tier 2	≥0.90	—	<0.90 (always report)

   Audit report output

   {
     "total_edges": 218,
     "total_before_audit": 218,
     "d_grade_removed": 0,
     "by_grade": {"A": 38, "B": 180, "C": 0, "D": 0},
     "by_method": {
       "s1_id_annotation": 35,
       "s2_exact_class": 12,
       "s3_techspec_path": 74,
       "s4_keyword_service": 45,
       "s5_prd_scene": 18,
       "s7_issue_crossval": 14
     },
     "average_trust_score": 82.4
   }

   Note: The 218 total aligned edges include 144 cross-domain (doc↔code) edges and 74 intra-domain (spec↔issue) alignment edges. The 144 cross-domain count is the primary metric.

   ---

   6. Results

   Before alignment

   Doc subgraph ──── 0 edges ──── Code subgraph
      ↓                              ↓
   PRD, Spec, Issues              Services, Models, Functions
   ~200 doc nodes                 ~1,600 code nodes
   Dense internal edges           Dense internal edges
   

   After alignment

   Doc subgraph ──── 144 cross-domain edges ──── Code subgraph
      ↓                                              ↓
   Spec→Code coverage: 90%                All modules linked to docs
   PRD→Code path: complete chain          Issue→Code: traceable
   

   Validation queries

   Query	Before	After
   "Which module implements auth spec?"	No results	AuthService linked to spec/01-auth
   "If I change storage, which docs are affected?"	No results	3 doc paths returned
   "Issue #3 affects which code?"	No results	Storage + API modules identified
   "Full requirement→spec→code chain"	0 chains	100+ complete chains

   Key metrics

   Metric	Value
   Cross-domain edges (native)	0
   Cross-domain edges (aligned)	144
   Total aligned edges	218
   Spec→Code coverage	90%
   S1 ID annotation accuracy	100% (35/35, zero false positives)
   TrustScore A/B grade ratio	100%
   D-grade auto-deleted	0

   ---

   7. Design Decisions

   Why post-processing instead of modifying graphify source?

   Approach	Pros	Cons
   Post-processing script	Non-invasive, upgradable independently, works with any graphify version	Extra step after build
   Modify graphify core	Seamless integration	Fork maintenance, version coupling
   Modify subagent prompts	No post-processing needed	Unreliable LLM compliance, prompt drift

   The post-processing approach is the safest short-term solution. If graphify adopts a Global Linking Pass (as discussed in #65), the script can gracefully retire.

   Why # IMPORTANT: prefix?

   graphify's AST rationale extractor only captures specific prefixes. # IMPORTANT: is semantically appropriate ("this annotation is important metadata") and guaranteed to be captured.

   Why is module isolation critical?

   Without module isolation, global keyword matching produces false positives:

   • "storage" in auth documentation matches StorageService in the storage module
   • "session" in storage documentation matches SessionHandler in the auth module

   Module isolation restricts search to 5-15 candidate nodes per module, making false positives nearly impossible.

   Why TrustScore weights at 50/20/15/15?

   Initial weights (35/25/20/20) underweighted the strategy confidence. A weak edge (confidence 0.5) could score 62.5 — passing the C-grade threshold — if topology/context/stability were all perfect. Rebalancing to 50% Match Score ensures that a low-confidence match is properly penalized regardless of other factors.

   ---

   8. Reference Implementation

   Project structure

   your-project/
   ├── src/
   │   ├── auth/
   │   │   └── __init__.py          ← # IMPORTANT: M-01 auth | E-01,E-02 | Scene-1
   │   ├── storage/
   │   │   └── __init__.py          ← # IMPORTANT: M-02 storage | E-03 | Scene-2
   │   └── api/
   │       └── __init__.py          ← # IMPORTANT: M-03 api | E-04,E-05 | Scene-3
   ├── docs/
   │   ├── spec/
   │   │   ├── 01-auth.md           ← references M-01, E-01, Scene-1
   │   │   ├── 02-storage.md        ← references M-02, E-03, Scene-2
   │   │   └── 03-api.md            ← references M-03, E-04, Scene-3
   │   └── requirements/
   │       └── PRD.md               ← references Scene-1, Scene-2, Scene-3
   ├── scripts/
   │   └── graphify_align_edges.py  ← the alignment script
   ├── graphify-out/
   │   └── graph.json               ← graphify output (input + output for alignment)
   └── .graphify-align.yaml         ← project-specific alignment config (optional)
   

   Workflow

   # Step 1: Build the graph with graphify
   graphify .    # or via Claude Code skill: /graphify
   
   # Step 2: Run alignment
   python scripts/graphify_align_edges.py
   
   # Step 3: Verify
   graphify query "auth spec implements which module?"
   # Expected: AuthService linked to spec/01-auth

   Integration with graphify update cycle

   # After any code or doc change:
   graphify . --update                    # incremental graph update
   python scripts/graphify_align_edges.py # re-align cross-domain edges
   
   # The script is idempotent: safe to run after every update

   ---

   Appendix A: Upstream Issues

   Issue	Core Problem	This solution addresses
   #65	Cross-chunk edges lost	Module isolation + structured IDs bypass chunk isolation
   #198	AST/semantic subgraphs disconnected	Post-processing bridge creates cross-domain edges
   #390	Node ID mismatch	Structured IDs + deterministic matching
   #1033	Path.stem vs relative path	Path pattern mapping handles both conventions
   #653	--update drops semantic nodes	Idempotent re-alignment after every update
   #1145	Ghost-duplicate merge limitation	Explicit edge creation bypasses merge entirely

   Appendix B: Glossary

   Term	Definition
   Cross-domain edge	An edge connecting a documentation node to a code node
   Rationale node	A node created by graphify from # IMPORTANT:, # NOTE: etc. comments
   Module isolation	Restricting fuzzy search to nodes within an identified code module
   TrustScore	Composite quality score (0-100) for aligned edges
   Tier 0/1/2	Safety tiers: deterministic, controlled fuzzy, free text
   Idempotent	Safe to run multiple times without side effects

   ---

   Validated on a real-world project with 1,800+ nodes, 3,000+ edges, 200+ communities. Results: 0→144 cross-domain edges, 0→90% spec coverage, 100% A/B grade quality.