Skip to content

Invalid-confidence warning is permanent: _fold_edge_aliases normalizes numeric legacy confidences but not string ones (high/medium/confirmed) #4495

Open

Description

@aglit87

Summary

Any graph whose semantic (LLM) pass emitted a string confidence outside the enum re-warns forever on every command:

[graphify] Extraction warning (259 issues): 259x other schema issue
(e.g. Edge 7335 has invalid confidence 'confirmed' - must be one of ['AMBIGUOUS', 'EXTRACTED', 'INFERRED'])

_fold_edge_aliases() in graphify/build.py normalizes only numeric legacy confidences; free-form strings pass through untouched and are never repaired by any CLI command.

Environment

  • graphify / graphifyy 0.9.91, Windows, Python 3.15
  • Corpus: mixed code + docs (OpenSpec markdown), 2777 nodes / 6648 edges, 153 communities
  • The graph was built by an earlier full build (semantic pass = host agent); graphify update . only re-extracts code, so the offending edges persist.

Impact

  • 259 edges carry a non-enum confidence: high = 154, confirmed = 93, medium = 6, inferred = 3, medium-high = 2, medium-low = 1.
  • All 259 have _origin: semantic, all have confidence_score, and all come from documents (253 .md, 6 .yaml).
  • graph["hyperedges"] carries the same class of bad values: floats (e.g. 0.75, 0.85, 0.9) and strings (high, medium, inferred, medium-high).
  • The warning is re-emitted on every graphify update, query, explain, label, … and is not fixable from the CLI.

Root cause

graphify/build.py:

def _fold_edge_aliases(edge: dict) -> None:
    ...
    _conf = edge.get("confidence")
    if isinstance(_conf, (int, float)) and not isinstance(_conf, bool):
        if edge.get("confidence_score") is None:
            edge["confidence_score"] = float(_conf)
        edge["confidence"] = "INFERRED"
    if not edge.get("confidence") and edge.get("confidence_score") is not None:
        edge["confidence"] = "INFERRED"

The docstring already reasons: "numeric confidences only ever came from the LLM semantic pass, and LLM-derived edges are INFERRED by definition" and "Without this fold, every reload of a pre-enum graph re-warns once per legacy edge, forever." That reasoning applies verbatim to string legacy values (high / medium / medium-high / medium-low / confirmed / lowercase inferred), but they are not folded — so they re-warn forever, exactly as the docstring warns for numerics.

The extraction prompt (references/extraction-spec.md) asks for "confidence":"EXTRACTED|INFERRED|AMBIGUOUS" plus a separate confidence_score float, so this string vocabulary is model drift, not a documented format.

Suggested fix

Extend the fold to non-enum strings, case-insensitively, mapping to INFERRED (LLM-derived by definition), moving a parseable value into confidence_score:

_ALLOWED = {"AMBIGUOUS", "EXTRACTED", "INFERRED"}
_conf = edge.get("confidence")
if isinstance(_conf, (int, float)) and not isinstance(_conf, bool):
    if edge.get("confidence_score") is None:
        edge["confidence_score"] = float(_conf)
    edge["confidence"] = "INFERRED"
elif isinstance(_conf, str) and _conf.upper() not in _ALLOWED:
    try:
        if edge.get("confidence_score") is None:
            edge["confidence_score"] = float(_conf)
    except ValueError:
        pass
    edge["confidence"] = "INFERRED"
if not edge.get("confidence") and edge.get("confidence_score") is not None:
    edge["confidence"] = "INFERRED"

The same fold should be applied to graph["hyperedges"], which currently is not normalized at all.

Steps to reproduce

  1. Build a graph whose semantic pass emits a non-enum string confidence (e.g. {"confidence": "high", "confidence_score": 0.85}).
  2. Run any graphify command (graphify query "...", graphify update .).
  3. Observe the "invalid confidence" schema warning; re-running never clears it and the edge values stay unchanged.

Workaround

Post-process graph.json: map every non-enum confidence to INFERRED (and mirror the same mapping into graph.hyperedges).

Activity

  1. github-actions commented on Oct 16, 2026

    @github-actions

    Thanks for opening this issue, @aglit87. A maintainer will take a look soon.

    If you would like to discuss it in real time, come say hi on our Discord server. For longer-form questions and ideas there is also GitHub Discussions.

    To help us triage, please make sure the report includes what you expected, what actually happened, and the steps (and a small sample) to reproduce it.

  2. omsenjalia commented on Oct 18, 2026

    @omsenjalia

    This looks fixed on the v8 branch by 5fec9d32 (Add regression tests for the drifted confidence fold). Could a maintainer confirm and close?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions