Skip to content

Four silent failures with success exit codes: cluster-only ignores --backend, label prompt collides with the Community {cid} sentinel, tree --root <abs> flattens the hierarchy, built_at_commit stamped from cwd #2534

Closed

Description

@elecnix

Four independent bugs, filed together because they share one shape: the command exits 0, prints a success line, and produces output that looks right but isn't. There is no error for a user — or an agent — to react to.

All line references are against v8 as of today, so these are not stale-version reports.


1. cluster-only --backend X accepts labeling flags and never calls the LLM

cli.py:1621 sets force_relabel = cmd == "label", and cli.py:1776 gates the whole labeling block on it:

force_relabel = cmd == "label"
...
if labels_path.exists() and not force_relabel:
    # reuse saved labels / hub-rename; --backend, --model, --batch-size unused

So once .graphify_labels.json exists, cluster-only --backend openai --model … --batch-size N silently discards every one of those flags. It prints Done - N communities. GRAPH_REPORT.md, graph.json and graph.html updated., and GRAPH_REPORT.md records Token cost: 0 input · 0 output.

There is a warning at cli.py:1830, but it only fires if changed: — i.e. only when the community set drifted. When the count matches, the no-op is completely silent.

The flags being accepted is what sells it: a rejected flag would have been obvious immediately. It took two labeling runs and a look at the token counter to notice nothing had happened.

Suggested fix: if --backend / --model / --batch-size are passed explicitly to cluster-only while the reuse branch is taken, either honour them or warn naming the ignored flags. Silently accepting a flag that does nothing is the worst of the three options.

2. The label prompt collides with the placeholder sentinel, so some models' answers get discarded

llm.py:2859 formats each prompt line using the same string that llm.py:2834 uses as the "unnamed" sentinel:

lines.append(f"Community {cid}: {', '.join(names)}")            # llm.py:2859
return {int(cid): f"Community {cid}" for cid in communities}    # llm.py:2834

Models that pattern-match the input key echo it straight back as the value — {"5": "Community 5"} — and cli.py:1874 then drops exactly that string:

labels.update({cid: v for cid, v in generated_labels.items()
               if v and v != f"Community {cid}"})

Net effect: the LLM answered, the answer was thrown away, and the deterministic hub name was kept. No warning, no error, no token-cost anomaly. It reads as "the model chose not to name this cluster".

Same graph, same prompt, same batch size — only the model varies:

model share of communities named
google/gemini-3.6-flash ~27%
google/gemini-3.5-flash echoed the sentinel on half of an isolated sample
openai/gpt-5.4-mini all of the same sample
google/gemma-4-31b-it ~100%

The same small set of communities, isolated (labels redacted, <name> = a correct plain-language label):

gemini-3.6-flash  -> {0: '<name>', 1: 'Community 1', 2: 'Community 2', 5: 'Community 5'}
gpt-5.4-mini      -> {0: '<name>', 1: '<name>',     2: '<name>',      5: '<name>'}

Every cluster in that sample was plainly nameable — two other models named all of them. Gemini is echoing the prompt's key prefix, not declining to answer.

Worth noting how this compounds with the hub-label fallback: because the fallback is plausible (a real symbol name), nothing about the output looks broken. The fallback is frequently a bare filename, and a large cluster can end up named after a trivial shared helper.

Suggested fix: stop using Community {cid} as both the prompt key and the sentinel. An opaque prompt key (C0:, [0], or a bare index) removes the collision for every model at zero cost. Optionally also warn when a batch returns >50% sentinel values, which would surface this on the first run.

3. tree --root <absolute path> silently flattens the hierarchy

graph.json stores source_file repo-relative (<dir>/<subdir>/<file>). tree_html.py:115-122 does:

for src_file, syms in sorted(by_file.items()):
    src_path = Path(src_file)
    try:
        rel = src_path.relative_to(root_path)
        parent_path = (root_path / rel).parent
    except ValueError:
        parent_path = root_path      # <-- every file lands here

--root is documented as "filesystem root for the hierarchy", so an absolute path is the obvious thing to pass. When you do, relative_to raises for every file and the except attaches all of them directly to the root.

The result: max depth 2, and the root's children become the flat set of every file basename in the graph rather than the top-level directories. Siblings collide — distinct files sharing a basename appear as indistinguishable top-level entries. It still writes the HTML and reports success, with nothing to indicate the hierarchy was discarded.

Suggested fix: treat a --root that resolves zero source_file entries as an error, or normalise it against the graph's own path convention before use. Even a 0 of N files resolved under --root warning would be enough.

4. cluster-only re-stamps built_at_commit from the shell's cwd, not from the analysed path

GRAPH_REPORT.md carries a provenance line the docs invite you to trust:

## Graph Freshness
- Built from commit: `<sha>`
- Run `git rev-parse HEAD` and compare to check if the graph is stale.

extract --out <dir> stamps that correctly. But a subsequent cluster-only <dir> run from a different working directory overwrites it with the cwd's HEAD, and the path being analysed is never consulted.

Reproduced by extracting two different revisions of the same repo into two separate output dirs, then running cluster-only on both from one shell:

graph A (extracted from revision X)  -> built_at_commit: <cwd HEAD>
graph B (extracted from revision Y)  -> built_at_commit: <cwd HEAD>   # identical

Both graphs claimed the same commit despite demonstrably different contents (different file counts, different node/edge totals). The failure is quiet and the consequence is specific: the one field designed to tell you whether a graph is stale is the field that goes wrong, and the suggested git rev-parse HEAD check actively confirms the false answer when you happen to run it from the stamping cwd.

This also makes graphs non-comparable across revisions, which is otherwise a natural use — extract two revisions, diff the graphs, measure what a refactor changed.

Suggested fix: resolve built_at_commit from the analysed path (the --out/target dir's source root), not from os.getcwd(); or leave the value written by extract untouched, since cluster-only re-clusters an existing graph rather than re-reading the source tree.


Happy to split these into four issues if that suits triage better. I grouped them because the shared "success exit, wrong output" property is arguably a more useful signal than any one of them alone.

Activity

  1. changed the title [-]Three silent failures with success exit codes: cluster-only ignores --backend, label prompt collides with the `Community {cid}` sentinel, `tree --root <abs>` flattens the hierarchy[/-] [+]Four silent failures with success exit codes: cluster-only ignores --backend, label prompt collides with the `Community {cid}` sentinel, `tree --root <abs>` flattens the hierarchy, built_at_commit stamped from cwd[/+] on Aug 7, 2026
  2. safishamsi commented on Aug 7, 2026

    @safishamsi
    Member

    Fixed in v0.9.36 (graphifyy==0.9.36 on PyPI). All four cases now surface: cluster-only warns when --backend/--model/--batch-size are ignored on the label-reuse path; the community-label prompt key no longer collides with the Community <id> discard sentinel (a model echoing the key back is no longer silently dropped); tree --root exits non-zero with a clear message when the root matches no source file instead of silently flattening the tree; and cluster-only stamps built_at_commit from the analysed graph rather than the shell working directory. Thanks for the precise list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions