Skip to content

Wiki and Obsidian exporters rewrite every page on every run, even when nothing changed (31,417 files / 137 MB per export, ~7 GB/day here) #3060

Closed

Description

@Azeem1985

What happens

to_wiki() and to_obsidian() regenerate the full page set on every call, and they write
every page unconditionally:

  • graphify/wiki.py deletes every *.md in wiki/ up front (for old_article in out.glob("*.md"): old_article.unlink()),
    which guarantees that all of them are then written from scratch even when the content is identical;
  • graphify/export.py::_owned_write calls target.write_text(content, ...) without ever comparing
    against what is already on disk.

Nothing in either path asks "did this page actually change?".

Why it hurts

An export is triggered by any graph.json change, so on an always-on setup it runs constantly.
Measured on one vault (26.7k nodes, 2.3k communities), watched by graphify watch --semantic with
a systemd path unit re-running graphify export wiki + graphify export obsidian on every
graph.json change:

pages regenerated per export 31,417 (2,354 wiki articles + 29,063 Obsidian notes)
bytes rewritten per export 137 MB
exports in a measured 24 h 51
disk written per day just for the views ~7 GB

Between two consecutive exports the graph moved by ~20 nodes out of 26,500 — and all 31,417 pages
were rewritten anyway. The disk cost is the smaller half of it: every rewrite is also a
modification event, so Obsidian re-indexes the whole vault, sync clients re-upload it, and any
inotify-driven pipeline downstream re-fires, dozens of times a day, for content that did not change.

What fixes it

Compare before writing. Concretely, in a fork of v8:

  1. paths.write_text_if_changed(path, text) -> bool — writes only on a difference and reports
    whether it wrote. Compare decoded text, not bytes: on Windows write_text/read_text
    translate newlines, so a byte compare reports "changed" for every file.
  2. _owned_write skips identical notes but still records them in _written — that list is
    both the ownership manifest and the prune-exclusion set (export obsidian: never prunes notes for removed nodes, so re-export merges old and new graphs #1896), so dropping an unchanged note
    from it would disown the note and delete it as stale on the same run. to_canvas() and the
    manifest go through the same helper.
  3. In to_wiki, replace the up-front unlink with a sweep at the end that removes only what the
    run did not produce. Same orphan guarantee for relabelled communities, and it also closes a
    window where a reader sees an empty wiki/ mid-export.

Measured result of that change, on the same vault

  • Export run twice with no graph change: git-independent mtime comparison across all 31,417
    files → 0 files rewritten on the second run. Wall clock 13.2 s → 4.9 s.
  • A real production export, with a real graph change (26,713 → 26,730 nodes, communities
    re-clustered): 2,075 of 2,354 articles and 25,552 of 29,075 notes left untouched — 3,802 files
    written instead of 31,429, an 88% cut on an export that genuinely had work to do.

Happy to open a PR against v8 with the implementation and the mtime-based tests
(12 of them: the primitive, both exporters' no-op re-run, the page that really did change, the
manifest/prune interaction, orphan removal after a relabel, and the no-empty-window guarantee) —
they are already written and green in our fork.

Activity

  1. Azeem1985 commented on Aug 27, 2026

    @Azeem1985
    ContributorAuthor

    Correcting one line of my own report.

    The "Why it hurts" section says the vault was "watched by graphify watch --semantic". That flag does not exist upstream — it is my fork's. Checked against v0.9.50 (43d54acbfa9e731f7a592bb582c1f4b9d48ed73e): --semantic appears zero times in graphify/watch.py, graphify/cli.py and graphify/__main__.py, and the watch sub-command parses no options at all. So a reader who copies that command gets a plain watch, and cannot reproduce my setup from it.

    What the measurement actually needs, and it is not that flag. The numbers come from any arrangement where graphify export wiki and graphify export obsidian re-run on every graph.json change. Upstream that is a path-triggered or post-commit re-export; the thinking pass is not part of it. The defect reported here is in the exporters and is independent of what triggers them:

    • wiki.py deletes every article before writing any
    • export.py::_owned_write writes unconditionally, with no comparison

    Both hold at v0.9.50 regardless of how the export is invoked.

    The measured figures — 31,417 pages and ~137 MB per export, ~51 exports observed in 24 hours — stand. Only the command line naming the trigger was wrong, and it was wrong by describing my own build as yours.

    🤖 Generated with Claude Code

  2. safishamsi commented on Sep 23, 2026

    @safishamsi
    Member

    Fixed in v0.9.67 via #3760 — export no longer rewrites unchanged wiki/Obsidian pages. Thanks @abhay-codes07.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions