Skip to content

to_pdf peaks at 19.2x the SVG text: 267 MB for an 11 MB PDF at 200k points #312

Description

@Alek99

export.to_image(fig, "pdf") converts by parsing the whole SVG document into an
ElementTree and walking it (python/xy/_pdf.py:1252):

def svg_to_pdf(svg: str) -> bytes:
    try:
        root = ET.fromstring(svg)
    ...
    return _Converter().run(root)

Every element becomes a Python object with an attribute dict, so the in-memory
tree is roughly an order of magnitude larger than the SVG text it came from —
and the SVG text for a scatter chart is itself linear in the point count.

Measured

Peak traced allocation of svg_to_pdf alone (tracemalloc.reset_peak() around
the call, native core held constant, warm __pycache__):

points SVG text PDF out peak peak / SVG
5,000 0.35 MB 0.28 MB 6.7 MB 19.2x
20,000 1.39 MB 1.10 MB 26.7 MB 19.2x
80,000 5.55 MB 4.40 MB 106.7 MB 19.2x
200,000 13.88 MB 11.00 MB 267.0 MB 19.2x

The ratio is a flat 19.2x across two orders of magnitude, so this is structural
rather than a fixed overhead: cost is 19.2 x len(svg), and len(svg) grows
with the marks in the document.

For comparison, the same figure's SVG export peaks at ~5.8 MB and, after
#309, the JPEG and PNG encoders
are bounded well below their own frame sizes. PDF is now the least
memory-efficient export path by a wide margin — a 200k-point vector PDF needs a
quarter of a gigabyte of headroom to produce an 11 MB file.

Why it was left alone

Found during the second memory audit (follow-up to #309). It was deliberately
not fixed there: every other hotspot that audit touched had a bounded, local
fix that kept output byte-identical, whereas this one needs the converter
restructured, which deserves its own review.

Suggested fix

Stream the conversion instead of materializing the tree: drive _Converter
from xml.etree.ElementTree.XMLPullParser (or expat directly) on
start/end events, emitting content-stream operators as elements close and
clearing each element (elem.clear()) once its subtree is written. xy's SVG
subset is closed and shallow — the mark elements that dominate the document are
siblings, not deeply nested — so a pull parser should hold O(depth) rather than
O(document).

Two things make this tractable to verify:

  • The output is checkable exactly: byte-identical PDFs against the current
    converter over the existing tests/test_pdf.py corpus plus a few large
    scatter/line/heatmap figures.
  • ValueError("unsupported SVG feature: ...") must still be raised for
    anything outside the emitted subset, including the unparseable-XML case,
    which a pull parser reports at a different point in the stream.

A cheaper partial step, if the full rewrite is not wanted: parse and convert
<g> subtree by subtree, clearing each as it is consumed. That bounds the live
tree to one mark group rather than the whole document without changing the
converter's structure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions