export.to_image(fig, "pdf") converts by parsing the whole SVG document into an
ElementTree and walking it (python/xy/_pdf.py:1252):
def svg_to_pdf(svg: str) -> bytes:
try:
root = ET.fromstring(svg)
...
return _Converter().run(root)
Every element becomes a Python object with an attribute dict, so the in-memory
tree is roughly an order of magnitude larger than the SVG text it came from —
and the SVG text for a scatter chart is itself linear in the point count.
Measured
Peak traced allocation of svg_to_pdf alone (tracemalloc.reset_peak() around
the call, native core held constant, warm __pycache__):
| points |
SVG text |
PDF out |
peak |
peak / SVG |
| 5,000 |
0.35 MB |
0.28 MB |
6.7 MB |
19.2x |
| 20,000 |
1.39 MB |
1.10 MB |
26.7 MB |
19.2x |
| 80,000 |
5.55 MB |
4.40 MB |
106.7 MB |
19.2x |
| 200,000 |
13.88 MB |
11.00 MB |
267.0 MB |
19.2x |
The ratio is a flat 19.2x across two orders of magnitude, so this is structural
rather than a fixed overhead: cost is 19.2 x len(svg), and len(svg) grows
with the marks in the document.
For comparison, the same figure's SVG export peaks at ~5.8 MB and, after
#309, the JPEG and PNG encoders
are bounded well below their own frame sizes. PDF is now the least
memory-efficient export path by a wide margin — a 200k-point vector PDF needs a
quarter of a gigabyte of headroom to produce an 11 MB file.
Why it was left alone
Found during the second memory audit (follow-up to #309). It was deliberately
not fixed there: every other hotspot that audit touched had a bounded, local
fix that kept output byte-identical, whereas this one needs the converter
restructured, which deserves its own review.
Suggested fix
Stream the conversion instead of materializing the tree: drive _Converter
from xml.etree.ElementTree.XMLPullParser (or expat directly) on
start/end events, emitting content-stream operators as elements close and
clearing each element (elem.clear()) once its subtree is written. xy's SVG
subset is closed and shallow — the mark elements that dominate the document are
siblings, not deeply nested — so a pull parser should hold O(depth) rather than
O(document).
Two things make this tractable to verify:
- The output is checkable exactly: byte-identical PDFs against the current
converter over the existing tests/test_pdf.py corpus plus a few large
scatter/line/heatmap figures.
ValueError("unsupported SVG feature: ...") must still be raised for
anything outside the emitted subset, including the unparseable-XML case,
which a pull parser reports at a different point in the stream.
A cheaper partial step, if the full rewrite is not wanted: parse and convert
<g> subtree by subtree, clearing each as it is consumed. That bounds the live
tree to one mark group rather than the whole document without changing the
converter's structure.
export.to_image(fig, "pdf")converts by parsing the whole SVG document into anElementTree and walking it (
python/xy/_pdf.py:1252):Every element becomes a Python object with an attribute dict, so the in-memory
tree is roughly an order of magnitude larger than the SVG text it came from —
and the SVG text for a scatter chart is itself linear in the point count.
Measured
Peak traced allocation of
svg_to_pdfalone (tracemalloc.reset_peak()aroundthe call, native core held constant, warm
__pycache__):The ratio is a flat 19.2x across two orders of magnitude, so this is structural
rather than a fixed overhead: cost is
19.2 x len(svg), andlen(svg)growswith the marks in the document.
For comparison, the same figure's SVG export peaks at ~5.8 MB and, after
#309, the JPEG and PNG encoders
are bounded well below their own frame sizes. PDF is now the least
memory-efficient export path by a wide margin — a 200k-point vector PDF needs a
quarter of a gigabyte of headroom to produce an 11 MB file.
Why it was left alone
Found during the second memory audit (follow-up to #309). It was deliberately
not fixed there: every other hotspot that audit touched had a bounded, local
fix that kept output byte-identical, whereas this one needs the converter
restructured, which deserves its own review.
Suggested fix
Stream the conversion instead of materializing the tree: drive
_Converterfrom
xml.etree.ElementTree.XMLPullParser(orexpatdirectly) onstart/endevents, emitting content-stream operators as elements close andclearing each element (
elem.clear()) once its subtree is written. xy's SVGsubset is closed and shallow — the mark elements that dominate the document are
siblings, not deeply nested — so a pull parser should hold O(depth) rather than
O(document).
Two things make this tractable to verify:
converter over the existing
tests/test_pdf.pycorpus plus a few largescatter/line/heatmap figures.
ValueError("unsupported SVG feature: ...")must still be raised foranything outside the emitted subset, including the unparseable-XML case,
which a pull parser reports at a different point in the stream.
A cheaper partial step, if the full rewrite is not wanted: parse and convert
<g>subtree by subtree, clearing each as it is consumed. That bounds the livetree to one mark group rather than the whole document without changing the
converter's structure.