JVM insert_many: pause the cyclic garbage collector for the call when the input is a list or tuple #235

On the arcadedb-jvm build, Database.insert_many spends a large part of its wall time in CPython's cyclic garbage collector, running inside the call and re-traversing the caller's live row list again and again. Pausing the collector for the duration of the call removes that time. This issue asks for that change on the JVM build only.

Where the time goes

A profile of a document ingest on the JVM wheel put 14.5% of the calling thread's CPU samples in CPython GC frames (list_traverse, visit_reachable, gc_collect_main, deduce_unreachable). A split by gc.callbacks showed the collections happen inside the insert_many call, triggered by the allocations of the call itself (JPype crossings, chunk lists) and paid on the caller's live row lists: about 1,270 young and 32 to 46 full collections per 1,000,000 rows. On the native build there are zero collections inside the call (its crossing allocates no GC-tracked objects), so this does not apply there and the native build is not changed.

Measurement

A/B on the laptop with ab.py --noise, ABBA order, round 0 dropped, 500,000 dict rows (id, grp, val, cat, txt, a 32-float vector, UNIQUE index on id) in slices of 100,000, one fresh process per run, one wheel for every arm: arcadedb-jvm 26.11.1.dev0 from CI run 41646452646 on main 413f54ffe1 (sha256 1df4d16f396f42edb87bc38349ccd0758e4e5f57e22bef51cfecd2f5b910b255). Arm C is that wheel's own core.py with the pause patched in; arm B is the same pause done by the caller; Z is the default again as the A/A control. Ratios are arm over default, below 1 is faster or smaller; intervals are the 95% CI of the median per-round ratio.

The A/A control interval contains 1.00 on both wall figures, so the comparison is clean. The rule was written before the first timed run: the candidate passes when the upper end of the insert-wall interval is at most 0.95, the load-wall upper end is at most 1.00, the peak RSS upper end is at most 1.10 and the A/A control is clean. C passes all of them, and so does B.

The figures above are generated by script from ab/score.out of the study python-gc-ingest-ab-laptop-20261025.

Risk and mitigation

While the collector is paused, cyclic garbage created during the call is not collected until it returns. For a list of plain dict rows the call creates nothing cyclic, and the peak RSS did not rise. For a lazily read generator that builds objects with reference cycles, the garbage of the whole load would accumulate, so memory could grow without bound on a long generator. Mitigation: the pause applies only when rows is a list or a tuple, where no user code runs between the library's chunk reads. A generator, an iterator, a DataFrame or any other lazy input keeps the collector running. The previous state is restored in finally, so a caller that had already disabled the collector keeps it disabled, and an exception cannot leave it off.

gc.disable() is process wide: other Python threads do not collect automatically while a call with a list runs. With overlapping calls from several threads, a call that starts while another has paused the collector sees it disabled and restores "disabled", so the pause is not guaranteed for the full length of overlapping calls, and it is never left stuck off.

Change

insert_many records gc.isenabled(), calls gc.disable() only for a list or tuple, runs the unchanged body, and restores the state in finally. The docstring and the bulk-loads docs page say so, with no speed figure. Tests cover list, tuple, an exception, a caller that had disabled the collector, and a generator. The decision is recorded in the HumemAI benchmark notes (DECISIONS #238 addendum on TA-2 and #239): a measured default of a shipped package is adopted when the study's pre-written rule passes.