You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native iter_json_batches drains the whole result before it batches, so batch_size does not bound memory #239
On the native build, iter_json_batches(batch_size=...) does not stream: batch_size only cuts a list that already holds the whole result, so it bounds nothing.
NativeResultSet.iter_json_batches (src/arcadedb/_native/database.py:298 on main 1b315c1) calls to_json_list(), which calls _drain_dicts() (:274): every reply of the cursor is read into one list of dicts, then json_native builds a second list of converted rows, and only then are slices of that list yielded. At the peak the process holds the cursor's rows twice, whatever the batch size. The JVM build does stream here (results.pyiter_json_batches reads one Java-serialized batch at a time and its docstring says "bounds memory to one batch"), so the two builds disagree on the memory contract of the same method. A caller that reads a large result in batches to keep memory flat (an export, a copy into another store, the ETL loop iter_json_batches exists for) gets the whole result in memory on native.
to_columns (:310), to_arrow (:322) and to_polars (:333) also drain the whole result first. They are different: the type of a column is decided over all its values (an int column that gets a float in the last row becomes float64), so a batch-wise build needs a second pass or a merge rule, and the result is one table that holds every row anyway. They are left as they are in the fix for this issue, and their peak is measured next to iter_json_batches so a follow-up (columns built per batch without the per-row dicts) can be judged on a figure.
Proposed change (draft PR follows): iter_json_batches reads the cursor one batch at a time, holding at most one reply (_NEXT_BATCH, 2048 rows) plus one output batch. The rows, their order, their types and the batch boundaries are the ones to_json_list gives, cut at batch_size; the first batch is returned after reading only as many rows as it needs. One behaviour is different on purpose and follows the other iterators of the result set: the cursor is read when the next batch is asked for, so a result closed between two batches raises the "closed before all its rows were read" error, where today the batches already built keep coming. An error met while reading surfaces after the batches before it, as it does when iterating.
Not measured yet: the peak RSS of a 1M-row read through each path, native against JVM (a probe is prepared on the maintainer's side; it drops this change if the native peak is not clearly above the JVM wheel's). The PR stays a draft until the figure is in.
Activity