Skip to content

Native iter_json_batches drains the whole result before it batches, so batch_size does not bound memory #239

Open

Description

@tae898

On the native build, iter_json_batches(batch_size=...) does not stream: batch_size only cuts a list that already holds the whole result, so it bounds nothing.

NativeResultSet.iter_json_batches (src/arcadedb/_native/database.py:298 on main 1b315c1) calls to_json_list(), which calls _drain_dicts() (:274): every reply of the cursor is read into one list of dicts, then json_native builds a second list of converted rows, and only then are slices of that list yielded. At the peak the process holds the cursor's rows twice, whatever the batch size. The JVM build does stream here (results.py iter_json_batches reads one Java-serialized batch at a time and its docstring says "bounds memory to one batch"), so the two builds disagree on the memory contract of the same method. A caller that reads a large result in batches to keep memory flat (an export, a copy into another store, the ETL loop iter_json_batches exists for) gets the whole result in memory on native.

to_columns (:310), to_arrow (:322) and to_polars (:333) also drain the whole result first. They are different: the type of a column is decided over all its values (an int column that gets a float in the last row becomes float64), so a batch-wise build needs a second pass or a merge rule, and the result is one table that holds every row anyway. They are left as they are in the fix for this issue, and their peak is measured next to iter_json_batches so a follow-up (columns built per batch without the per-row dicts) can be judged on a figure.

Proposed change (draft PR follows): iter_json_batches reads the cursor one batch at a time, holding at most one reply (_NEXT_BATCH, 2048 rows) plus one output batch. The rows, their order, their types and the batch boundaries are the ones to_json_list gives, cut at batch_size; the first batch is returned after reading only as many rows as it needs. One behaviour is different on purpose and follows the other iterators of the result set: the cursor is read when the next batch is asked for, so a result closed between two batches raises the "closed before all its rows were read" error, where today the batches already built keep coming. An error met while reading surfaces after the batches before it, as it does when iterating.

Not measured yet: the peak RSS of a 1M-row read through each path, native against JVM (a probe is prepared on the maintainer's side; it drops this change if the native peak is not clearly above the JVM wheel's). The PR stays a draft until the figure is in.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions