Skip to content

Native append_samples: send numeric columns as binary buffers, not .tolist() and one JSON text per chunk #228

Open

Description

@tae898

On the native wheel, NativeAsyncExecutor.append_samples turns every column into a Python list and sends the whole chunk as one JSON text. plain() runs .tolist() on each numpy column, stamps = [int(t) for t in plain(timestamps)] walks the timestamps a second time, and _op("append_samples", type=, ts=, cols=) encodes everything with _enc (the _prep walk plus json.dumps) for arcadedb_n_async. NativeExtras.async parses the text back into Object[] columns. The primitive=True flag is only a length check on the native build.

The JVM wheel does the opposite for the same call: _append_samples_primitive sends one JNI array per numeric column through TimeSeriesBatcher, and only the string tag column goes element by element. The library already has binary routes for other calls (arcadedb_n_exec_bin takes a float32 side buffer and arcadedb_n_insert_colp takes an address table), arcadedb_n_async has none.

Measurement (untimed, laptop, CI wheels of main 413f54f, native and JVM 26.11.1.dev0, one run per cell, so orientation and not a ratio to quote). The shape is the l4 time-series lane's: 1,000,000 points, 100 hosts, 100,000-point chunks, numpy int64 and float64 columns, a Python list of host strings, primitive=True, one wait_completion() at the end. Both runs read the stored samples back by count, sums and windows and match the source.

figure (lane shape, one run, untimed) native wheel JVM wheel
append calls, us per point 0.755 0.314
whole load wall, s (includes the harness's own column building) 0.957 0.736
Python glue (tolist() of every column and the int() loop), us per point 0.100 n/a
JSON encode (extras._enc), us per point 0.347 n/a
crossing into arcadedb_n_async (Java parse and queue), us per point 0.318 n/a
glue plus encode, share of the load wall 46.7% n/a

The text is 43.6 bytes per point, against 38.9 bytes for the same columns as raw int64 and float64 plus the host text, so the size is not the cost: formatting, parsing and the two list conversions are. Glue plus encode is 0.447 of the 0.755 us per point of the append calls, and 46.7% of the load wall (median chunk of the run: append 75.6 ms, encode 34.7 ms). The crossing figure also holds the Java parse and any wait for the writers, so a binary route may move it too; that is for the A/B to say.

Proposal: add arcadedb_n_async_bin(handle, op, json, table, n_table) next to arcadedb_n_async, following the address-table convention of arcadedb_n_insert_colp. For append_samples, a one-dimensional numpy int or float column (and an integer or datetime64[ns] timestamp array) crosses by address as int64 or float64 and the JSON carries only a kind marker per column; the host tag column and anything else stays as text. Java builds the same long[] and boxed Object[] columns and calls the same appendSamples, so the stored samples are bit-identical to today (a float32 array widens to float64, bool is 0 and 1, NaN is the canonical NaN the text path gives). A library without the entry, or ARCADEDB_NATIVE_TS_BIN=0, keeps the JSON path. Lists, float arrays as timestamps, 2-D arrays, uint64 past the int64 range and a column whose length differs from the timestamps keep the JSON path. primitive=True keeps its meaning (the same length check). The JVM wheel is not touched.

Not claimed: the table is a probe, not a timed A/B. A wheel-versus-wheel A/B with the usual repetitions comes after the PR's CI wheel exists. The Java side still boxes each value once; the engine's TimeSeriesBatch route is a possible follow-up, not part of this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions