You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native GraphBatch.create_vertices: send float32 vectors in a binary side buffer, not as F-tagged JSON text #222
On the native wheel, GraphBatch.create_vertices sends a float32 vector as number text. NativeGraphBatch._op encodes the op arguments with _enc and calls arcadedb_n_gb(handle, op, json), and codec._default writes a float32 ndarray as {TAG: ["F", value.tolist()]} (TAG is the control character U+0001, the key of every tagged value): Python formats every float and the library parses it back. new_edges and new_edge props carry vectors the same way.
The library already has the binary route for query parameters: arcadedb_n_exec_bin takes a float32 side buffer, codec._default writes ["B", offset, count] into the JSON, and Json.decodeTag reads the floats from the buffer into a float[]. arcadedb_n_gb has no side-buffer variant.
Measurement (untimed, laptop, CI wheels of main 413f54f, native 26.11.1.dev0, one run per cell, so orientation and not a ratio to quote). Shapes are the ones the vector lanes load with: SIFT 128-d in batches of 10,000, Deep 96-d in batches of 10,000 and a 64-d hybrid shape in batches of 5,000. The reference loads the same vertices from the same matrix through insert_columns, which crosses by address.
shape
dim
batch
JSON text over float32 bytes
create_vertices ndarray, load us per vertex
insert_columns 2-D reference, load us per vertex
ndarray / reference
Python JSON encode of the rows, us per vertex
l3d_sift
128
10000
1.74x
34.55
12.60
2.74x
13.44
l3d_deep
96
10000
5.28x
70.35
12.20
5.77x
33.32
e2_64
64
5000
5.35x
48.59
9.90
4.91x
22.87
The text is 1.7 to 5.4 times the float32 bytes, and the create_vertices load is 2.7 to 5.8 times the 2-D reference.
Proposal: add arcadedb_n_gb_bin(handle, op, json, floats, n_floats) next to arcadedb_n_gb. Python sends one-dimensional non-empty float32 ndarrays as B tags in a side buffer when the library has the entry, and keeps the F text otherwise (a library built before the entry, or ARCADEDB_NATIVE_GB_BIN=0). Lists, float64 arrays and rows without a vector keep their text, and the stored float32 values are bit-identical to today. The JVM wheel is not touched. The PR that follows has the tests and the parity check.
Not claimed: the table is a probe, not a timed A/B. A wheel-versus-wheel A/B with the usual repetitions comes after the PR's CI wheel exists.
Activity