You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native insert_many: ragged int and float lists (sparse-vector rows) go as one JSON text, not as columns #231
Native insert_many sends a chunk whose rows hold lists of ints (and ragged lists of floats) as one JSON text, because the columnar entry arcadedb_n_insert_colp has no layout for them. Today columnar._list_column returns None for a column of lists of ints, so pack refuses the whole chunk and Database._insert_chunk falls through to arcadedb_n_insert_rows. A fixed-length list of floats for an ARRAY_OF_FLOATS property does go as a float32 matrix; a ragged one, or one with a None row, does not.
The shape that hits it is the sparse-vector lane: Doc(id LONG, tokens ARRAY_OF_INTEGERS, weights ARRAY_OF_FLOATS) with an LSM_SPARSE_VECTOR index on (tokens, weights), loaded with insert_many. The probe (untimed orientation, one run per cell, laptop, 100,000 rows, commit_every=500, wheels of 26.11.1.dev0) logged columns_declined_by_columnar={'tokens': 200} for all 200 chunks of every sparse shape. Net microseconds per row is the wall time minus the row generator:
shape
native net us/row
JVM net us/row
native / JVM
native text chars
JVM text chars
native / JVM chars
sparse30
82.80
39.77
2.08x
81621820
19969590
4.09x
sparse30_noidx
35.60
14.42
2.47x
81621820
19969590
4.09x
sparse_ragged
92.62
66.89
1.38x
88288760
88288760
1.00x
The JVM build sends weights of the fixed-length shapes as a float32 matrix (split_float_vectors) and keeps the rest as JSON text, so on the fixed shape its text is about a quarter of the native text. On the ragged shape both builds send the same text (the two counts are equal) and the native build is 1.38x the JVM per row, so there the native build only gets ahead by leaving the text behind.
Figures are untimed single runs from paper-ingest-probes-20261104/out-20261103 (generated by script from the two logs); a timed wheel-vs-wheel A/B will decide whether the change ships.
Proposal: add four ragged list kinds to the columnar entry, laid out like the existing string column (int32 element offsets in the data buffer, the elements in the aux buffer, optional Arrow null bitmap): I int32 elements stored as int[] (property declared ARRAY_OF_INTEGERS), E float32 stored as float[] (declared ARRAY_OF_FLOATS), Q int64 stored as a list of Long and D float64 stored as a list of Double (undeclared or declared LIST), which is what the JSON text parses to today. A library without the decoder is detected by the absence of a probe entry (arcadedb_n_colp_lists) and keeps such columns on the JSON path, the way lib.NEXT_BIN does for arcadedb_n_next_bin. Anything the kinds cannot carry unchanged (mixed int and float elements, bools, nested lists, NaN or an infinity, an int past 64 bits, an int outside int32 for ARRAY_OF_INTEGERS, another declared array type) stays on the JSON path.
Tests: a spy proves the columnar entry ran and no JSON text was sent, and the stored records (values, Python types, the engine's type(), and vector.sparseNeighbors results through the sparse index) are compared with the JSON path. They run in CI.
Activity