Skip to content

Native insert_many: ragged int and float lists (sparse-vector rows) go as one JSON text, not as columns #231

Open

Description

@tae898

Native insert_many sends a chunk whose rows hold lists of ints (and ragged lists of floats) as one JSON text, because the columnar entry arcadedb_n_insert_colp has no layout for them. Today columnar._list_column returns None for a column of lists of ints, so pack refuses the whole chunk and Database._insert_chunk falls through to arcadedb_n_insert_rows. A fixed-length list of floats for an ARRAY_OF_FLOATS property does go as a float32 matrix; a ragged one, or one with a None row, does not.

The shape that hits it is the sparse-vector lane: Doc(id LONG, tokens ARRAY_OF_INTEGERS, weights ARRAY_OF_FLOATS) with an LSM_SPARSE_VECTOR index on (tokens, weights), loaded with insert_many. The probe (untimed orientation, one run per cell, laptop, 100,000 rows, commit_every=500, wheels of 26.11.1.dev0) logged columns_declined_by_columnar={'tokens': 200} for all 200 chunks of every sparse shape. Net microseconds per row is the wall time minus the row generator:

shape native net us/row JVM net us/row native / JVM native text chars JVM text chars native / JVM chars
sparse30 82.80 39.77 2.08x 81621820 19969590 4.09x
sparse30_noidx 35.60 14.42 2.47x 81621820 19969590 4.09x
sparse_ragged 92.62 66.89 1.38x 88288760 88288760 1.00x

The JVM build sends weights of the fixed-length shapes as a float32 matrix (split_float_vectors) and keeps the rest as JSON text, so on the fixed shape its text is about a quarter of the native text. On the ragged shape both builds send the same text (the two counts are equal) and the native build is 1.38x the JVM per row, so there the native build only gets ahead by leaving the text behind.

Figures are untimed single runs from paper-ingest-probes-20261104/out-20261103 (generated by script from the two logs); a timed wheel-vs-wheel A/B will decide whether the change ships.

Proposal: add four ragged list kinds to the columnar entry, laid out like the existing string column (int32 element offsets in the data buffer, the elements in the aux buffer, optional Arrow null bitmap): I int32 elements stored as int[] (property declared ARRAY_OF_INTEGERS), E float32 stored as float[] (declared ARRAY_OF_FLOATS), Q int64 stored as a list of Long and D float64 stored as a list of Double (undeclared or declared LIST), which is what the JSON text parses to today. A library without the decoder is detected by the absence of a probe entry (arcadedb_n_colp_lists) and keeps such columns on the JSON path, the way lib.NEXT_BIN does for arcadedb_n_next_bin. Anything the kinds cannot carry unchanged (mixed int and float elements, bools, nested lists, NaN or an infinity, an int past 64 bits, an int outside int32 for ARRAY_OF_INTEGERS, another declared array type) stays on the JSON path.

Tests: a spy proves the columnar entry ran and no JSON text was sent, and the stored records (values, Python types, the engine's type(), and vector.sparseNeighbors results through the sparse index) are compared with the JSON path. They run in CI.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions