Skip to content

Native insert_many: date-times, Decimals, bytes, numpy scalars and pandas NA send the whole chunk through INSERT ... CONTENT per row #217

Closed

Description

@tae898

On the native build, insert_many sends a whole 10,000-row chunk through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many) as soon as one value is a datetime, date, Decimal, bytes, a numpy scalar, or a pandas NA/NaT. The chunk takes that path because _native/database.py::_insert_chunk calls the untagged json_bulk_dumps, which raises on those types. columnar.pack refuses date-times and Decimals too, so the columnar entry does not help.

The JVM build stopped doing this in #82 (json_bulk_dumps_tagged, used by core.py only). Json.java in the native image already decodes the tagged values (T, D, n, b), so the native build needs a Python change only, no native-image rebuild. This was first done in humemai/arcadedb-embedded-python#532, which was closed unmerged when the work moved here.

Untimed, one run each (so no claim): 1,000,000 rows of six properties with a Python datetime per row took 15.2 s through arcadedb_n_exec_many (100 crossings) and 11.2 s through arcadedb_n_insert_rows (100 crossings). A separate untimed probe on the laptop (native, none/one/all rows carrying a datetime) gave 17.7/19.6/24.4 us per row. A timed A/B on the main CI wheel with the PR's files overlaid will follow before any speed claim.

Proposed change, in the PR for this issue: insert_many writes those values as tagged values inside the arcadedb_n_insert_rows text; a row with a value the text still cannot carry (NaN, Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own while the rows around it stay on the bulk entry in the order given.

Activity

  1. added this to the 26.11.1 milestone on Nov 5, 2026
  2. added
    bugSomething isn't working
    nativeThe arcadedb package (native library build)
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingnativeThe arcadedb package (native library build)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions