Skip to content

vectors: the per-bucket PRODUCT guard refuses small buckets the engine searches exactly #173

Closed

Description

@tae898

VectorIndex._ensure_product_quantization_ready() (src/arcadedb/vector.py, called by find_nearest, find_nearest_approximate and build_graph_now) raises "PRODUCT quantization in this ArcadeDB version requires at least pq_clusters indexed vectors per bucket" when a bucket index holds fewer vectors than pq_clusters. It was added in March (fc44025, ArcadeData/arcadedb#3723). Since ArcadeData/arcadedb#5417 (b37d610cd0, July), which every engine these wheels ship includes, the engine handles that case itself: a bucket with fewer vectors than pq_clusters is built and searched exactly, without PQ (LSMVectorIndex.isPQTrainable, and findNeighborsFromVectorApproximate falls back to the exact path).

Measured on both 26.11.1.dev0 dev wheels, 4 buckets of 5, 15, 16, 17 and 40 vectors, 8-d and 32-d, pq_clusters 16: with the guard bypassed, the JVM build behaves exactly like the native build. Below 16 per bucket the build succeeds, PQ is unavailable per bucket, and find_nearest_approximate returns the exact neighbours with exact scores (recall@1 and @5 of 1.0, score difference 0); from 16 per bucket up PQ is trained. With the guard, the JVM build raises at 5 and 15 per bucket. The native build applies the same check to the whole type (one shim), so it can refuse a small type that the engine would search exactly.

Fix: remove the guard and its calls, and replace the tests that pin it with a shared test of the engine behaviour (a small PRODUCT index builds and its approximate search returns the exact nearest neighbour).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions