algo.apsp materialises n² Result objects eagerly, so the call the working-memory budget capped at 64MB then allocates gigabytes it never sees

[lvca]

Surfaced while implementing #6263 (PR #6285), and deliberately not folded into it: that PR bounds the working set of an algo.* call, and this is the result set of the same call. Same failure mode, one level up, and for algo.apsp the result set is the larger of the two by more than an order of magnitude.

The gap

AlgoAPSP.execute() builds its rows eagerly and only then hands back a stream:

final List<Result> results = new ArrayList<>();
for (int i = 0; i < n; i++) {
  for (int j = 0; j < n; j++) {
    if (i == j || dist[i][j] >= INF)
      continue;
    final ResultInternal r = new ResultInternal();
    r.setProperty("source", graph.getRID(i));
    r.setProperty("target", graph.getRID(j));
    r.setProperty("distance", dist[i][j]);
    results.add(r);                        // up to n² - n of these, all live at once
  }
}
return results.stream();

One row per reachable ordered pair. On a connected graph that is n² - n rows, every one of them a ResultInternal with a backing map of three properties, all alive simultaneously before the caller sees the first row.

Why the #6263 budget makes this worse, not better

#6263 caps the n x n distance matrix at arcadedb.cypher.algoMaxWorkingMemory, default max(64MB, maxHeap/8). At the 64MB floor that admits roughly n = 2890 (n x (32 + 8n) ≈ 67 MB).

A connected graph of 2890 nodes then produces about 8.3 million Result objects. At a conservative hundred-plus bytes each - the object, its property map, the map's entries - that is well over a gigabyte, against the 64 MB the budget just finished enforcing on the matrix beside it.

So the budget is doing its job and the call still dies, for a reason the error message will not mention. That is precisely the shape #6065, #6216 and #6263 exist to eliminate: the largest allocation of the call sitting outside the thing that bounds allocations.

Why eager at all

Nothing requires it. The four embedding procedures already return lazily from the same base class:

return IntStream.range(0, n).mapToObj(i -> { ... });

algo.apsp can stream its pairs the same way - the distance matrix is complete before the first row is emitted, so a lazy IntStream.range(0, n * n) (or a flat-mapped nested range) filtered on i != j && dist < INF produces identical output with O(1) row-side memory. The consumer then decides how much it holds, which is the correct place for that decision.

Worth checking in the same pass whether the row count should also be counted against a bound: n² rows is a lot to return whatever the memory shape, and a LIMIT-aware or top-k form of the procedure may be the honest interface. That is a design question rather than a bug, and separable from the eager/lazy fix.

Scope

The other algo.* procedures that build a List<Result> eagerly are fine as they stand - their row counts are bounded by a top-k (algo.knn by topCount), by a component count, or by a single node's neighbourhood. algo.apsp is the one whose row count is quadratic in the graph, which is what makes it the outlier rather than one of a family.

Secondary, same area, much smaller

AbstractAlgoProcedure.toEmbeddingList(double[]) boxes every element into a List<Double>: ~24 bytes per dimension against the 8 the double occupies, so the returned form of an embedding is three times the size of the matrix row #6263 prices. Because those procedures stream lazily, this is throughput and GC pressure rather than a peak - a 1M-node run at the default dimension 128 churns ~3 GB through the young generation for rows that die immediately - so it is a smaller and different problem from the one above. Fixing it means a primitive-backed List<Double> view rather than a copy, and is only worth doing if a profile says the boxing shows up.

Related

#6263, PR #6285, #6216, #6065, #6289.