CacheStore never initializes its LRU from an existing cache_store, so a persistent cache directory can grow by a further full max_size for every process that opens it #4282

Zarr version

3.3.0

Numcodecs version

0.16.5

Python Version

3.14.7

Operating System

Windows 11 (10.0.26200); the downstream production impact below was on Ubuntu 22.04

Installation

using pip into a virtual environment

Description

CacheStore keeps its size accounting and LRU order in _CacheState, which is constructed empty (src/zarr/experimental/cache_store.py:125) and is never populated from the cache_store it was handed. With an in-memory cache_store that is exactly right — the cache and the accounting are born and die together. With a persistent cache_store, which is the case the docstring advertises ("This provides persistent caching capabilities", line 42), the two disagree the moment a second process opens the same directory: the files are there, the ledger says zero.

The consequences follow directly:

• A new CacheStore admits up to a full max_size of new values on top of whatever an earlier run left on disk, because none of those bytes are counted.
• Files it did not write are never evicted, because _accommodate_value only walks _state.cache_order (:164).
• A cache hit on such a file does not register it either: _update_access_order is a no-op for keys it has never seen (:230), so the hit is counted in hits but the entry stays untracked and unrankable forever.

So max_size is not a bound on the cache store. It is a per-CacheStore-instance allowance, and the only thing bounding a persistent directory is however many instances you happen to construct.

Expected: on-disk <= max_size after every process, as it is after process 1.
Actual: the directory settles at 2.00x max_size with evictions=0 from process 2 onward, and cache_info() reports current_size=81920 while 163,840 bytes are on disk.

Where it comes from

Why it matters in practice

This is not a corner case for anyone using LocalStore as the cache_store, which is the configuration the class exists to enable. The concrete instance I hit it on: ScrollPrize's villa ink-detection pipeline caches compressed chunks of a public Vesuvius Challenge scroll volume streamed from S3, with a user-set byte budget passed straight through as max_size. On a RunPod pod with a 20 GB disk and a 4 GB budget, a real PHerc1203 render measured 3,999,538,943 bytes after the first pass (correct) and 5,979,366,890 bytes after the second — and that pipeline's own tutorial recommends a 120 GB budget. Every long-running job that restarts, every training run followed by an inference run over the same cache, and every worker pool that re-opens the store pays this.

Downstream, the caller cannot fix it cleanly: cache_info() is read-only and __init__ takes no starting state, so the only available workaround is to write into CacheStore._state after construction. villa is shipping exactly that, guarded and version-checked (ScrollPrize/villa#1587), which is a thing neither project should want to be long-lived.

Suggested fix

Have CacheStore account for the cache store it was given. Roughly, at the end of __init__ when max_size is not None:

1. list() the cache_store,
2. size each key (Store.getsize exists; LocalStore.getsize is a stat),
3. insert them into cache_order / key_sizes / current_size, ordered oldest-first where the backend can supply an age and in listing order where it cannot,
4. and let the existing _accommodate_value bring the store under max_size on the first admission.

Design questions I do not have a strong opinion on, and which are probably why this was left out:

• Cost. Scanning a large cache directory at construction is not free, and CacheStore is constructed per process. An opt-in (seed_from_cache_store: bool = False, or a seed= callable) would sidestep that, at the price of leaving the default wrong.
• Ordering. Store has no generic mtime concept, so exact LRU order is not reconstructible for every backend. Insertion in listing order is still strictly better than not tracking the entries at all — it makes them evictable and countable, which is what max_size needs.
• Freshness. max_age_seconds needs an insertion time per key. Anything seeded has no monotonic timestamp; treating seeded entries as newly inserted is the simple choice, but for a finite max_age_seconds it would resurrect stale entries. Backends that can report mtime should probably use it.

Happy to open the PR if one of these shapes is preferred — say which and I will write it.

This also looks like a blocker for #4228 ("move cachestore out of experimental"): max_size not bounding a persistent cache_store is the kind of thing that should be settled before the class leaves experimental.

I hit this while screening the Herculaneum scroll PHerc1203 on small cloud pods, where the cache directory is shared across sequential render processes and a budget that resets per process fills the disk mid-run.

Disclosure: I used Claude (Anthropic) as a coding assistant to investigate this and draft the text. The repro, the measurements and the downstream numbers were produced and checked in my own sessions, and I have reviewed and edited everything above.

Steps to reproduce

The form requires a script with inline script metadata that reproduces under uv run issue.py. This one is zarr-only, network-free and sub-second.

Canonical copy: trackD/issue_drafts/evidence/zarr_issue_repro_pep723.py

```python
# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
import asyncio
import tempfile
from pathlib import Path

import zarr
from zarr.core.buffer.core import default_buffer_prototype
from zarr.experimental.cache_store import CacheStore
from zarr.storage import LocalStore, MemoryStore

VALUE_BYTES, VALUE_COUNT = 4096, 40
MAX_SIZE = VALUE_BYTES * (VALUE_COUNT // 2)  # room for exactly half the values


def dir_bytes(path: Path) -> int:
    return sum(f.stat().st_size for f in path.rglob("*") if f.is_file())


def dir_files(path: Path) -> int:
    return sum(1 for f in path.rglob("*") if f.is_file())


async def main() -> None:
    print(f"zarr {zarr.__version__}  max_size={MAX_SIZE}  values={VALUE_COUNT}")
    cache_dir = Path(tempfile.mkdtemp()) / "cache"
    cache_dir.mkdir()
    prototype = default_buffer_prototype()

    source = MemoryStore()
    for i in range(VALUE_COUNT):
        await source.set(f"c/{i}", prototype.buffer.from_bytes(bytes([i % 251]) * VALUE_BYTES))

    for process in range(1, 4):
        # Each iteration stands in for a fresh process opening the same cache directory.
        store = CacheStore(store=source, cache_store=LocalStore(cache_dir), max_size=MAX_SIZE)
        info = store.cache_info()
        print(
            f"  process {process} at construction: current_size={info['current_size']} "
            f"cached_keys={info['cached_keys']} (directory already holds {dir_bytes(cache_dir)} bytes)"
        )
        for i in range(VALUE_COUNT):
            await store.get(f"c/{i}", prototype)
        on_disk = dir_bytes(cache_dir)
        print(
            f"  process {process} after reads:    on-disk={on_disk} B / {dir_files(cache_dir)} files "
            f"= {on_disk / MAX_SIZE:.2f}x max_size, evictions={store.cache_stats()['evictions']}"
        )


asyncio.run(main())
```

Output (zarr 3.3.0, verbatim):

```
zarr 3.3.0  max_size=81920  values=40
  process 1 at construction: current_size=0 cached_keys=0 (directory already holds 0 bytes)
  process 1 after reads:    on-disk=81920 B / 20 files = 1.00x max_size, evictions=20
  process 2 at construction: current_size=0 cached_keys=0 (directory already holds 81920 bytes)
  process 2 after reads:    on-disk=163840 B / 40 files = 2.00x max_size, evictions=0
  process 3 at construction: current_size=0 cached_keys=0 (directory already holds 163840 bytes)
  process 3 after reads:    on-disk=163840 B / 40 files = 2.00x max_size, evictions=0
```

Note evictions=0 in processes 2 and 3. Process 2 misses on the 20 values process 1 had already evicted, writes them (reaching exactly max_size on its own books, so it never needs to evict), then hits on the 20 files process 1 left behind — hits that neither count toward max_size nor enter the LRU. Process 3 hits on all 40 and therefore does nothing at all.

Additional output