[Epic] Sleeping-disk strategy: cache metadata + power-state flag + 503-retry + partial results #78

Coordinating issue for the "sleeping HDD" class of problems. Lengthening timeouts (#72) has been the tactical stopgap, but it's the wrong permanent architecture: a blanket long timeout makes every hard-down node cost ~16s on every request, ties up locator tasks / the oneshot replicator queue, and still doesn't bound tail latency. This epic lays out the strategic direction and the order to build it.

Root cause: two failure modes the design currently conflates

Every linked issue stems from treating these as the same event:

• asleep-but-healthy — answers in ~15s, has real data. We want to wait for it / retry it.
• hard-down — never answers. We want to skip it instantly.

A short timeout calls the sleepy node "down" (→ false 404 #75, duplicate write #76, spurious 507 / aborted replication #77). A long timeout calls the down node "sleepy" (→ pay 16s for nothing). No single timeout is correct, because the timeout is being asked a question it structurally can't answer. The strategy is to (a) remove the question for most traffic and (b) answer it directly for the rest.

Strategic layers (not mutually exclusive)

1. Cache metadata in RAM on the object server — root-cause fix, primary lever (#1, #3)

The traffic causing the whole class is metadata (existence, listings, checksums, health), which is small and changes only on write. The object server is the sole writer of its own filesystem, so it can authoritatively cache its own metadata in RAM and answer HEAD/existence/listing without touching the platter. Only actual object-byte GET/PUT then needs the disk — and those already carry long timeouts where a 15s spin-up is negligible against a multi-GB transfer.

This serves the actual goal (we want disks asleep to save power): timeouts/retries keep poking disks; caching lets them stay asleep and keeps metadata fast.

Elegant fit with the existing contract: <bucket>.sha256 is already an on-disk metadata index, and the checksum-append is the commit marker, so the committed-object set is exactly the parsed .sha256 files. "Cache metadata" largely means load the .sha256 files into RAM at startup and update on each PUT-commit. Disk stays source of truth; the cache is a derived in-memory index, rebuilt on restart (scrub already walks the dir). Keeps the "pure filesystem state, no database" design goal intact — no new persistent state.

Payoff: once a healthy node always answers metadata fast, a slow metadata response genuinely means down → metadata timeouts can shrink back to short and fast-fail correctly. This dissolves #72/#75/#76/#77 rather than papering over them.

2. Disk power-state health flag — cheap complement (second priority)

Object-server /health reports power state (awake / asleep / recently-spun). Lets the locator (a) deliberately prefer an awake replica for data GET/PUT instead of relying on the current 1/3-SSD luck, and (b) distinguish asleep-vs-down so 503-vs-skip decisions are honest. Caveat: querying power state (e.g. hdparm -C) must not itself wake the disk — solvable but platform-dependent.

3. Respect 503 + Retry-After on the data path — the right contract for the residual

For object-byte transfer the disk must wake. A short locator timeout + honest 503 Retry-After ("a copy exists, disk spinning up, retry in N") beats a long blind wait: the client gets an explicit signal and the locator task isn't pinned. Only safe combined with (1)/(2) — otherwise a hard-down node makes clients retry forever, and metadata that should be cached-instant would 503. Requires a bounded, shared retry helper in the client lib + replicator.

4. Partial results for GET /{bucket}/ — endpoint-specific resilience (#63)

One slow/down server shouldn't sink the whole listing. The response already carries per-object locations/error; extend it to flag which servers were unreachable/stale. Independent availability win; pairs with the flag in (2).

5. Long timeouts — demote to backstop, not strategy

Keep only as (a) the transitional tactic already shipped in #72 and (b) the floor for the in-flight transfer itself (#49). Walk the metadata timeouts back down once (1) lands.

Recommended sequencing (by leverage)

1. Object-server metadata cache (from .sha256 + dir; update on checksum-append commit; rebuild on start). Then revert the tactical metadata timeouts to short.
2. Disk power-state health flag.
3. Formalize 503-Retry-After on the data path + bounded shared retry helper (client + replicator). Safe once (2) prevents retry-forever.
4. Partial results for GET /{bucket}/ (GET /{bucket}/: optional override to return partial results when a server is down #63).

Net end state: metadata is fast and fast-failing (no spin-up in the hot path), the data path waits only when there's genuinely data worth waiting for and says so explicitly, and a single down node degrades instead of failing the request. Long timeouts stop being load-bearing.

Risks / open questions

• Cache memory footprint per disk — bounded by the .sha256 index it derives from (name+size+digest), but verify on target hardware with the largest expected object counts.
• Write-path cache coherence — easy in principle (sole writer; hook the existing checksum-append commit point), but needs care around partial/failed PUTs and the scrub/rebuild path.
• "Check power state without waking the disk" — confirm the chosen mechanism (hdparm -C or a last-IO heuristic) genuinely doesn't spin the platter.
• Whether the locator should also cache (server up/down, recent listings) layered on top of object-server caches.

Linked issues

• locator API gives 503 if even one obj server takes 2 seconds #72 — bucket-listing 503 (read path; tactical timeout fix shipped)
• find_object returns false 404 for objects whose only copy is on a sleeping disk #75 — read-path false 404 (sole copy on sleeping disk)
• add_object can write a duplicate copy when the existing copy is on a sleeping disk #76 — write-path duplicate (existence check misses sleeping copy)
• Complete the sleeping-disk timeout sweep on the write & replication paths #77 — write/replication path: spurious 507 + replicator direct HEADs
• GET /{bucket}/: optional override to return partial results when a server is down #63 — partial results for GET /{bucket}/
• is 34 minutes long enough to replicate huge files? #49 — data-path streaming timeout (the legit long-timeout case)
• Locator performance enhancement on GET #1, Locator periodically read whole list of files #3 — locator/metadata caching and periodic file-list reads