add_object can write a duplicate copy when the existing copy is on a sleeping disk #76

Split out from the #72 sleeping-disk timeout work. This is the write-path correctness analogue of #75 (#72 = read 503, #75 = read false-404, this = write false-"doesn't-exist").

Problem

Before choosing a write target, add_object (PUT /{bucket}/{key} on the locator) fans out a HEAD to every server to confirm the key doesn't already exist (locator_api.py:97):

async def check_exists(server):
    try:
        result = await client.head(server + object_path, timeout=1)
        return server, result.status_code
    except httpx.HTTPError:
        return server, None
...
for server, status in exist_results:
    if status is None:
        candidates.pop(server, None)
    elif status != 404:
        raise HTTPException(status_code=409)

A sleeping HDD takes ~15s to spin up (per #72, "even 15 seconds is normal"), so the timeout=1 HEAD against it raises and returns status is None. The code treats that as "no answer, drop this server from the write candidates" and proceeds — it never learns the object already exists there.

If the only existing copy of that key is on the sleeping server, the locator picks a different server and writes the object there. The result is two copies of a key the system treats as unique and immutable — objects are O_CREAT | O_EXCL, never overwritten, and the whole design assumes a key resolves to one logical object. The duplicate is silent (no 409), and the two copies can later diverge in how readers/replication see them.

This is a correctness symptom (a present object reported absent, leading to a duplicate write), distinct from the availability symptoms in #72.

Notes / options

• The intended 409-on-conflict path only fires for an explicit non-404 status. A timeout currently can't yield 409 because we can't tell "absent" from "asleep" in 1s.
• Lengthening the HEAD timeout to tolerate spin-up (~16s) would let the existence check actually see the sleeping copy, at the cost of slower PUTs when a disk is cold.
• Alternatively, distinguish "timed out / unreachable" from "confirmed 404": if any server's existence check is merely unreachable (not a confirmed 404), refuse to write (e.g. 503 + Retry-After) rather than risk a duplicate — fail safe instead of duplicating.
• Metadata caching / a "disk asleep" health signal would let the locator answer without waking the disk (see locator API gives 503 if even one obj server takes 2 seconds #72 discussion, Locator performance enhancement on GET #1, Locator periodically read whole list of files #3).

Related

• locator API gives 503 if even one obj server takes 2 seconds #72 — bucket-listing 503, same sleeping-disk root cause
• find_object returns false 404 for objects whose only copy is on a sleeping disk #75 — read-path false 404 (sole copy on a sleeping disk)
• Locator performance enhancement on GET #1, Locator periodically read whole list of files #3 — metadata caching