simple_download(): return server-reported digest instead of computing a local streaming hash #79

Proposal

Make simple_download() not compute a local streaming SHA-256. Instead, return the digest the server reports in its response headers (or None if the server sent none), leaving any verification to the caller.

This is essentially the original requests-based behavior from objectindex's old simple_download, but with the bugs fixed.

Current behavior

simple_download() streams the body through hashlib.sha256() (via a WRITEFUNCTION closure), then verifies the computed digest against the server's Repr-Digest header, raising ClientError on mismatch, and returns the locally computed digest (always 32 bytes, never None).

Proposed behavior

• Drop the local hashing: stream straight to disk (WRITEDATA=open_file instead of a hashing WRITEFUNCTION closure).
• Parse the server digest from Repr-Digest, falling back to Content-Digest, and return it (bytes or None).
• No in-function verification (there is nothing locally computed to compare against).
• Keep FOLLOWLOCATION, the partial-file cleanup on pycurl/HTTP errors, and the ClientError raising for HTTP >= 400.

Return tuple stays (digest, mime, sugg_fname, mtime); only digest changes meaning (server-reported, may be None).

Bugs to fix vs. the original objectindex implementation

1. Read the right header. The old code read only Content-Digest, but this object server replies with Repr-Digest — so it effectively never received a digest from its own server. Read Repr-Digest (with Content-Digest as a fallback for arbitrary sites).
2. The objectindex caller already compares with .hex() and guards if dl_cksum: / if checksum_val:, so it tolerates a None digest.

Caller impact (objectindex)

None required — both call sites already handle None and .hex():

• download() still recomputes from disk via file_checksum, so it remains fully verified; the server-header compare becomes a cheap independent sanity check (no longer redundant with the disk recompute).
• upload_remote() passes the (possibly None) digest to upload_core, which recomputes from disk regardless and only asserts when a digest is present.

Tradeoff

The downside is upload_remote() ingesting arbitrary external URLs: those rarely send a digest header, so there's no longer a wire-vs-disk integrity check during download (matches the original objectindex behavior). download() from this object server loses nothing because it always re-hashes the file. The win is dropping one redundant hash per flow (the streaming hash is the cheap one; the disk re-read remains either way).

Tests

objectindex's tests/test_simpler_objects.py / tests/test_client_io.py are being made tolerant of either behavior (streaming-hash or header-only) so this can land without lockstep coordination.