Publish fbuild-owned sha256 for every bucket object (sharded manifest); registry paths only, no GitHub mirror, no PlatformIO fallback or metadata calls #1815

Follow-up to #1806 / #1810.

Problem

1. Bucket bytes go unchecked for most downloads. fbuild's mirror bucket (fbuild.sfo3.digitaloceanspaces.com) holds the whole PlatformIO registry: 29,300 objects, 102 GB. The server-side manifest (site/mirror/manifest.json) only carries a sha256 for the 16 artifacts named statically in the crates. Every other registry download, including those resolved at runtime, is served from the bucket by the built-in {path} rule with no integrity check.

Non-goal: no GitHub mirroring

The bucket mirrors only the PlatformIO registry layout (<owner>/<type>/<name>/<ver>/<file>) for backward compatibility. GitHub archives and release assets are not mirrored: archive URLs pin arbitrary commits, so a mirror would grow without bound. mirror_path drops the github.com/… mapping that #1806 added, so GitHub downloads go straight to GitHub.

Non-goal: nothing from PlatformIO beyond the primary download

PlatformIO does not want its infrastructure used as a mirror or fallback by other tools. fbuild must not:

• walk the registry's mirror list (?bypass= iteration);
• add any PlatformIO mirror host as a candidate (the GitHub pioreg-ci-mirror releases, Contabo storage, registry.nm1.platformio.org, …);
• send extra requests to the registry for metadata, such as HEADing for X-PIO-Content-SHA256.

The vendor URL fbuild already downloads from stays the only PlatformIO request. Checksums and fallbacks come from fbuild's own infrastructure: the bucket plus documents published from this repo.

Fix

1. fbuild-owned checksums for every bucket object (priority).
   • Hash: a one-off job hashes all bucket objects. Run it on a droplet in sfo3 so egress over the VPC is free. Re-running it is incremental: only keys missing from the published hashes get fetched. Bytes come from the bucket.
     • Cross-check a random sample against the vendor's bytes and record the sample in the job log.
     • Downloading those vendor bytes is a one-time check by a maintainer-run job. It is not part of the client.
   • Publish: the hashes ship as zackees/manifest.json documents on GitHub Pages, never in the bucket itself, so one compromised bucket can't serve both the bytes and the checksum.
     • 29k entries won't fit the 8 MiB single-document cap, so site/mirror/manifest.json becomes an Index whose tools point (via Descriptor{url, sha256, size_bytes}) at per-owner shard Release documents, e.g. site/mirror/shards/<owner>.json.
   • Client (fbuild-packages-fetch):
     • Fetch: fetch the Index once, and fetch only the shard for the owner being downloaded. Verify the shard against its descriptor sha256, then cache it under the same TTL / stale-cache ladder as today.
     • Verify: the shard's sha256 feeds the existing Candidates::mirror_sha256 check, so bucket bytes that don't match are rejected and the vendor URL serves the file.
     • No entry: behaviour stays as today: bucket first via {path}, unverified. This only happens for objects added to the bucket after the last hash run.
   • Generator: ci/gen_mirror_manifest.py learns to emit the Index plus shards from the hash job's output. The 16 statically scanned entries stay covered.
2. Drop the github.com/… key mapping from mirror_path (and the generator) so the bucket and manifest cover only registry paths.

Acceptance

• Validation: Index and shards validate with manifest-validate, and every shard's descriptor sha256 matches its served bytes.
• Tampered shard: a shard whose bytes don't match its descriptor is rejected and the stale shard cache is used.
• Wrong bucket bytes: an unlisted-before, now-sharded registry package whose bucket bytes are wrong is rejected and the vendor serves it. Correct bytes are served from the bucket.
• Shard fetch failures: a shard fetch that times out or returns 404 falls back to unverified bucket-first without failing the download.
• One shard per owner: the client fetches the Index plus only the shard(s) for owners it downloads, never all shards.
• No PlatformIO extras: a test asserts no candidate list contains a PlatformIO mirror host or bypass= URL, and that the only request to *.platformio.org per download is the primary GET.
• End to end: a real-CLI run (fbuild install, fresh cache) against a runtime-resolved registry package shows bucket bytes verified against its shard.
• GitHub not mirrored: a GitHub URL gets no bucket candidate, and the generator's --check passes.