Repository navigation
mirror manifest: decouple Pages deploy from benchmark build; automate incremental hashing #1825
Description
Follow-up to #1815, from an independent review of the shipped work.
- Pages deploy is tied to the benchmark build.
site/mirroronly reaches Pages throughbenchmark-build-comparison.yml. That workflow builds the ESP32-S3 benchmark before it copies the mirror files, so a broken benchmark blocks the mirror deploy. Move the copy and deploy into their own job that doesn't depend on the benchmark job. This touches an existing workflow, not a new file. - New bucket objects are never hashed automatically. Nothing runs
ci/gen_mirror_manifest.py, so new objects stay unverified until someone runs it by hand. Proposed: a scheduled or dispatch run with read-only bucket credentials as secrets that runs the incremental hash and opens a PR. This needs maintainer approval, since it is a new workflow and new secrets. - Generator fixes:
--versionhelp says the default keeps the existing version, but the code writes"1"(ci/gen_mirror_manifest.py:234vs:252).- The change check compares size only (
:149), so an object replaced with same-size bytes is never re-hashed. Compare the ETag or LastModified as well.
- Optional hardening: when the Index lists an owner but that owner's shard can't be fetched, try the vendor URL before unverified bucket bytes. Today it behaves as if the object had no entry.
Out of scope here but worth a decision: fbuild still calls api.registry.platformio.org to resolve packages (crates/fbuild-core/src/platformio_package.rs:14, crates/fbuild-library/src/library/registry.rs:8). If "nothing but the primary download" is meant to cover metadata too, that needs its own issue.
Activity
zackees commentedon Nov 1, 2026
Generator items (3) are in PR #1845.
--versionno longer writes"1"; it keeps the version the shards carry (most common,names-*ignored), falling back to"1"only when there is nothing to keep.site/.mirror-hashes.tsv(beside the published tree, git-ignored,--checkpointstill overrides), and oneis_stale()rule serves both--checkand the hash pass. A record read from a committed shard has no ETag, so the first run after this change compares by size only — the whole mirror is not re-downloaded.finally.ci/test_gen_mirror_manifest.py, registered inci-workflow-drift.ymlandci/local_gate.pyso something actually runs them.Items 1, 2 and 4 remain open here:
benchmark-build-comparison.ymleither races the benchmark deploy or wipes the benchmark stats. The correct shape is a separate workflow triggered onsite/mirror/**that composessite/mirror/with the already-publishedbenchmark-statsbranch and shares the benchmark workflow's concurrency group (otherwise the two deploys clobber each other). That is a design change, not a YAML move — happy to take it as its own PR if that shape is right.crates/fbuild-packages-fetch, not the generator.zackees commentedon Nov 2, 2026
Item 1 (Pages deploy tied to the benchmark) is in PR #1854.
publishnow runs when the benchmark job is finished, not when it succeeded (!cancelled()), so a broken measurement no longer freezessite/mirror/**.benchmark-statsbranch, and the benchmark artifact is overlaid only onneeds.benchmark.result == 'success'— the previous report stays online while the mirror still updates.7 new tests in
ci/test_benchmark_publish.py(registered in the drift workflow +local_gate.py) plus a 3-scenario local-bare-repo E2E of the publish script. Items 2 (scheduled hashing workflow + bucket secrets) and 4 (vendor-URL fallback) are untouched; item 3 shipped as #1845.zackees commentedon Nov 2, 2026
Item 4 (vendor before unverified bucket bytes) is in #1859.
The root cause was an
Option:entry_forreturnedNoneboth for "the indexhas no entry for this URL" and for "the index lists this owner, but its shard
could not be fetched". The second case fell through to the built-in-bucket
precedence, so a shard outage silently tried unverified bucket bytes for the
object before the vendor.
entry_lookupnow returnsFound/NotListed/Unavailable, and an unavailable shard resolves vendor-firstwith the bucket as a last resort (an explicit
FBUILD_DOWNLOAD_MIRROR_FIRSTstill wins). A failed shard load also warns once per process instead of failing
silently. Tests: 2 new (one per module), full crate 252 passed / 0 failed,
bash test4139 passed / 0 failed.Remaining here: item 2 (a scheduled incremental-hash workflow with bucket read
credentials as secrets) — maintainer-only, still open.