meta: broker-safe global artifact repository feature burndown

?## Context

Issue #601 started as the managed zccache sidecar install gate, then uncovered a broader broker/global-cache design problem: fbuild has large shared artifacts that must remain in one authoritative per-user artifact repository, not private stores scoped by whichever daemon/backend version the running-process broker selects.

This issue is the parent/meta burndown for that feature area. The original investigation remains on #601; implementation slices and completion state should roll up here.

Sub-Issue Map

Parent investigation: fix: gate managed zccache sidecar install and surface blocking status to clients #601
Original broker/cache strategy concern: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
Sub-issue split: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
A: daemon identity and broker resolution: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
B: global artifact repository ownership: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
C: download/install concurrency and client blocking: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
D: compatibility, migration, and recovery policy: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
Synthesis: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)

Goal

Make fbuild broker adoption safe for large shared artifacts by enforcing a clear local daemon/cache identity model, preserving the global artifact repository, and adding install/download gates plus client-visible wait status.

The desired invariant is:

Local fbuild has one authoritative shared artifact repository per daemon identity/cache root. Broker backend version negotiation must not create private package/toolchain/framework/sidecar stores.

Feature Burndown

1. Define daemon/cache identity

Specify the local daemon identity tuple, likely (user, dev/prod mode, canonical FBUILD_CACHE_DIR or default cache root, trust domain).
Document that daemon/backend version is not a cache-owner dimension.
Decide policy for parallel daemon versions against the same cache root:
- strict refusal / single active daemon, or
- allowed only when an explicit cache-schema compatibility check passes.
Keep CI EXPLICIT_INSTANCE "ci-trusted" as an intentional trust-domain exception.

2. Align broker resolution with that identity

Ensure local SHARED_BROKER resolution converges on the daemon identity above.
Prevent broker negotiation from routing clients to a backend whose cache root differs from the legacy direct HTTP daemon identity.
Decide and implement replace/upgrade/refuse behavior when a newer fbuild asks for an existing cache-root identity.
Treat broker CacheRuntime as daemon provenance only, not as an artifact repository root.
Add tests that same FBUILD_CACHE_DIR / dev-mode inputs publish the same CacheData across broker-negotiated daemon versions.

3. Preserve global artifact repository ownership

Document fbuild-owned global roots for package/toolchain/framework archives, managed sidecars, cache data, indexes, temp, logs, locks, runtime, and config.
Clarify which roots may be shared across daemon versions and which are per-runtime/provenance only.
Audit code paths that could accidentally introduce broker-instance or daemon-version path segments into artifact storage.
Add regression tests around fbuild_paths::get_cache_root() / FBUILD_CACHE_DIR and CacheManifest output.

4. Add shared install/download gates

Add a process-shared managed zccache install lock before network fetch begins.
Add package/toolchain/framework install gates below project-lock level so different projects do not race on the same global package root.
Re-check final install state while holding each install lock before downloading.
Prefer unique staging dirs plus atomic final rename for all large installs, matching the safer managed zccache pattern.
Recover cleanly from stale _staging dirs, partial sidecar installs, interrupted downloads, and dead installers.

5. Surface blocked-client status

Expose structured wait/install state: lock name, dependency/package name, version, phase, active installer vs waiter, and human message.
Broadcast phase changes over /ws/status.
Include dependency wait/install state in /health, /api/daemon/info, or a dedicated status endpoint.
Emit periodic NDJSON/status events for streaming builds while waiting on project/package/sidecar locks.
Make non-streaming clients avoid request timeout when intentionally waiting behind another install.

6. Compatibility and recovery guardrails

Keep broker version-block/refusal fatal instead of falling through into an incompatible daemon.
Add cache schema/version compatibility checks before allowing multiple fbuild versions to share a repository.
Guard disk-cache migration ID immutability.
Add stale dead-PID lease recovery tests.
Add interrupted install recovery tests for managed zccache and package/toolchain/framework installs.
Add CI explicit-instance manifest tests so ci-trusted isolation stays intentional and local shared stays shared.

Acceptance Criteria

Local fbuild has a documented daemon/cache identity contract.
Broker-negotiated local daemons cannot silently fork package/toolchain/framework/sidecar artifacts into private versioned stores.
Same cache-root identity resolves to one compatible daemon policy: reuse, upgrade/replace, or clear refusal.
Concurrent fbuild clients/projects do not duplicate large downloads for the same package/toolchain/framework/sidecar.
Blocked clients receive clear status and do not appear hung while waiting on another fbuild operation.
Recovery from stale locks, partial installs, and interrupted downloads is covered by tests.
CI trust-domain isolation remains explicitly tested and separate from local shared-cache behavior.

fix: gate managed zccache sidecar install and surface blocking status to clients #601 tracks the original managed zccache install-gate investigation and broker/cache discussion.
chore: remove zccache crate linkage and use the managed sidecar for cache APIs #600 removed compile-time zccache crate linkage; this tracker is about runtime broker/cache/install policy.

feat: define broker cache identity

fix: verify broker cache identity

fix: address broker cache review feedback

Merged implementation slice #604 (feat: define broker cache identity) into main.

Landed in this slice:

local daemon/cache identity contract and broker cache-ownership labels
cache schema/version diagnostics for future compatibility checks
CI ci-trusted explicit-instance manifest helpers
managed zccache and package install gates before network download/extract
dependency install status propagation through daemon info, websocket/status snapshots, streaming build status events, and CLI status output
non-streaming build/API wait behavior that avoids client-side total timeout while blocked behind intentional install contention
CLI/Python broker daemon cache identity/schema verification before reuse

This advances #603 but does not close it. Remaining parent-burndown work still includes final resolver policy for parallel daemon versions, deeper cache-schema compatibility rules, broader interrupted-install/dead-lease recovery coverage, and any follow-on tests/docs needed to mark the meta issue complete.

Merged recovery/compatibility guardrail slice #607 into main.

Landed in this slice:

locked existing disk-cache migration ids/order as an append-only compatibility prefix
added a dead-PID lease reaping regression test that proves stale lease rows are removed and entries.pinned is recalculated

This advances the #603 recovery guardrails, specifically migration ID immutability and stale dead-PID lease recovery coverage. It does not close #603; remaining parent-burndown work still includes final parallel-daemon resolver policy, deeper cache schema compatibility behavior, interrupted install recovery coverage, and any remaining docs/tests needed for the full broker-safe global artifact repository feature.

Progress update: opened #609 for the interrupted-install recovery guardrail.

This adds regression coverage that a recorded package/toolchain/framework-style install directory without .install_complete is treated as incomplete during disk-cache reconciliation: the directory is removed and the cache index no longer reports it as installed.

Validation run:

cargo test -p fbuild-packages disk_cache::gc --lib
cargo fmt --all --check
git diff --check

test: cover interrupted install reconciliation

Progress update: expanded #609 to cover the managed zccache sidecar interrupted-install path as well.

Additional coverage/change:

while holding the managed zccache install lock, stale sibling .zccache-*.tmp staging directories are removed before a new sidecar download/extract begins
tests verify stale zccache staging cleanup removes only old sidecar temp dirs, preserves fresh temp dirs, and does not touch the final zccache- install directory

Validation run:

cargo test -p fbuild-build managed_zccache --lib
cargo test -p fbuild-packages disk_cache::gc --lib
cargo fmt --all --check
git diff --check

Progress update: opened follow-up PR #610 for managed zccache sidecar staging recovery.

Why follow-up: #609 merged before the sidecar cleanup amendment landed on main, so #610 carries that remaining interrupted-install piece cleanly from current main.

Landed/proposed in #610:

stale managed zccache .zccache-*.tmp staging dirs are removed while holding the sidecar install lock
tests verify stale sidecar cleanup removes only old zccache temp dirs, preserves fresh temp dirs, and ignores the final zccache- install directory
Cargo.lock workspace package versions are synced after the 2.2.30 release bump so locked Cargo runs work from current main

Validation run:

cargo test -p fbuild-build managed_zccache --lib
cargo fmt --all --check
git diff --check

Progress update: opened #611 for the broker cache-root ownership guardrail.

This slice covers the path-layer root ownership item:

documents CacheRoots ownership in fbuild-paths, including FBUILD_CACHE_DIR as the authoritative artifact/index root and CacheRuntime as daemon provenance only
adds regression tests that FBUILD_CACHE_DIR drives CacheRoots::artifact/index
adds regression tests that changing daemon runtime directories does not fork artifact/index/temp/log/lock/config roots
syncs Cargo.lock workspace package versions after the 2.2.30 release bump so locked Cargo runs work from current main

Validation run:

cargo test -p fbuild-paths running_process --lib
cargo test --locked -p fbuild-paths running_process --lib
cargo fmt --all --check
git diff --check

test: document daemon lock recovery status (#615)

test: enforce broker daemon version policy (#616)

Progress update for #603 burndown:

Merged clean follow-up slices into main:

fix: clean stale managed zccache staging dirs #610 managed zccache sidecar staging recovery
test: guard broker cache root ownership #611 broker cache-root ownership guardrail
test: guard broker refusal policy #612 broker refusal compatibility guardrail
test: guard broker cache manifest roots #613 broker cache-manifest root ownership guardrail
test: cover legacy package staging recovery #614 legacy package/toolchain/framework staging recovery coverage
test: document daemon lock recovery status #615 daemon lock recovery/status coverage
test: enforce broker daemon version policy #616 strict broker daemon version-policy guardrail

Opened the next manifest-specific follow-up: #619

#619 covers the remaining explicit-cache-root manifest path:

verifies FBUILD_CACHE_DIR -> CacheRoots::discover -> CacheManifest publishes CacheData as the explicit fbuild cache root
verifies CacheIndex stays below that explicit root
verifies CacheRuntime remains daemon provenance and cannot become the artifact repository root

Local validation for #619:

soldr cargo test --locked -p fbuild-daemon broker::service::tests::cache_manifest_respects_fbuild_cache_dir_as_artifact_owner
soldr cargo fmt --all --check
git diff --check

Also still open: #617, which is the global artifact ownership contract/docs slice; CI was nearly complete when checked.

test: guard global artifact root ownership (#617)

Merged #617 into main.

That lands the global artifact ownership contract slice:

packages, toolchains, platforms, libraries, and core artifacts stay under the authoritative fbuild cache root
managed zccache sidecars stay under the mode/trust-domain get_fbuild_root()/bin/ root rather than broker runtime/cache provenance
docs now separate artifact roots, schema-gated indexes, daemon state, broker runtime provenance, shared temp, and mode config ownership

Still open/running: #619 for the explicit FBUILD_CACHE_DIR broker manifest regression.

Opened another #603 follow-up: #620

This covers the package/toolchain/framework install gate status regression:

verifies a caller waiting behind the process-shared install lock publishes structured wait status
asserts dependency name, version, waiting_for_lock phase, waiter role, lock path, and human wait message
closes a coverage gap around the blocked-client status guarantee for large shared package/toolchain/framework installs

Local validation:

soldr cargo test --locked -p fbuild-packages install_lock::tests
soldr cargo fmt --all --check
git diff --check

Still open/running with no failures when checked: #619 and #620.

test: guard package install lock wait status

Progress update for the #603 burndown:

Opened #621: #621

This adds explicit dev-mode cache root stability coverage for the broker artifact repository contract:

FBUILD_DEV_MODE=1 with no FBUILD_CACHE_DIR keeps the artifact owner at the default dev cache root.
Varying daemon runtime directories does not fork CacheData or CacheIndex ownership.
Runtime paths remain distinct, so daemon-local state can move without changing the global artifact repository identity.

Local validation:

soldr cargo test --locked -p fbuild-paths running_process --lib
soldr cargo fmt --all --check
git diff --check

test: guard manifest FBUILD_CACHE_DIR ownership (#619)

test: guard package install lock wait status

test: guard dev-mode cache root stability

test: guard package install lock wait status (#620)

test: guard dev-mode cache root stability (#621)

Final #603 roll-up:

Merged the remaining feature-burndown slices into main:

test: guard manifest FBUILD_CACHE_DIR ownership #619: explicit FBUILD_CACHE_DIR -> CacheRoots::discover -> CacheManifest ownership regression. This verifies CacheData and CacheIndex stay under the authoritative cache root while CacheRuntime remains daemon provenance.
test: guard package install lock wait status #620: package/toolchain/framework install-lock waiter status regression. This verifies blocked clients publish structured wait state with name, version, phase, waiter role, lock path, and human message.
test: guard dev-mode cache root stability #621: dev-mode default cache-root stability regression. This verifies dev mode keeps one artifact/index owner even when daemon runtime directories vary.

Final local validation on the rebased heads used soldr cargo ...:

soldr cargo test --locked -p fbuild-packages install_lock::tests
soldr cargo test --locked -p fbuild-paths running_process --lib
soldr cargo fmt --all --check
git diff --check

#619 reached a clean GitHub merge state before merge. #620 and #621 were merged after local focused validation, CodeRabbit success, no reported failures, and no required branch checks configured; the full board-build matrix was still queued/non-required after the branch rebase.

With the previous landed slices (#604, #607, #609-#617) plus #619-#621, the parent invariant is now covered: local fbuild has one authoritative shared artifact repository per daemon identity/cache root, broker/runtime provenance cannot become a private artifact store, install/download contention is gated, blocked clients get visible status, and recovery/compatibility guardrails are test-covered.

Navigation Menu

Repository navigation

meta: broker-safe global artifact repository feature burndown #603

Sub-Issue Map

Goal

Feature Burndown

1. Define daemon/cache identity

2. Align broker resolution with that identity

3. Preserve global artifact repository ownership

4. Add shared install/download gates

5. Surface blocked-client status

6. Compatibility and recovery guardrails

Acceptance Criteria

Related

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

meta: broker-safe global artifact repository feature burndown #603

Description

Sub-Issue Map

Goal

Feature Burndown

1. Define daemon/cache identity

2. Align broker resolution with that identity

3. Preserve global artifact repository ownership

4. Add shared install/download gates

5. Surface blocked-client status

6. Compatibility and recovery guardrails

Acceptance Criteria

Related

Activity

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

12 remaining items

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

zackees commented on Jun 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions