Orphaned mkdir install-locks deadlock after daemon crash; stale daemon endpoint survives 'daemon stop'; esptool provisioning failure kills daemon #1213

1. Orphaned mkdir install-locks deadlock permanently after a daemon crash

fbuild_packages_fetch::install_lock uses directories as mutexes:

~/.fbuild/prod/cache/toolchains/tasmota-esptool/<hash>/.5.3.0.install.lock
~/.fbuild/prod/cache/toolchains/toolchain-riscv32-esp/<hash>/.riscv32-esp-elf-...install.lock

If the holding process dies (mine died from concurrent-compile contention), the directories survive and there is no staleness timeout and no PID/owner check. Every subsequent build blocks forever:

fbuild: waiting for another process to install esp32-riscv-gcc riscv32-esp-elf-14.2.0_...
fbuild: waiting for another process to install tool-esptoolpy 5.3.0

I confirmed no fbuild process was alive while these repeated for ~40 minutes. Manually rm -rf-ing the two lock directories cleared it immediately and the install then completed normally.

Worth noting the diagnostic trap: because they are directories, the usual "is this lock held?" probe (File.Open(..., FileShare.None)) returns Access is denied, which reads as "held by a live process" when nothing holds it.

Suggested fix: write owner PID + timestamp into the lock dir, and treat it as stale when the PID is gone or the age exceeds a bound. A fbuild daemon stop / fbuild clean --locks escape hatch would also help — neither daemon stop nor deleting the port/pid files clears these.

2. daemon stop does not clear a stale endpoint; clients keep dialing a dead port

After the crash, clients kept failing with:

daemon error: request failed: error sending request for url (http://127.0.0.1:52289/api/deploy)

fbuild daemon stop reported daemon is not running while ~/.fbuild/prod/daemon/ still held daemon-<hash>.port (52289), fbuild_daemon.pid, and daemon_status.json for a dead PID. Deleting them by hand did not help either — the next invocation recreated the same records and redialed 52289.

Suggested fix: validate the recorded PID before reusing a cached endpoint, and have daemon stop remove the port/pid/status records even when it believes the daemon is already down.

3. Freshly-installed esptool exits status 2, and the daemon then dies

Once the locks were cleared, the install succeeded and then:

INFO  downloaded esptool-windows-amd64.zip (63677179 bytes)
INFO  installed tool-esptoolpy v5.3.0
WARN  could not provision esptool: package error: cached esptool executable
      ...\tasmota-esptool\<hash>\5.3.0\esptool-windows-amd64\esptool.exe
      exited with status 2 (falling back to esptool on PATH)

The deploy then failed with the daemon connection error above, and the daemon PID was dead afterwards. So a provisioning failure appears to take the daemon down rather than surfacing a clean error to the client.

Suggested fix: surface the underlying esptool stderr (status 2 with no output is undiagnosable), and make a provisioning failure a recoverable client-visible error rather than a daemon death.

Environment

• Windows 10 Pro 19045, fbuild 2.5.2
• Target: ESP32-C6-DevKitC-1 (esp32c6, riscv32) on COM9
• Trigger: two concurrent ESP32 compiles against one daemon (my fault — but the recovery path is the issue)

Impact

This blocked on-device validation of FastLED/FastLED#3764 entirely. Every failure mode above was silent at the client: bash compile exited 0 while printing ERROR! There were 1 failures, and fbuild deploy exited 1 printing only a version-upgrade notice. The real cause was only visible in ~/.fbuild/prod/daemon/daemon.log.