CI: linux-arm64 long-term clean() reboot times out under TCG

[mobileskyfi]

Observed

Run: https://github.com/tikoci/quickchr/actions/runs/28758858863
Failed job: https://github.com/tikoci/quickchr/actions/runs/28758858863/job/85270515858 (Integration (linux/aarch64 · long-term))

This is separate from #69. The issue69-focused run https://github.com/tikoci/quickchr/actions/runs/28758125602/job/85268595976 passed on linux/aarch64 · long-term, and the arm64 REST ordering lab also passed in the failing full-suite job.

Failure shape

• Target: long-term, resolved in ci-data as RouterOS 7.21.4.
• Host/job: GitHub hosted Linux arm64, QEMU 8.2.2.
• Accel: quickchr detectAccel(arm64): tcg (crossArch=false, timeoutFactor=4).
• KVM: /dev/kvm absent; explicit KVM open probe failed.
• Test file: start-stop.test.ts failed after 788s.
• Failing test: clean() resets disk to factory defaults — custom users disappear on next boot.
• Failing step: after instance.stop(); await instance.clean();, QuickCHR.start({ name: "integration-clean-test" }) timed out in _launchExisting().
• Error: BOOT_TIMEOUT, CHR did not respond within 480s (accel=tcg).
• qemu.log tail only recorded QEMU termination during cleanup, so it did not prove a QEMU launch error.

Relevant artifact data:

start-stop.test.ts 788s fail
(pass) remove() on a running machine stops QEMU and deletes the directory [30466ms]
(fail) clean() resets disk to factory defaults — custom users disappear on next boot [523342ms]
QuickCHRError: CHR did not respond within 480s (accel=tcg). Machine "integration-clean-test" has been cleaned up automatically.

Comparisons

Same run (28758858863):

• linux/x86_64 · long-term passed.
• macos/arm64 · long-term passed.
• windows/x86_64 · long-term passed.
• linux/aarch64 · testing passed (7.24rc1).

ci-data rollup currently records:

• 7.21.4: fail only on linux-arm64; pass on linux-x86, macos-arm64, windows-x86.
• 7.24rc1: pass on recorded platforms, including linux-arm64.

Why this is interesting

clean() deliberately preserves ARM64 EFI vars because wiping them previously caused a long UEFI device scan. This failure hits the exact 480s same-arch TCG boot timeout after clean, so the investigation should check whether long-term ARM64 CHR is losing/ignoring the expected boot order, whether the cleaned disk path behaves differently on 7.21.4, or whether same-arch TCG simply needs more than 480s for this post-clean boot case.

The recent autosupout.rif observations are worth capturing if RouterOS gets far enough to expose files/logs, but passing #69 probes also generated autosupout entries, so do not assume supout is the root cause without correlation.

Investigation plan

1. Reproduce locally on the Intel Mac with ARM64 CHR under cross-arch TCG using a focused script, not the whole suite: create custom user, stop, clean(), relaunch, capture elapsed times, qemu.log, serial output, and machine dir state before cleanup. Compare long-term, stable, and testing.
2. Re-run focused CI: integration.yml on platforms=linux-arm64, routeros-targets=long-term, test-filter=start-stop.test.ts. If needed, add a temporary lab/probe variant that preserves machine dir and serial/qemu logs on timeout.
3. Bucket the result before changing behavior:
   • boots after >480s: boot-order scan / timeout-envelope problem;
   • never boots and serial/firmware shows disk/UEFI issue: clean()/EFI/disk path bug;
   • only 7.21.4: RouterOS long-term ARM64 behavior difference.
4. Do not skip, platform-gate, or simply bump timeouts until one of those buckets is proven.

[mobileskyfi]

Local Intel Mac repro attempt — no repro

Local host:

• macOS/Darwin x86_64 (process.platform=darwin, process.arch=x64)
• Bun 1.3.13
• qemu-system-aarch64 11.0.0
• forced CHR arch: arm64
• quickchr detected accel=tcg, crossArch=true, timeoutFactor=15
• isolated state/cache: /tmp/quickchr-issue79-data

Probe path matched the failing test’s lifecycle but forced ARM64 on the Intel Mac:

1. QuickCHR.start({ channel, arch: "arm64", user: cleanuser })
2. verify cleanuser REST auth works
3. stop()
4. clean()
5. QuickCHR.start({ name })
6. verify cleanuser is gone (401) and factory admin: works (200)

Results:

target      version   first start ms   first boot ms   post-clean start ms   post-clean boot ms   old user   admin
long-term   7.21.4   48362            34014           35000                 34313                401        200
long-term   7.21.4   42165            34014           34757                 34103                401        200
long-term   7.21.4   42789            34346           39761                 39113                401        200
long-term   7.21.4   47675            39167           87634                 86916                401        200
stable      7.23.1   48361            34078           34795                 34126                401        200
testing     7.24rc1  49351            34075           34727                 34071                401        200

The exact local x86/HVF baseline also passed:

QUICKCHR_DATA_DIR=/tmp/quickchr-issue79-x86-data \
HOME=/tmp/quickchr-issue79-x86-home \
QUICKCHR_TEST_TARGET=long-term \
QUICKCHR_INTEGRATION=1 \
bun test test/integration/start-stop.test.ts --test-name-pattern 'clean\(\)'

Result: 1 pass, clean test completed in 84564.90ms.

No leftover machine files in either isolated data dir after success, and no QEMU processes were left running.

Interpretation

This argues against a generic clean() bug, a generic 7.21.4 ARM64 CHR clean/reboot bug, or preserved efi-vars.qcow2 being fundamentally broken. The local post-clean ARM64 boots were normally ~34-40s, with one slower ~87s run, still far below the CI 480s timeout.

It does not rule out the CI failure. The failing job was a different execution bucket:

• CI: Linux arm64 host, same-arch TCG, QEMU 8.2.2, after many prior full-suite boots in the same job.
• Local: Intel macOS host, cross-arch TCG, QEMU 11.0.0, focused clean repro only.

Next useful check

Run focused CI repeats on the actual failing bucket before changing code:

gh workflow run integration.yml --ref main \
  -f platforms=linux-arm64 \
  -f routeros-targets=long-term \
  -f test-filter=start-stop.test.ts \
  -f run-examples=false

If start-stop.test.ts alone passes repeatedly on linux-arm64/long-term, then the failure is probably full-suite interaction / cumulative runner state / load. Next narrowed matrix would be provisioning.test.ts,start-stop.test.ts, then a lab variant that preserves the post-clean machine dir and serial output on timeout.