tests: intermittent ECONNRESET on first REST probe with freshly-provisioned credentials (arm64-biased)

[mobileskyfi]

Evidence

Two consecutive main.yml runs on 2026-07-05 (run 28728342669, first run + rerun) failed the linux/aarch64 integration leg with the SAME error class but DIFFERENT tests:

• run 1: user provisioning > background mode: custom user created and accessible via REST — error: The socket connection was closed unexpectedly. For more information, pass verbose: true in the second argument to fetch() (~42s)
• rerun: instance lifecycle — remove and clean > clean() resets disk to factory defaults — same fetch() error (~42s)

The same suite passed on arm64 earlier the same day (runs 28723050898, 28727737748) — intermittent, arm64-biased. The arm64 REST ordering repro (test/lab) PASSED in the failing run, so the library's node:http path is fine.

Cause class

The error message is Bun's fetch() — but .github/instructions/general.instructions.md / routeros-rest.instructions.md ban fetch() for CHR REST precisely because of Bun's connection reuse against RouterOS (the documented sessions-008-017 lesson; src/lib/rest.ts exists because of it). The integration tests themselves still use raw fetch() against 127.0.0.1:<http>/rest/… in many places (provisioning.test.ts ×10+, start-stop.test.ts, others) — test code was never migrated when the library was.

Done-when

• A small test helper (node:http, agent:false, custom-auth capable — thin wrapper over src/lib/rest.ts) replaces every fetch() call that targets a CHR REST endpoint in test/integration/ (external-URL fetches stay).
• Full integration suite green on linux/x86_64 + linux/aarch64.

🤖 Generated with Claude Code

[mobileskyfi]

Correction after deeper log analysis — the initial attribution was wrong.

The rerun's failing call (clean() resets disk test) is a restGet — already node:http with agent:false, NOT raw fetch(). Bun's error message mentions fetch() for ALL socket-level failures (its node:http is built on the same stack), which misled the first diagnosis. Raw evidence from run 28728342669 (rerun):

error: The socket connection was closed unexpectedly …
  path: "http://127.0.0.1:9100/rest/system/resource"
  code: "ECONNRESET"

Revised signature: both arm64 failures were a fresh TCP connection using freshly-provisioned non-admin credentials immediately after boot/provisioning → ECONNRESET from the CHR (~42s test time = boot + first probe). Working theory: RouterOS's www service resets connections around user-table changes shortly after boot; arm64's slower settling widens the race. The lab REST-ordering repro passed in the same runs, so the library's request path is fine.

Revised done-when:

1. Ground the theory first (lab experiment: create user via provisioning, hammer REST with the new credentials during the settling window, observe RST behavior — x86 vs arm64).
2. Then a bounded retry-on-ECONNRESET/ECONNREFUSED (few seconds, connection-class errors only — never retries wrong data) in the tests' first-probe helpers, citing the repro.
3. The 'replace raw fetch() in provisioning.test.ts' cleanup is still worthwhile hygiene (12 call sites) but is no longer believed to be the cause.

🤖 Generated with Claude Code

[mobileskyfi]

Working theory: RouterOS's www service resets connections around user-table changes shortly after boot; arm64's slower settling widens the race.

It's maybe a little deeper.

So maybe overly a ping in "Ground the theory first", as a perhaps it a port flap, which reset connection when L2 goes down. But that flap may have ill effects on SLiRP.

It also be curious to know if native-api has same issue, as if it mere www, native-api might still be "up" as REST API essentially proxies to native-api internally, but clear if two are bound in the "www service resets connections around user-table changes shortly after boot" theory, as that point more directly at a L2 flap (or similar).

[mobileskyfi]

New occurrence — and it materially updates the client-side picture.

Run https://github.com/tikoci/quickchr/actions/runs/28749821964 (main push, linux-arm64, RouterOS 7.23.1):

(fail) provisioning corner cases > createUser sets the user group to 'full' by default [42146.43ms]
error: The socket connection was closed unexpectedly ... code: ECONNRESET
  path: http://127.0.0.1:9100/rest/system/resource
  at test/integration/provisioning.test.ts:417

The reset hit the FIRST request authenticated as the just-created groupcheck user — a raw fetch() call this time, where the previously logged occurrence was restGet (node:http). Same failure shape on two different HTTP clients kills any remaining client-stack theory (and with it the 'optional hygiene' idea of swapping the test's raw fetch() for restGet — no reason to expect that to help). The consistent invariant across both: first request with freshly-provisioned credentials, arm64, immediately after user creation — the settling-window theory stands, now double-grounded.

Also matches the pattern: the request one line earlier (same connection target, admin credentials) succeeded; the very next request with the new user's Basic auth got the reset. Failed jobs re-run to reopen the freshness gate; this comment is the durable record.

[mobileskyfi]

Investigation plan, not a fix yet.

I pulled the issue text/comments, Actions artifacts, and the relevant job logs. Grounded observations:

• Run 28728342669 attempt 1 failed in provisioning.test.ts at the first testuser REST probe after QuickCHR.start({ user }) returned. Artifact boot record: integration-prov-bg, RouterOS 7.23.1, arch=arm64, accel=tcg, boot_ms=33835.
• The rerun failed in start-stop.test.ts at the first cleanuser REST probe after the custom-user boot; same URL and ECONNRESET. Boot record: integration-clean-test, arch=arm64, accel=tcg, boot_ms=33832.
• The observed failing product path is faithful and narrow: QuickCHR.start({ user }) -> createUser() -> first request using the newly created credentials. No observed failure involved disableAdmin.
• The latest checked linux/x86_64 job says KVM is available — hardware acceleration active and logs - Accel: KVM. The linux/aarch64 job logs Failed to open the device kvm: Invalid argument, then KVM not available — integration tests will use TCG, and logs - Accel: TCG (no /dev/kvm).
• That means the current arm64-biased signal may really be TCG-biased: linux/x86_64 gating is KVM, while linux/aarch64 gating is the only gating leg running under TCG.
• Both fetch() and restGet have now hit the same class, so replacing the test HTTP client is not a root-cause investigation.

Implications for repro fidelity:

• Local Intel Mac arm64 CHR is useful for building and debugging instrumentation, but it is cross-arch TCG: x86_64 host emulating an arm64 guest. The CI linux/aarch64 leg is same-arch TCG: arm64 host running arm64 guest without KVM. Those are materially different QEMU timing paths.
• Treat the focused CI run on linux/aarch64 as the source of truth for this issue. Treat local Intel Mac loops as cheap development loops unless they reproduce the failure.
• A clean 5-iteration local arm64/TCG run should be called inconclusive, not exculpatory. The observed failure rate is low enough that small clean samples do not rule anything out.

Proposed lab:

Create a focused, non-gating lab around the user-auth settling window, e.g. test/lab/provisioned-user-settling/repro.test.ts. It should not try to make the suite green. It should run many short observations and emit NDJSON/console rows with timestamps and outcomes.

For each iteration:

1. Boot a fresh CHR without user provisioning, then call the exported createUser(httpPort, name, password) directly. That keeps the product path involved but lets the lab instrument before, during, and immediately after user creation.
2. Record timestamps for: boot-ready, /user/add start/finish, admin-side /rest/user visibility, and first success/failure using the new credentials.
3. Probe with admin credentials and new-user credentials separately against /rest/system/resource and /rest/user.
4. Probe the new credentials with both restGet and raw fetch() only as labels. If both fail similarly, the client stack stays ruled out.
5. If a new-credential probe gets ECONNRESET, immediately retry the identical request on a fresh connection and record whether attempt 2 succeeds. That is the cheapest signal for one-shot first-lookup/auth-table miss versus a several-second reset window.
6. Run a parallel TCP probe loop every 100-250 ms for instance.ports.http, instance.ports.api, instance.ports.ssh, and maybe instance.ports.winbox for ~15s around user creation. This separates www/auth reset from all forwarded services, SLiRP, or guest NIC flapping.
7. Native API: start with raw TCP connect stability on instance.ports.api. If HTTP resets while API TCP stays stable, add a minimal RouterOS API login probe or tiny external API client to test whether the new user can authenticate there. Do not spend that effort until port-level data says it matters.
8. Enable RouterOS memory logging before creating the user and collect logs after failure/success. If REST is unavailable, use serial console for /log/print detail and /ip/service/print so the lab still gets guest-side evidence.

Local Intel Mac path:

# Fast control: x86 CHR under HVF on this Intel Mac.
QUICKCHR_INTEGRATION=1 QUICKCHR_ISSUE69_ARCH=x86 QUICKCHR_ISSUE69_ITERATIONS=20 bun test test/lab/provisioned-user-settling/repro.test.ts

# Instrumentation dev loop: arm64 CHR under cross-arch TCG on this Intel Mac.
# A clean result here is inconclusive; CI linux-arm64 remains the source of truth.
QUICKCHR_INTEGRATION=1 QUICKCHR_ISSUE69_ARCH=arm64 QUICKCHR_ISSUE69_ITERATIONS=5 bun test test/lab/provisioned-user-settling/repro.test.ts

# Optional packet capture when chasing RST vs close vs refusal.
sudo tcpdump -i lo0 -nn -s0 -w /tmp/quickchr-issue69-lo0.pcap "tcp port 9100 or tcp port 9103 or tcp port 9102"

Focused CI path:

The current integration.yml test-filter only resolves files under test/integration/, not test/lab/. So there are two clean options:

1. Put the probe temporarily or intentionally under test/integration/issue69-settling.test.ts, keep it opt-in/non-default, and dispatch the existing reusable workflow. This is the primary source-of-truth run:

gh workflow run integration.yml --ref <branch-with-probe> -f platforms=linux-arm64,linux-x86 -f test-filter=issue69-settling.test.ts -f run-examples=false -f routeros-targets=stable -f collect-metrics=true

2. To distinguish arm64-specific from TCG-specific, run the same probe on the x86 TCG platforms. If x86 TCG reproduces the first-new-credential reset, that points away from the RouterOS arm64 build and toward generic TCG/network/service timing:

gh workflow run integration.yml --ref <branch-with-probe> -f platforms=macos-x86,windows-x86 -f test-filter=issue69-settling.test.ts -f run-examples=false -f routeros-targets=stable -f collect-metrics=true

If there is no probe branch yet, the existing focused substitute is:

gh workflow run integration.yml --ref main -f platforms=macos-x86,windows-x86 -f test-filter=provisioning.test.ts,start-stop.test.ts -f run-examples=false -f routeros-targets=stable -f collect-metrics=true

3. sweep.yml has no listed run history at the moment. Triggering it once is useful as an all-platform smoke baseline, but it is not a direct tests: intermittent ECONNRESET on first REST probe with freshly-provisioned credentials (arm64-biased) #69 reproduction because the sweep uses tcg-smoke: true, so TCG integration legs are narrowed to anchor.test.ts. Use manual integration.yml dispatches above for issue signal.

4. If we need packet captures in CI, use a short-lived issue69-lab.yml or add a narrow lab mode that installs tcpdump, runs only the probe, and uploads: probe NDJSON, pcap, integration-output.txt, boot-log.ndjson, qemu logs, and any machine JSON. Keep it clearly non-gating and remove or generalize it after the investigation.

Decision criteria before changing product or test behavior:

• All forwarded TCP services flap together: investigate QEMU/SLiRP/guest NIC/runner accel first.
• Only HTTP/REST resets while API/SSH/WinBox stay stable: likely RouterOS www or auth reload behavior after user changes.
• Admin REST stays good but new-user REST flips from reset/401 to 200 after a measurable delay: add a bounded “new credentials are usable” readiness check after provisioning, with the lab cited.
• Native API accepts the new user while REST resets: target www/REST only, not provisioning as a whole.
• Native API and REST both reject/reset only the new user while admin works: user/auth propagation window.
• x86 TCG reproduces the same signature: treat TCG timing/network-service behavior as the leading hypothesis.
• x86 TCG stays clean but linux-arm64 TCG reproduces: look harder at the RouterOS arm64 guest build, linux-arm64 runner networking, or arm64 QEMU device path.

Only after one of those buckets is proven would I add retry/wait logic. If it is needed, it should be a narrow post-user-creation readiness check or first-probe helper with connection-class retry only, not a broad timeout bump or a test skip.

[mobileskyfi]

Implementation update from the investigation plan:

• Added a focused probe: test/integration/issue69-settling.probe.ts. It intentionally does not match *.test.ts, so the normal integration suite will not pick it up. integration.yml now invokes filtered files as paths, so this probe can still be run by test-filter.
• The probe follows the minimal path we identified: fresh CHR with secureLogin:false, then createUser() directly, then the first request using the newly-created credentials. No disableAdmin, no pre-probe console work that would perturb timing.
• It writes issue69-*.ndjson with platform metadata, target arch/accel/cross-arch, CPU/memory, QEMU versions, QuickCHR machine state, 250ms TCP probes for http/api/ssh/winbox, REST attempt details, and an immediate second attempt when attempt 1 fails with ECONNRESET.
• After the REST probes, it collects serial-console diagnostics (/system/resource, /ip/service, /user, /interface, /file, /log) plus the machine.json and qemu.log tail into the same NDJSON before cleanup.
• Added a general platform-info.txt integration artifact for every leg: matrix id, runner/process arch, CPU model/count, memory, QEMU versions, /dev/kvm state, quickchr accel detection, and a Linux QEMU KVM open probe. That should help future TCG-vs-KVM/arch/memory failures too, not just this issue.

Suggested next CI runs after this branch is pushed:

gh workflow run integration.yml --ref <branch> -f platforms=linux-arm64 -f test-filter=issue69-settling.probe.ts -f run-examples=false -f routeros-targets=stable

gh workflow run integration.yml --ref <branch> -f platforms=linux-x86 -f test-filter=issue69-settling.probe.ts -f run-examples=false -f routeros-targets=stable

gh workflow run integration.yml --ref <branch> -f platforms=macos-x86,windows-x86 -f test-filter=issue69-settling.probe.ts -f run-examples=false -f routeros-targets=stable

How I would read those:

• linux-arm64 is the source-of-truth repro path because it is the hosted same-arch arm64 environment where the failures keep appearing.
• linux-x86 gives the KVM control.
• macos-x86/windows-x86 are cheap x86 TCG contrast points. If they show the same first-new-credential reset, this becomes more likely generic TCG/network timing. If they stay clean while linux-arm64 reproduces, that keeps arm64 CHR guest behavior or hosted arm64 networking on the table.

Local validation done on the Intel Mac was only a harness check, not evidence against #69:

QUICKCHR_DATA_DIR=/tmp/quickchr-issue69-data HOME=/tmp/quickchr-issue69-home QUICKCHR_TEST_TARGET=7.23.1 QUICKCHR_INTEGRATION=1 QUICKCHR_ISSUE69_ITERATIONS=1 QUICKCHR_ISSUE69_ARCH=x86 QUICKCHR_ISSUE69_OUT=/tmp/quickchr-issue69-local.ndjson bun test ./test/integration/issue69-settling.probe.ts

That passed one x86/HVF iteration: boot ~21s, createUser() ~2s, first new-user /system/resource 200, new-user /user 200, admin /system/resource 200, after-5s new-user /system/resource 200, no observed problems. Again, this validates the probe path and artifact capture only; it does not rule out the arm64/TCG CI failure.

Also ran bun run check successfully after the workflow/docs/probe changes.

[mobileskyfi]

Additional probe evidence from the 2026-07-05/06 dispatches:

• Stable probe run https://github.com/tikoci/quickchr/actions/runs/28758157337 covered all five platforms (linux-arm64, linux-x86, macos-arm64, macos-x86, windows-x86): 5 legs × 8 iterations = 40 iterations, with 0 observed problems, 0 REST probe failures, and 0 TCP probe non-connects.
• Long-term/testing probe run https://github.com/tikoci/quickchr/actions/runs/28758125602 covered all five platforms across both targets: 10 legs × 8 iterations = 80 iterations, again with 0 observed problems, 0 REST probe failures, and 0 TCP probe non-connects.
• The user-noted failed run https://github.com/tikoci/quickchr/actions/runs/28758858863 was not an issue69-probe failure. Its concrete recorded failure is now tracked separately as CI: linux-arm64 long-term clean() reboot times out under TCG #79 (linux/aarch64 · long-term post-clean() boot timeout); the macos-x86 full-suite no-log failures continue under CI: macos-x86 (TCG) full suite hangs silently mid provisioning.test.ts #76.

So the current #69-specific signal is still negative: the isolated first-new-credential REST path has not reproduced across stable/long-term/testing or across KVM/HVF/TCG legs in these dispatches. That does not close the historical intermittent report, but it shifts the next investigation away from “generic arm64/TCG first-new-user REST reset” and toward either the normal full-suite path or separate boot/lifecycle failures.