Skip to content

Flaky in the acceptance lane: tps auth > revoke deletes credential file passed and failed on the same commit #358

Closed

Description

@tps-flint

tps auth > revoke deletes credential file is flaky in the Docker Integration lane, and the evidence is unusually clean.

The evidence

Two workflow runs on the same commit (35741977, cli#352), started in the same minute:

run conclusion
35114289250 success
35114288648 failure

Same SHA, same workflow, opposite outcomes. That is flakiness by definition — no comparison against main required, though for completeness every recent Docker Integration run on main is green.

The failing assertion:

1 tests failed:
(fail) tps auth > revoke deletes credential file [0.52ms]
 1 fail
error: script "test" exited with code 1

0.52ms suggests it is failing fast on a filesystem state check rather than timing out.

Why this matters more than one annoying red

Docker Integration is the acceptance lane for the sandbox and office work in cli#341. A lane that fails at random for reasons unrelated to the change teaches everyone to re-run it until it is green, and once that habit exists the lane has stopped gating anything. That is the same failure shape as a scanner with false positives or a merge gate that fires on an unmergeable PR: a control that cries wolf gets worked around, and then it is not there for the real failure.

It also directly undermines fails-first discipline. If a red can mean "the code is wrong" or "roll again", then a red proves nothing and a green proves less.

What to fix

Find the shared state the test depends on. A revoke-deletes-file test failing in under a millisecond points at either a path that another test in the same run also writes or removes, or an assertion that the file is gone running before the delete has been observed. Isolate the credential path per test, and make the assertion wait for the observable condition rather than assuming ordering.

Acceptance: run the test in a loop (100 iterations, in the Docker lane) and have it pass every time. A single green run is exactly what we already have and it is not evidence.

Activity

  1. moved this to Triage in Office queueon Oct 1, 2026
  2. tps-flint commented on Oct 1, 2026

    @tps-flint
    ContributorAuthor

    Fixed in #432 (b6f529a): the flake came from auth.ts building ~/.tps/auth at import time, so two test imports in the same millisecond shared one module and the second revoke used the first test's HOME. authDir() now resolves HOME on every call (packages/cli/src/commands/auth.ts:27), and packages/cli/test/auth.test.ts:134 reproduces the collision deterministically instead of relying on timing.

    (Daily triage, verified on main.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions