Skip to content

CI: per-file runtime caps derived from ci-data timing (hang → fast honest fail) #77

Description

@mobileskyfi

User suggestion from the first full-TCG runs (#76 context): a hung test file currently burns the whole job timeout (300 min on full TCG legs) in silence. The sequential per-file loop in integration.yml should cap each file's runtime so a hang becomes a fast, attributed failure — while staying data-driven, not guess-driven.

Sketch

  • We already record per-file wall-clock per platform × target in ci-data (runs/*.ndjson, kind:test-file). That gives an empirical baseline per (file, platform).
  • Wrap each file in the loop with a cap: cap = clamp(factor × observed_max(file, platform), floor, job_remaining) — factor ~3×, floor ~10 min (TCG floor higher, e.g. 20 min), so normal variance never trips it.
  • Simplest v1: a static per-platform cap baked into the plan matrix (e.g. 15 min accelerated / 45 min TCG per file), revisited against ci-data once more full-TCG runs exist. Data-driven lookup (fetch tested timings in the plan job) is the v2 if static proves too blunt.
  • On cap hit: kill the bun process and reap QEMU orphans immediately (today the reap step only runs after the whole loop — a starved runner can't even reach it, see CI: macos-x86 (TCG) full suite hangs silently mid provisioning.test.ts #76), write <file> <cap>s timeout to the timing file, continue to the next file. Red job at the end, but every remaining file still reports.
  • timeout status flows into metrics.ndjson so ci-data distinguishes hang from fail.

Not aggressive by design: the cap exists to convert silent starvation into signal, not to police slow-but-progressing suites (the 300-min job timeout stays the outer bound).

🤖 Filed from user direction by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions