fix: latency chunkSize replaySpeed logLevel in startFromConfig#294
Merged
jpr5 merged 4 commits intoJul 14, 2026
Conversation
… startFromConfig
startFromConfig built its own subset of MockServerOptions, so latency,
chunkSize, replaySpeed and logLevel set in aimock.json were silently
dropped. The aimock-cli config reference lists latency, chunkSize and
logLevel under llm — its example config sets them — and all four work via
the llmock CLI flags and createMockSuite({ llm }). AimockConfig.llm was
the only llm-keyed surface that lost them.
replaySpeed was unreachable from a config file entirely, which leaves
fixture suites unable to opt out of replaying recorded inter-chunk timings
at their original wall-clock pace.
A non-positive llm.replaySpeed is ignored with a warning, mirroring the
fixture-level guard in fixture-loader.ts: calculateDelay ends with
`speed > 0 ? delayMs / speed : delayMs`, so 0 — the value that reads like
"no delay" — would apply the full recorded delay instead.
Contributor
Author
|
@jpr5 could you please allow the workflow run? |
commit: |
…d test from Logger internals Add a startFromConfig test proving llm.logLevel reaches the running server: a matched request emits a "Fixture matched" debug line only when the level is high enough, so "debug" logs it and "silent" suppresses it — a direct probe of the passthrough rather than a construction check. Rework the non-positive replaySpeed guard test to spy on Logger.prototype.warn (the logger the code path actually uses) instead of console.warn, dropping the dependency on the "[aimock]" prefix and the console transport.
…reaming behavior
The streamingTasks fixture used an A2A-protocol event shape
({ kind: "status-update", taskId, status }) that does not match this
library's A2AStreamEvent union, which is discriminated by `type`
("status" | "artifact"). The runtime (a2a-mock.ts SSE loop, a2a-handler.ts)
reads `event.type`/`event.state`/`event.parts`, never `kind`/`taskId`/`status`,
so the literal was simply wrong test data (TS2353, invisible to build/CI
because tsconfig excludes src/__tests__).
Correct the fixture to the valid discriminated shape and extend the test to
actually POST SendStreamingMessage and assert the streamed status + artifact
events, so it meaningfully exercises streaming rather than only checking the
agent card.
jpr5
approved these changes
Jul 14, 2026
jpr5
left a comment
jpr5
left a comment
Contributor
There was a problem hiding this comment.
Reviewed and verified end-to-end. The four llm passthrough fields (latency, chunkSize, replaySpeed, logLevel) reach LLMock correctly and the non-positive replaySpeed guard is sound. CI is green across the full Node x Python matrix.
I pushed two test commits on top: one covering logLevel passthrough and decoupling the guard test from Logger internals, and one fixing a pre-existing invalid A2AStreamEvent fixture — the runtime discriminates on type and never reads kind/taskId/status, so the fixture was mis-shaped — plus strengthening it to a real SSE round-trip assertion. Merging. Thanks @jantimon.
jpr5
approved these changes
Jul 14, 2026
jpr5
left a comment
jpr5
left a comment
Contributor
There was a problem hiding this comment.
Verified end-to-end; CI green. Merging.
Merged
jpr5
added a commit
that referenced
this pull request
Jul 14, 2026
## Release v1.36.1 (patch) Cuts a patch release for the `llm` config passthrough fix merged in #294. ### Version bump `package.json` `1.36.0` → `1.36.1`. (Matches this repo's practiced release convention: only `package.json` + `CHANGELOG.md` are version-carrying for aimock — `charts/aimock/Chart.yaml`, `.claude-plugin/*`, and `src/cli.ts` are not bumped per release here, as v1.36.0 also demonstrated, and no CI parity check enforces them.) ### CHANGELOG Rolled the existing `[Unreleased]` entry into `## [1.36.1] - 2026-07-14`: > ### Fixed > - `aimock.json` now honours `llm.latency`, `llm.chunkSize`, `llm.replaySpeed` and `llm.logLevel`, which the config loader previously accepted and ignored. Each behaves as its `llmock` CLI flag and `createMockSuite({ llm })` equivalent. `llm.replaySpeed` divides every delay source (recorded timings, streaming profiles, `latency`), so a fixture suite can replay recorded inter-chunk timings faster than they were recorded; a fixture's own `replaySpeed` still takes precedence. A non-positive `llm.replaySpeed` is ignored with a warning, matching the existing fixture-level guard (#294) ### Docs completeness audit Checked every place the `llm` config is described: | Location | Before | Action | |---|---|---| | `docs/aimock-cli/index.html` — Config Fields prose row | all four fields listed (fixed in #294) | already complete | | `docs/aimock-cli/index.html` — `aimock.json` JSON example | showed `latency`, `chunkSize`, `logLevel`; **missing `replaySpeed`** | **added `"replaySpeed": 1`** so the example matches the documented field set | | `src/config-loader.ts` — `AimockConfig.llm` type | all four typed; no per-field JSDoc (existing style) | no change (would not force JSDoc where none exists) | | `docs/index.html` landing example | illustrative marketing snippet (`providers` array, stale banner), not a field reference | no change | | `docs/chat-completions/index.html`, `packages/aimock-pytest/README.md`, `README.md` | partial illustrative examples, not a full `llm` field reference | no change | Net: one real gap fixed (JSON example ↔ prose consistency on the canonical config-reference page); everywhere else was already complete or is deliberately illustrative. ### Quality gate - `pnpm run format:check` — pass (changed files clean; only a gitignored `.impeccable/hook.cache.json` local artifact warns) - `pnpm run lint` — pass - `pnpm test` — 4364 passed (147 files) - `pnpm run build` — pass Draft until you're ready to merge; merging to `main` triggers `publish-release.yml` (npm publish + tag + GitHub Release + Slack).
Contributor
|
Thank you for the contribution @jantimon! 🙏 |
pull Bot
pushed a commit
to TheTechOddBug/aimock
that referenced
this pull request
Jul 14, 2026
### Fixed
- aimock.json now honours llm.latency, llm.chunkSize, llm.replaySpeed and
llm.logLevel, which the config loader previously accepted and ignored. Each
behaves as its llmock CLI flag and createMockSuite({ llm }) equivalent.
A non-positive llm.replaySpeed is ignored with a warning, matching the
existing fixture-level guard (CopilotKit#294)
Contributor
Author
|
thanks for the fast merge and release |
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
I ran into this bug while trying to set `replaySpeed` to speed up an e2e suite which uses aimock The aimock-cli docs list latency/chunkSize/logLevel under `llm` and the example config sets them. But copying that example does not work Our fixtures have recordedTimings so every replay artificially slows down the e2e tests by over ~70s across the suite `relaySpeed` already exists in MockServerOptions as an llmock flag and as a fixture field but not in a config file... I ran it against our real suite and set `llm.replaySpeed: 10000` and successfully reduced a single test duration from `11.1s` to `0.9s` **Disclaimer**: This code was written with Claude Code but I reviewed and tested it manually
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
### Fixed
- aimock.json now honours llm.latency, llm.chunkSize, llm.replaySpeed and
llm.logLevel, which the config loader previously accepted and ignored. Each
behaves as its llmock CLI flag and createMockSuite({ llm }) equivalent.
A non-positive llm.replaySpeed is ignored with a warning, matching the
existing fixture-level guard (#294)
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
## Release v1.36.1 (patch) Cuts a patch release for the `llm` config passthrough fix merged in #294. ### Version bump `package.json` `1.36.0` → `1.36.1`. (Matches this repo's practiced release convention: only `package.json` + `CHANGELOG.md` are version-carrying for aimock — `charts/aimock/Chart.yaml`, `.claude-plugin/*`, and `src/cli.ts` are not bumped per release here, as v1.36.0 also demonstrated, and no CI parity check enforces them.) ### CHANGELOG Rolled the existing `[Unreleased]` entry into `## [1.36.1] - 2026-07-14`: > ### Fixed > - `aimock.json` now honours `llm.latency`, `llm.chunkSize`, `llm.replaySpeed` and `llm.logLevel`, which the config loader previously accepted and ignored. Each behaves as its `llmock` CLI flag and `createMockSuite({ llm })` equivalent. `llm.replaySpeed` divides every delay source (recorded timings, streaming profiles, `latency`), so a fixture suite can replay recorded inter-chunk timings faster than they were recorded; a fixture's own `replaySpeed` still takes precedence. A non-positive `llm.replaySpeed` is ignored with a warning, matching the existing fixture-level guard (#294) ### Docs completeness audit Checked every place the `llm` config is described: | Location | Before | Action | |---|---|---| | `docs/aimock-cli/index.html` — Config Fields prose row | all four fields listed (fixed in #294) | already complete | | `docs/aimock-cli/index.html` — `aimock.json` JSON example | showed `latency`, `chunkSize`, `logLevel`; **missing `replaySpeed`** | **added `"replaySpeed": 1`** so the example matches the documented field set | | `src/config-loader.ts` — `AimockConfig.llm` type | all four typed; no per-field JSDoc (existing style) | no change (would not force JSDoc where none exists) | | `docs/index.html` landing example | illustrative marketing snippet (`providers` array, stale banner), not a field reference | no change | | `docs/chat-completions/index.html`, `packages/aimock-pytest/README.md`, `README.md` | partial illustrative examples, not a full `llm` field reference | no change | Net: one real gap fixed (JSON example ↔ prose consistency on the canonical config-reference page); everywhere else was already complete or is deliberately illustrative. ### Quality gate - `pnpm run format:check` — pass (changed files clean; only a gitignored `.impeccable/hook.cache.json` local artifact warns) - `pnpm run lint` — pass - `pnpm test` — 4364 passed (147 files) - `pnpm run build` — pass Draft until you're ready to merge; merging to `main` triggers `publish-release.yml` (npm publish + tag + GitHub Release + Slack).
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
Implements the reviewed **aimock Drift-Detection Ironclad** hardening spec: https://app.notion.com/p/39d3aa38185281c582b1d819c2842518 Built via 4 gated waves (17 micro-tasks), each red-green against the real failure surface, per-wave reviewed to zero, plus a final full-branch integration CR. 21 files, ~+2238/-288. 224 drift tests green. ## What changed, by wave - **Wave A — collector taxonomy + primitive.** Pure `computeExitCode` (crit>0→2, quarantine>0→5, agUiSkipped→1, else 0); `collectDriftEntries` now *quarantines* unmapped/unparseable failures (capturing file:line) instead of throwing — the exit-1 crash that hid incident #294's root cause is gone. Report schema gains `DriftClass`/per-item `id`/`class`/`quarantine[]`. Shared `normalizeModelFamily(id, provider)` primitive (voice-models migrated onto it, behavior-identical). - **Wave B — exit-5 threading + W1 /models rewrite.** `drift-retry` + both workflows thread exit-5 (quarantine → run fails, no auto-fix abort). **Model identity no longer scrapes prose/source** (the incident root cause): enumerate live `/models` → normalize → subtract curated include/exclude → unclassified delta is the signal, with an LLM-triage seam. `models-scrape.drift.ts` retired. - **Wave C — classified alerting.** `InfraError` carries a structured HTTP `status`; Slack summary emits a closed `class` enum (real-drift / false-positive / infra-transient / stale-key / quarantine) with per-item id + file:line; key-freshness preflight fails loud on 401/403 (never a green skip); notify routes exit-5 → quarantine before the HTTP-drift fallback. - **Wave D — delta-gated PR check.** New `drift-live-pr` job (`pull_request`, path-filtered, no secrets on forks) runs base+head and **blocks only on failures the PR introduces (new-in-head), regardless of class** — a drift-code PR is judged on "did your change break the detector," not "is the world drifting today." Base-present failures are advisory. Fork PRs use a maintainer dry-run. ## Bugs the review process caught (not shipped) 1. **Anthropic `-YYYYMMDD` normalize gap** — dated Claude snapshots wouldn't collapse to their family → would false-positive (incident-2 class, untested provider). Fixed via the provider hook. 2. **Class-gating hole** — a first cut gated on collector-assigned class; a new-in-head `real-drift` would've been advisory → **would have re-greenlit the exact #292 incident**. Fixed to gate purely on new-in-head. 3. **NUL-byte** in `drift-delta.ts` rendered the module binary/unreviewable in diffs — caught only by the final full-branch CR. ## Follow-up before merge - **Keyed CI run required:** the live `/models` provider blocks were skipped in the impl env (no keys). A keyed CI run is the remaining real-surface validation. **DRAFT — awaiting review before merge to main.**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I ran into this bug while trying to set
replaySpeedto speed up an e2e suite which uses aimockThe aimock-cli docs list latency/chunkSize/logLevel under
llmand the example config sets them.But copying that example does not work
Our fixtures have recordedTimings so every replay artificially slows down the e2e tests by over ~70s across the suite
relaySpeedalready exists in MockServerOptions as an llmock flag and as a fixture field but not in a config file...I ran it against our real suite and set
llm.replaySpeed: 10000and successfully reduced a single test duration from11.1sto0.9sDisclaimer: This code was written with Claude Code but I reviewed and tested it manually