| title | Testing |
|---|---|
| description | Reproducible quality gates for contributors. |
Use Corepack and the pinned pnpm toolchain:
corepack pnpm@10.12.4 run typecheck
corepack pnpm@10.12.4 run lint
corepack pnpm@10.12.4 run test:coverage
corepack pnpm@10.12.4 run eval:bugbot
corepack pnpm@10.12.4 run build
corepack pnpm@10.12.4 run validate:agent-docs
corepack pnpm@10.12.4 run validate:docs-page
corepack pnpm@10.12.4 run validate:documentation
corepack pnpm@10.12.4 run validate:workflows
corepack pnpm@10.12.4 run validate:specifications
git diff --checkvalidate:documentation also enforces heading fragments, the complete setup
workflow and single-action catalogs, parity between the six embedded issue
examples and their setup templates, and high-risk release contracts such as
the default-branch continuation prerequisite, npm direct-publish policy,
polling controls, and orchestration Variables. It also rejects removed public
inputs, actions, commands, and orchestration modes.
validate:specifications validates unique capability ownership, safe existing
repository-relative evidence paths, complete registration of top-level SDDs,
retrospective-baseline structure, and byte-for-byte freshness of the generated
specs/CATALOG.md view. It also enforces the communication-program test ledger:
at least 128 semantic-publication cases and 136 localization cases, allocated to
disjoint test files. Parameterized declarations count once, and removing tests,
inflating an allocation, or assigning one file to both budgets fails the gate.
The ledger lives at
src/architecture/github_communication_test_budget.json. Its qualifying counts
are a conservative audit of new Jest it/test declarations since the two SDDs
landed at baseline 0bb83b75^; a table-driven declaration is not multiplied by
its data rows.
Merge-queue contract tests cover ruleset-only targets, classic/ruleset union, GitHub Actions workflow parsing on target and candidate refs, external exact attestations, setup/doctor drift, pre-PR and pre-enqueue ordering, queue membership idempotency, and head-bound enqueue. These tests use provider fakes; they never create a live queue entry or poll check runs.
These deterministic checks cannot inspect live GitHub/npm settings or judge desktop/mobile, light/dark, and assistive-technology rendering; keep those as explicit operator and human UX gates.
Tests MUST isolate environment variables such as AGENT_PROVIDER, AGENT_MODEL_PROVIDER, AGENT_MODEL, and AGENT_EXECUTABLE. Real provider smoke tests are separate from deterministic unit tests and MUST NOT run without authorization and cost controls. Coverage thresholds are enforced by Jest and must not be reduced to accommodate a new code path; add focused tests instead.
eval:bugbot runs the offline Bugbot quality harness. Corpus cases score precision, recall, F1, exact-location, severity, category, line distance, false positives/negatives, and confidence calibration independently, so prompt or model changes cannot hide false positives behind a single pass/fail number. copilot bugbot-benchmark can generate predictions with the real configured agent for a cost-controlled manual or scheduled gate; copilot bugbot-eval scores them. Ground truth is not passed to the agent. Extend the corpus when a production review exposes a new failure mode.
copilot bugbot-analytics aggregates content-free telemetry exported by the Action. Do not add prompt, diff, source, comment, response, credential, or rule content to the telemetry schema.
Workflow queue tests inject a clock, random source, and scheduler/delay port. They
cover workflow-scoped active-status traversals and later-page matches, strict
lower-ID/workflow-file filtering, malformed responses, fail-closed provider errors, 429 and
rate-limited 403 headers, bounded fallback backoff/jitter, and the absolute deadline.
The workflow contract test keeps the nine queue-bearing workflow names synchronized with the
validator and verifies the 90m queue / 120m job budget, required runners, action
inputs, missing/short timeouts, forbidden durable-mutation concurrency, and the
distinct branch concurrency and event-ownership contracts for Commit and Pull
Request jobs. The Bugbot ownership policy matrix and push-shaped integration
test additionally prove that exact-head provider discovery—not a nonexistent
pull_request field on the push payload—stops duplicate agent review before
the diff and conversation are loaded.
GitHub lifecycle tests cover admission before queue polling, PAT-owner normal-run discard, valid single-action preservation, identity lookup failure, and the absence of project composition, agent provisioning, setup, publication, and persistence on discarded runs.
The lifecycle replay integration suite feeds normalized, serialized-shaped
pull_request_review payloads through the real lifecycle synchronization use
case. It verifies deterministic labels for approved reviews, requested changes,
and ambiguous multi-PR events without invoking an agent route. External check
workflows are intentionally outside the workflows shipped by Copilot.
Configuration tests cover current-schema round trips and fail-closed handling of missing, malformed, or different-version durable values. Architecture tests also guard the narrow context contracts used by extracted policies and lifecycle reconciliation.
Branch synchronization tests cover durable configuration and linked-branch discovery, pagination, parent/child comparisons, bot-owned sticky comments, explicit and natural-language commands, authorization, dry runs, clean merges, conflict eligibility, agent postflight, credential-free verification, unexpected workspace mutations, and remote-head races. The all-branch observer workflow is also contract-checked to remain free of agent inputs.
Architecture tests additionally verify that production imports remain acyclic,
application code does not depend on concrete adapters, and pure model/policy
code remains free of runtime, provider, and logging dependencies. The
product-facing agent inventory test also requires exactly one guarded query for
Think, help, recommendations, progress, PR descriptions, and Bugbot, and prompt
tests require both targetLocale and outputLocale. Workflow tests prove that
a mismatched response cannot publish or mutate labels, descriptions,
recommendation state, or findings. Refresh the
local topology before an architecture review with Graphify, and use the
RepoWise index-only health/dead-code reports to distinguish current complexity
from history-based churn signals:
RepoWise Code Health is an optional manual workflow. PRs, pushes and merge
groups do not launch it, and it must not be a required branch check. To run an
analysis, open Actions → RepoWise Code Health → Run workflow, select a
reviewed repository branch, and launch it. The CLI equivalent for development is
gh workflow run repowise.yml --ref develop. The workflow already exists on the
default branch, as required for manual dispatch.
Open the run's Job Summary for the analyzed commit, phase status, elapsed time
and peak RSS. Download repowise-code-health (14 days) for health Markdown/JSON,
safe-only dead-code JSON, the security JSON aggregate, index/phase logs,
revision.txt, scope.json, timings.json and history-resources.json.
The pinned RepoWise 0.58.0 builds one fresh standard index and exports its stored health/dead-code results. Index git analytics retain a 500-commit limit; security separately covers all reachable history, including removed secrets and generated bundles, using RepoWise's supported source extensions, secret patterns and allowlists. Two spawned workers read one blob at a time each, with at most two tasks/results in flight and results consumed in upstream order. Worker failures propagate. The adapter avoids computing code-smell findings discarded by history mode. This remains an advisory signal scanner; an aggregate with no findings is not proof that no secrets exist.
Indexing has a 900-second/4 GiB virtual-memory cap. Each health export and dead-code has a 60-second/2 GiB cap; security has a 1200-second/2 GiB coordinator cap and a 1 GiB cap per worker, inside the 40-minute job. Limits apply per process; reported peak RSS is the largest process, not aggregate simultaneous memory. These budgets accommodate the roughly 9 GB of historical JavaScript without retaining all of it in memory. Two available CPU cores are recommended for the bounded parallel history scan.
An interrupted scan, Git error, stale/failed index or invalid report marks analysis incomplete and fails this optional run. Completed reports and diagnostics still upload; invalid JSON is removed. Inspect the named phase log, fix the cause, then rerun. A newer manual run for the same ref cancels the older one; cancellation or runner loss can prevent publication. No manual run blocks automatic CI. Upgrading the pin requires the Python compatibility fixtures, which run before every manual analysis. They compare secret detection with upstream and verify deleted secrets, other refs, bounded reads and failures.
Manual RepoWise runs use self-hosted Ubuntu codex runners. CI Check uses
that pool for repository-owned PRs and pushes; fork PRs and all merge groups
use disposable GitHub-hosted Ubuntu runners. The automatic setup fixture matrix remains hosted on three
operating systems, with service-runner fixtures available by manual dispatch.
Runner routing is contract-checked, but actual runner availability and remote
results require evidence from the corresponding Actions runs.
graphify update .
repowise init . --no-workspace --no-prose --no-agents --no-editor-setup \
--coverage-report coverage/lcov.info
repowise dead-code --safe-only --format json --no-workspaceSecurity regression tests must cover untrusted prompt fields, control-sequence neutralization, schema rejection, local finding fingerprints, stale finding resolution, provider credential isolation, and the language capability's separate application port. A passing model smoke test does not replace these deterministic tests.
Setup tests also cover the setup/workflow PAT separation, provider validation,
per-credential remote health results, invalid-secret replacement, workflow update
approval and backups, .env exclusion, and the read-only doctor path.
The reproducible metrics:architecture collector is the full evidence
workflow; it requires a clean working tree and writes reports outside the
repository.
Architecture fixtures that resolve TypeScript symbols should share one compiler program containing production sources and isolated fixture modules. Retain the real bindings and every assertion; rebuilding the whole project for each fixture can exhaust a Jest worker during full platform verification. The raw-error boundary suite shares one real TypeScript compiler graph across the repository scan and its six fixture modules. It keeps all seven assertions and checks the real logger/sanitizer declarations. This reduces repeated compiler construction without changing workers, memory limits or platform gates.
Use a frozen pnpm lockfile when validating dependency changes. Security overrides are constrained to affected version ranges; the Handlebars, source-map-js and brace-expansion patches retain their existing major versions.
The runtime lockfile resolves the Actions SDK HTTP clients to undici 6.29.1,
which fixes GHSA-xjj6-f33q-929c.
Keep the production audit gate enabled and regenerate the bundled Action and CLI
after changing this dependency; a source-only lockfile update does not replace
the embedded HTTP client.
The legacy @istanbuljs/load-nyc-config edge uses js-yaml 4.3.2, whose load
API is compatible with that loader. This removes the old argparse/sprintf-js
chain from development tooling rather than claiming a patched sprintf-js release
exists. The Action's direct runtime YAML dependency is independent of this edge.
Verify the real loader with representative .nycrc.yaml configuration and run
the complete coverage, build and packaged CLI checks. Retain Windows, macOS and
Linux CI and every acceptance/coverage budget. Inspect the final resolved graph
and refreshed Dependabot alerts after promotion to the default branch. An audit
result applies to the current lockfile; it is not an exploitability assessment.
Codex service integration is unsupported on Windows. Keep [self-hosted, codex]
for Mac/Ubuntu and omit codex from Windows service registrations. The manual
Codex fixture matrix includes only macOS and Ubuntu. Hosted general fixture and
full-coverage jobs still include Ubuntu, Windows and macOS, including fake
launcher, ACL, process and installer tests. These doubles do not authenticate a
model or exercise the official Codex sandbox.
The workflow validator and six routing regressions prevent explicit Windows
Codex jobs/matrix entries and retain ordinary Windows coverage. A green fixture
or model-only READY response cannot certify real tool execution. See
tool smoke evidence.
setup_permission_operator_actions.test.ts uses a real temporary directory and
journal, including file writes, fsync, atomic replacement, recovery and cleanup.
Its tests and hooks have a bounded 30-second Jest budget. A rejected dispatch
alone flushes six journal versions, so the generic five-second unit-test deadline
can expire during slow filesystem operations under hosted Windows coverage.
For issue #577, adding 900 ms to each real journal flush reproduced the original five-second timeout. With the 30-second integration budget, all eight cases completed, including the two-transaction recovery case. This demonstrates the budget mismatch; historical CI logs do not identify the underlying host storage or scheduling delay. Keep real persistence and every authority/cleanup assertion. Fake timers cover Actions polling; recovery switches back to real timers before starting its next journal transaction. Other suites retain their existing budgets. Hosted Windows full coverage remains the platform verification gate.
The component suites use a real Vite/Svelte SSR renderer isolated from Jest. One renderer serves each suite; every request renders its own props and locale, including sequential English, Spanish, French and Portuguese assertions. HTML is not cached or replaced by snapshots. The helper closes the renderer after the suite and propagates startup, process and render failures. Keep these real semantic assertions when changing component content or controls.
This repository currently selects [self-hosted, codex, Linux] for Copilot
operations while the four Apple runners are unstable. Mac support remains
available. The maintainer has authorized temporarily selecting only Ubuntu for
the manually dispatched self-hosted smoke while all four Apple Intel runners
are offline. Restore its explicit macOS row when a runner with self-hosted,
codex and macOS is available. Consumer setup templates retain their existing
runner configuration.
Hosted Ubuntu, Windows and macOS fixture jobs retain their full coverage and acceptance checks. The workflow validator accepts the original supported pool and its narrower Linux selection for operational jobs, and rejects Windows service Codex routing. The smoke matrix accepts the supported macOS/Ubuntu pool and this temporary Ubuntu-only selection; Ubuntu cannot be removed.
Run Setup Platform Smoke on master with the default inputs to validate the
full hosted Ubuntu, Windows and macOS matrix and the current self-hosted Ubuntu
matrix. Every selected platform keeps package checks, full coverage, acceptance
budgets and contract validators. Native macOS self-hosted acceptance remains
pending until the operator restores the Apple Intel runners and macOS row;
hosted macOS results cannot substitute for that evidence.
Hosted Windows has a 30-minute job limit; hosted Ubuntu and macOS retain 20 minutes. This includes native setup, package checks, isolated fixtures and the full coverage suite. A timeout is missing evidence and must be corrected or rerun before merging; it never substitutes for successful verification.
upload_windows_coverage defaults to false. Coverage export is a separate
explicit opt-in, never a prerequisite for hosted fixture validation. Set
windows_startup_probe=true with coverage export disabled for the isolated
five-minute hosted PowerShell diagnostic; both full fixture matrices are then
skipped. Selecting the probe together with coverage export runs the diagnostic
and hosted fixtures/export, while self-hosted fixtures remain skipped.
Inspect the run's individual platform jobs before claiming full success. A queued, canceled or failed job is missing evidence, even if other jobs passed. Fixtures do not prove authenticated agent tool execution.