id: design: per-interface benchmark matrix (from #5685) #5896

Activity

1. dezren39

   added

   app/idPackage/app: id (pkgs/ or apps/ in this repo)Package/app: id (pkgs/ or apps/ in this repo)

   on Nov 5, 2026
2. developing-today commented on Nov 5, 2026

   

   developing-todayBot

   on Nov 5, 2026 – with developing.today

   Contributor

   More actions

   CODE-1579
3. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   task/triageQueue packet kind: verify, reproduce, and route this itemQueue packet kind: verify, reproduce, and route this item

   on Nov 5, 2026
4. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   Triage outcome — task 10298 (2026-11-05, lane code-p)

   Verdict: CONFIRMED as a valid standalone design ticket — routed, with axis-dependency guidance recorded (no parent-recorded ordering exists for this family yet).

   Fresh verification (2026-11-05): issue #5896 OPEN, labels task/triage + queue/queued + app/id — no hard-stop labels. Parent #5685 (DoS suite & performance benchmark programme) is OPEN; the WS-F gap pass filed all seven children — #5891 (shared load & workload generator), #5892 (DoS scenario catalogue), #5893 (CI tiering), #5894 (standard server baseline), #5895 (scaling-technique comparisons), #5896 (this ticket), #5897 (benchmark regression gates) — all OPEN. Unlike the #5675/#5679 families, the parent has not yet been triage-ordered, so the dependency guidance below is this triage's recommendation, flagged as such.

   Existing assets the design must build on (current main 04f78696ab): pkgs/id/benches/ holds the world_bench harness, committed baseline.txt medians, and BASELINES.md (reference machine = the fleet laptop; re-baseline procedure; slice-3 CI regression reporting diffs future runs against medians at >20%). There is no per-interface matrix, results format, or publication mechanism yet — that absence is exactly this ticket. The matrix results format should stay comparable with the existing baseline.txt line format where that makes sense, rather than orphaning the committed baselines.

   Dependency guidance (recommendation, not a parent ruling):

   • The matrix's axes are workloads × interfaces × configurations. The workload axis should reference id: design: shared load & workload generator (from #5685) #5891's generator/workload definitions and the configuration axis id: design: the standard server baseline (from #5685) #5894's standard server baseline — designing axis values in isolation here would fork both. The design can proceed on the matrix structure + results format + publication now, treating id: design: shared load & workload generator (from #5685) #5891/id: design: the standard server baseline (from #5685) #5894 outputs as named inputs; if those tickets settle after this one, only the axis enumerations should need revisiting.
   • id: design: benchmark regression gates (from #5685) #5897 (regression gates) consumes this ticket's results format — the format is a contract; state it as one (fields, units, machine-class tagging per the BASELINES.md comparability warning).
   • Boundary: the DoS scenario catalogue is id: design: DoS scenario catalogue per interface (from #5685) #5892 and CI tiering is id: design: CI tiering for adversarial & benchmark suites (from #5685) #5893 — this ticket is the benchmark matrix only, not the adversarial suite's placement in CI.

   Deliverable (ticket's own terms): a recorded decision — matrix shape, results format, publication — with alternatives rejected and implementation invariants, on this issue or in a doc/ record linked from it. Implementation timing is a separate later decision.

   Routing: follow-up design task enqueued with the axis-input guidance in its packet.

   agent-note:v1
   {"kind": "triage", "task": 10298, "issue": 5896, "state": "done", "outcome": "confirmed_design_ticket_routed", "evidence": "parent #5685 OPEN, children #5891-#5897 all OPEN, parent not yet triage-ordered; pkgs/id/benches has world_bench + committed baselines but no per-interface matrix or results format"}
5. dezren39

   added

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 5, 2026
6. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   agent-note:v1
   {"kind":"worker","task":10503,"issue":5896,"state":"picked_up","outcome":"work_started","lane":"c8b9355e-319a-4640-8096-7ad19480d151","evidence":"Queue task 10503 claimed by repo-worker-code; issue #5896 OPEN with no linked PR and no hard-stop labels; parent #5685 OPEN. Revalidated 2026-11-05."}
7. dezren39

   removed

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   on Nov 5, 2026
8. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   Decision record — per-interface benchmark matrix, results and publication (2026-11-05)

   Decision

   The comparison unit is a matrix cell keyed by workload × interface × configuration. Workload IDs come from #5891 and configuration IDs from #5894; this record deliberately does not define either sibling's enumerations. The interface axis uses stable IDs for the actual protocol/transport being measured, not the presentation format. The existing shell contract names iroh, websocket and ssh-exec as distinct transport IDs; WorldSurface::Screen and WorldSurface::Text are presentation kinds, not additional transports. Add other interfaces only when an implementation exposes a separately measurable protocol surface, with a stable ID.

   For each cell, use the same workload definition and shared load/measurement machinery across interfaces. A workload/configuration that does not apply to an interface is recorded explicitly as not_applicable or unsupported with a reason; it is never silently omitted or represented as zero. A failed run is also explicit and is distinct from a measured regression.

   Results contract for #5897

   Publish a versioned results.jsonl bundle with a run-manifest record and one cell record per matrix tuple. The manifest carries schema_version, run_id, git_sha, started_at_utc, machine_class, OS/architecture, CPU/runner description, toolchain and build profile. Each cell carries run_id, workload_id, interface_id, configuration_id, status, and reason when not measured. A measured cell contains metric entries with metric, statistic, unit, numeric value, sample_count and raw sample values. The comparison identity is (machine_class, workload_id, interface_id, configuration_id, metric, statistic); each run also has a distinct run_id.

   Use explicit units and statistics, e.g. throughput / median / operations_per_second; latency / p50, p95 and p99 / microseconds; CPU per operation / median / microseconds_per_operation. The machine_class is mandatory and is the canonical class supplied by #5894—not a free-form runner label. Include the commit and environment provenance so a consumer can reject incomparable results. Do not compare across machine classes, configuration IDs, units or workload/interface IDs.

   Keep the existing bench <name> unit=<unit> median=<value> min=<value> reps=<n> output and baseline.txt representation for compatible world_bench medians. Treat it as a compatibility/human-readable view, not a second source of truth: p50/p95/p99 and other matrix metrics stay explicit in JSONL, and the legacy median field must not be used to disguise another statistic. #5897's comparator must key on the full matrix identity rather than only the current second-token benchmark name (the existing comparator keys by name and compares unit/median, so matrix rows can otherwise collide).

   Publication

   For every matrix workflow run, upload the JSONL bundle and raw benchmark output as a run artifact, and link that artifact from the GitHub Actions job summary. The summary is an index, not the record. Keep accepted reference baselines versioned in the repository and update them only through the documented BASELINES.md re-baseline procedure and review. #5897 owns the durable artifact-retention/archive period and the comparison/gate policy; #5893 owns CI tiering and triggers. Current precedent is the job summary plus uploaded artifacts in id-bench.yml; its workflow_dispatch-only trigger differs from the push behavior described in BASELINES.md, so that discrepancy is left to #5893 rather than changed here.

   Rejected alternatives

   • Encoding interface/configuration into an informal display name alone: the current comparator's name-only key is collision-prone and does not establish machine-class comparability.
   • A prose-only baseline or dashboard-only publication: neither gives id: design: benchmark regression gates (from #5685) #5897 a versioned machine-readable contract and raw provenance.
   • Separate workload generators per interface: duplicates the shared machinery fixed by design(id): DoS suite & performance benchmark programme — adversarial CI coverage and per-interface regression gates #5685 and risks comparing different loads.
   • Treating Screen/Text presentation kinds as transport IDs: they are a different layer from iroh/websocket/ssh-exec.

   Implementation invariants and boundaries

   1. Preserve design(id): DoS suite & performance benchmark programme — adversarial CI coverage and per-interface regression gates #5685's fixed programme: per-interface benchmarks remain regression gates and share load/measurement machinery with the adversarial suite. This decision does not redesign that suite.
   2. Reuse id: design: shared load & workload generator (from #5685) #5891 workload definitions and id: design: the standard server baseline (from #5685) #5894 configurations; compare interfaces only with identical workload inputs and configuration on the same machine class.
   3. Every planned cell produces measurements or an explicit status/reason. Preserve raw samples, units, machine class and build/environment provenance; never silently mix units or machine classes.
   4. Baseline changes remain versioned and reviewed under BASELINES.md; a changed baseline is not silently accepted from a quick run.
   5. Regression budgets, waivers, blocking behavior, scenario catalogue and CI tier selection remain with id: design: benchmark regression gates (from #5685) #5897, id: design: DoS scenario catalogue per interface (from #5685) #5892 and id: design: CI tiering for adversarial & benchmark suites (from #5685) #5893 respectively.

   Evidence checked on main 04f78696ab: pkgs/id/benches/world_bench.rs currently has host-side microbenchmarks and many_worlds, not independent transport paths; pkgs/id/contract/shell-contract.v1.json names the shell transport IDs; pkgs/id/benches/baseline.txt and BASELINES.md establish the current line format and machine-class rule; .github/scripts/id-bench-compare.sh keys by benchmark name and compares unit/median. #5891 and #5894 are still open, so their axes remain named inputs.

   agent-note:v1
   {"kind":"worker","task":10503,"issue":5896,"state":"done","outcome":"design_record_posted","lane":"c8b9355e-319a-4640-8096-7ad19480d151","evidence":"Decision record posted here; cross-checked against main 04f7869 and parent #5685. Design-only; no code or benchmark runs."}
9. github-actions

   assigned

   dezren39

   on Nov 5, 2026