design(id): DoS suite & performance benchmark programme — adversarial CI coverage and per-interface regression gates #5685

Summary

Two programmes in one design, because they share infrastructure and each is half as useful without the other:

1. A denial-of-service test suite in CI — adversarial load and abuse coverage that runs continuously, extending id(world): F12.18 denial-of-service test suite in CI #560 (F12.18 DoS test suite in CI) from a test-suite idea into a full adversarial programme.
2. A performance benchmark programme — repeatable, comparable benchmarks of a standard server and of advanced scaling techniques, per interface, run as regression gates, not one-offs.

Relates to: #560.

Why both, why together

• A system can pass benchmarks and still fall over under adversarial traffic; it can survive abuse and still be unusably slow for honest users. The load-generation, measurement, and environment machinery is largely the same — designing them separately duplicates it and guarantees the two suites drift into incomparable numbers.
• Benchmarks without gates decay into dashboard decoration. DoS tests without baselines cannot distinguish "attack worked" from "server was already slow." Each programme supplies what the other lacks.

Part 1 — DoS / adversarial load suite (extends #560)

Threat and load classes to cover

• Connection-level abuse: slow clients (slowloris-style), connection floods, half-open churn, oversized / never-ending requests, rapid reconnect storms.
• Request-level abuse: expensive-query amplification, large payload / blob abuse, decompression and parsing bombs within protocol limits, request floods against the costliest endpoints (identify them — do not guess).
• Protocol / interface abuse: per transport surface (each protocol id speaks): malformed frames, replay floods, capability-check bypass attempts via volume, federation / bridge traffic floods from a single peer.
• State-exhaustion: storage growth attacks (journal / blob / index bloat), memory pressure via concurrent worlds / sessions, descriptor and task exhaustion equivalents in the server's runtime.
• Fairness attacks: one tenant / world / identity starving others; noisy-neighbour scenarios in multi-world hosting.
• Legitimate-surge control cases: flash-crowd patterns that are NOT attacks — the suite must show the server degrades gracefully rather than treating all load as hostile (rate-limit false-positive measurement).

Suite design requirements

• In CI, but tiered: a fast adversarial smoke tier on every relevant change, a heavier suite on a schedule / merge queue, and a soak tier run less frequently — with explicit budgets so CI time does not explode.
• Deterministic enough to gate on: pass/fail expressed as server behaviour under a defined load profile (stays responsive to a health probe, rejects excess within stated latency, recovers within N seconds after load stops), not as raw throughput numbers that flake between runners.
• Recovery is part of the test: every scenario asserts post-attack recovery — no restart required, no corrupted state, queues drain, honest clients succeed again.
• Blast-radius safety: the suite must never be runnable against a shared / production target by accident — environment guards, dedicated fixtures, and load-generator isolation.
• Findings loop: a failing DoS scenario files / links a hardening follow-up; the suite is a source of hardening tickets, not just a gate.

Part 2 — Performance benchmark programme

The standard baseline

• One standard server: a single, fixed, documented reference configuration (hardware class or CI runner class, server build, single database, default settings) that is THE baseline every result is compared against. Changing the baseline is a versioned event, announced and re-measured, never a silent drift.
• Baseline workload suite: representative honest workloads — identity operations, capability checks, journal append/read, world sync, blob transfer — at defined scales (single user, small community, large world).

Advanced scaling techniques (measured against the baseline)

Each technique is benchmarked as baseline-vs-technique on identical workloads, reporting gain, cost, and complexity honestly — including "no gain" results:

• Workers: additional worker processes / threads / async capacity — scaling curve as workers are added, and the point of diminishing returns.
• Additional / sharded databases: extra database instances, read replicas, and sharded storage — which workloads actually benefit, cross-shard costs, and consistency-visible effects.
• Other techniques the design identifies (caching layers, connection pooling, batching) — admitted only with the same baseline-comparison discipline.

Per-interface comparisons

• Every benchmark that touches the network is run per interface: each transport / protocol surface id exposes, on identical workloads, so interface choice becomes a measured fact (throughput, latency percentiles, CPU per operation, behaviour under the Part 1 load profiles) rather than folklore.
• A published comparison matrix: interface × workload × configuration (baseline / scaled), kept current by the programme rather than by one-off measurements.

Benchmarks as regression gates

• Thresholds, not eyeballs: each gated metric has a regression budget (e.g. p95 latency or ops/sec may not regress beyond X% vs the stored baseline without an explicit, reviewed waiver recorded on the PR / issue).
• Noise discipline: runner variance measured and published; gates tuned so real regressions fail and noise does not — including the procedure for re-baselining when a change is intentionally slower (with the trade-off stated).
• Artefacts: every run stores machine-readable results (environment, build, config, workload, raw numbers) so results are comparable across months, not just across one PR.
• Cadence: per-change micro-benchmarks for hot paths, scheduled full-suite runs for the matrix, and release-gate runs for the standard server.

Shared infrastructure

• One load / workload generator framework serving both programmes (honest workloads and adversarial profiles as configurations of the same tool).
• One results store and reporting format; dashboards are a view, never the only record.
• Environment specification strict enough that "works on my runner" is diagnosable: pinned instance types or runner classes, warm-up rules, co-tenancy noise handling.

Non-goals

• Production load testing or chaos programmes against live deployments (this is CI / dedicated-environment work).
• Fixing every DoS weakness the suite finds — fixes are follow-on tickets; this programme finds, measures, and gates.
• Benchmarking other systems as competitors (internal comparisons: baseline vs scaled, interface vs interface).
• A general observability / metrics platform (consume what exists; only add what the gates require).

Acceptance criteria for the design deliverable

• A design document covering both programmes, their shared infrastructure, CI tiering, and gating rules, linked from this issue.
• The adversarial scenario catalogue (Part 1 classes above, made concrete per interface) with pass/fail and recovery criteria per scenario.
• The standard-server baseline fully specified (environment, build, config, workloads, scales) and versioned.
• The per-interface comparison matrix defined, with the scaling-technique comparison protocol (baseline-vs-technique, identical workloads).
• Regression-gate rules: budgets, noise handling, waiver and re-baselining procedure.
• Follow-on design tickets filed from the Decomposition sketch (or the sketch revised with reasons).

Decomposition sketch

Follow-on design tickets to be filed from this issue (indicative; revise during the work):

1. Design: shared load & workload generator — one framework for honest workloads and adversarial profiles.
2. Design: DoS scenario catalogue per interface — concrete scenarios, load profiles, pass/fail and recovery assertions.
3. Design: CI tiering for adversarial & benchmark suites — smoke / scheduled / soak tiers, time budgets, and environment guards.
4. Design: the standard server baseline — reference environment, workload suite, scales, and baseline versioning.
5. Design: scaling-technique comparisons — workers and additional/sharded databases vs baseline, measurement protocol.
6. Design: per-interface benchmark matrix — workloads × interfaces × configurations, results format and publication.
7. Design: regression gates — budgets, noise model, waiver and re-baselining procedure, results store.

Open questions

• How deterministic can adversarial pass/fail criteria be on shared CI runners — and which scenarios must move to dedicated hardware to mean anything?
• What is the "standard server" when contributors' and CI hardware differ — a fixed cloud instance class, a container with enforced limits, or a normalised scoring model?
• Which single workload is most representative of real id usage today, and how will the programme notice when real usage drifts away from the benchmark workloads?
• Sharding: at what measured point does the baseline actually need it — is the sharded configuration a real roadmap item or a benchmark-only curiosity?
• Who may grant a regression waiver, and how is a pattern of repeated waivers surfaced before it becomes the de-facto baseline?
• How long are benchmark artefacts retained, and what storage / CI-minute budget does the whole programme get?

Expansion of #560: adversarial coverage plus a standing benchmark programme, with benchmarks as regression gates.