Pool intelligence: Artificial Analysis monitoring + our measured session speeds + add/retire flags

Work (Drew, 2026-10-27)

A daily pool-intelligence job (pool-intelligence) maintains the
fleet's model of the model market — ours and Artificial Analysis's:

AA monitoring: pull current Artificial Analysis intelligence /
speed / price data for every model in (or adjacent to) our pools.
If AA's API needs a key we don't have, record that as the gap and
use the public pages until a key exists; never invent a number.
Our own measurements: from real fleet runs (session records,
route attempts, babysitter log, queue outcomes), maintain per
provider/model/effort: output tokens/sec, wall-clock per completed
task, first-response latency where observable, success/stall
rates. Our measurements outrank AA's where they disagree — AA is
the prior, our runs are the evidence; record both.
Interpretive model (living doc, goal workspace files/): how
we read an AA point against our pools — what a 1-point AA
difference is worth in speed/cost on OUR work, per task class.
Updated as measurements accumulate; every claim in it cites its
data window.
Add/retire flags: the standing output. A flag names the model,
the pool, and the quantified trade: "X is missing: AA ≈ pool
median +2, 1.8× our measured speed of the pool's slowest member,
plan-covered" or "Y should retire from workhorse: fast enough but
costs the pool its scarcest window; Z is −1 AA, faster, and
cheaper on our measurements." Flags become issues with the numbers
attached; a flag is a hypothesis, and the controlled-test issue
(sibling) is how it earns a config change.
Feed the steward: flags + measurements land where the steward and
meta-reviewer read them each cycle.

Job live; first intelligence report covers every pooled model
with AA values (or the recorded AA-access gap) + our measured
speeds where runs exist.
The interpretive-model doc exists with its first data window.
First add/retire flag filed as its own issue with quantified
tradeoffs, or an explicit "no flag survives the numbers"
result with the table that shows why.

Child of the pool & routing intelligence epic. Filed by Muse
(operator) 2026-10-27.

🤖 Agent note · state: done

Agent: repo-queue worker lane C (task 1326, triage)
Session / ID: repo-queue-worker-shift-c, 2026-10-28
Found via: queue claim
Task: Triage verdict for Pool intelligence: Artificial Analysis monitoring + our measured session speeds + add/retire flags #1341 — the work is already live and every acceptance criterion is met with evidence; no decomposition or follow-up packets needed. Closing as completed.
Expectation: done as defined (triage note + documented no-action rationale).
Links: job pool-intelligence · files/pool-intelligence-model.md · flag issue pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409

Job live + first report: pool-intelligence (daily 08:12, enabled, owned by this goal) ran 2026-10-27. The AA-access gap is recorded exactly as the issue prescribes — the AA API returns 401 without a key, so AA data came from the public leaderboard (272 model/effort rows, parsed in full; never estimated). Our measurements came from the fleet's own route_attempt records (2,424 attempts in the run's 24h window; 6,319-attempt table overall) — our measurements outrank AA where they disagree, per spec.
Interpretive-model doc exists: files/pool-intelligence-model.md in this goal's workspace — "Pool intelligence model", first edition, every claim citing its data window, with the AA gap and data-source standings recorded at the top.
First add/retire flag filed: pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409 — "pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task" — a quantified flag as its own issue, as specified. It stands or falls on its own controlled-test path, not on this triage.

Routing: no follow-up packets — the daily job is the standing implementation, and the flag loop is already turning. Closing #1341 as completed.