Repository navigation
User-scenario matrix: map journeys to Nightly / live / perf coverage #358
Description
Why
Nightly today is a compatibility matrix (nightly.yml) plus one live path (live-e2e.yml: CSV upload → summarize → refine → reload → preview). That does not match how people actually use FlyBot day to day. Before we rewire Nightly or grow #357, we need a shared user-scenario matrix: what we care about, what is covered, what is missing.
Goal
One living table that maps real user journeys to coverage (unit / E2E mock / live E2E / manual / none) and owns the Nightly rewrite.
Proposed axes
- Persona: personal solo · personal with friends' Bots · team member · team manager.
- Surface: local Desktop · local web · deployed instance (Deployment step 1: official image and one isolated instance per person #352) · external channel (Proactive reminders 3/3: 24/7 runtime and Slack/email delivery #355).
- Plugin / capability: core chat · tree · social (friends, approvals, files) · face-to-face · timeline · agent inbox ([Feature RFC] Agent inbox board: one place for tasks, approvals and failures (personal first, team second) #349) · proactive reminders (Proactive reminders 1/3: event reminders (approval waiting, failed, finished) in app and desktop #353–Proactive reminders 3/3: 24/7 runtime and Slack/email delivery #355).
- Risk: happy path · slow network · failure and retry · permission denied · empty / first-run.
Draft scenarios (fill coverage in comments)
| ID | Scenario | Persona | Plugins | Today | Nightly target |
|---|---|---|---|---|---|
| P1 | Cold start → first message | personal | core | none | perf (#357) |
| P2 | Single Bot chat with tools | personal | core | unit / live CSV only | live + mock |
| P3 | Tree: Chief → children → one answer | personal | tree | unit | live E2E |
| P4 | Invite friend, message friend's Bot, approve held reply | personal+friend | social | unit / README demo | live E2E |
| P5 | Face-to-face join, @bot, leave | personal+friend | social | unit / README demo | live E2E |
| P6 | Enable Timeline, read swimlanes after a multi-Bot run | personal | timeline | unit | live + perf |
| P7 | Long session scroll / reopen | personal | core | none | perf (#356/#357) |
| P8 | Background tab recover | personal | core | once #259 | nightly |
| T1 | Assign task, owner sees Needs you, approve | team | inbox + social | none (#349/#149) | after task layer |
| T2 | Approval latency across two instances | team | social + #352 | none | nightly |
| T3 | Reminder: approval waiting too long | personal/team | #353 | none | after phase 1 |
| T4 | Scheduled morning summary | personal | #354 | none | after phase 2 |
| T5 | Reminder while Desktop closed (24/7) | personal | #355+#352 | none | after phase 3 |
| X1 | Slow / failing model API: clear wait, timeout, retry | any | core | none | nightly |
| X2 | Permission denied / no grant | team | #51 | none | after grants |
How to use this issue
- Comment with corrections to the rows (add / drop / rename).
- Mark each row's Today and Nightly target until the table is agreed.
- Follow-ups: rewrite Nightly job list from the agreed matrix; wire metrics into Performance dashboard: Nightly CI metrics for FlyBot smoothness #357; keep Compatibility as a separate, thinner matrix.
Out of scope here
Implementing the tests. This issue is the map; #357 and later PRs do the work.
Process (2026-10-21)
Discussion in this issue. The earlier line about "OpenCode + Codex review" is void (it predates the owner decision of 2026-10-19); once the map is agreed, an agent may implement it and review follows .github/agent/review-policy.json. Because this issue produces a map only (a docs change, no test facility), whether that counts as routine work — and therefore agent-ready — is with the Chief; not labelled until that is answered.
Activity
Birfy commentedon Oct 19, 2026
Triage on main
7b9e90b, 2026-10-19 ~12:10 Asia/Shanghai: needs-human, not agent-ready. Needs the owner to confirm the journey list and which ones must be covered by Nightly vs live runs.Add a user-scenario runner for the ThinkFlowLab#358 matrix
Add a user-scenario runner for the ThinkFlowLab#358 matrix (ThinkFlow…
HugoXK commentedon Oct 21, 2026
Design for the owner to pick. The deterministic half is already merged: #522 ships
core/dev/eval/— a pure rubric (must/mustNot/order), the journey specs (specs.mjs, P3/P4/P6/memory), an optional model judge and grader hashing (#398). What #358 still needs is the matrix itself and its link to Nightly.(a) A hand-maintained table in docs. A
docs/scenarios.mdtable: journey × coverage (unit / mock E2E / live E2E / manual / none). Cheap, but drifts.(b) A machine-readable matrix the runner reads (recommended). A
core/dev/eval/scenarios.json(id, journey, persona, surface, risk, coverage,runner: mock|live|none) plus a docs table generated from it.run.mjslists and runs by id; a--checkfails when a journey has no coverage or a mock scenario fails. The Nightly job (#357) then runs themockrows; live rows stay inlive-e2e.--check) + the table.(c) Wire live rows now. As (b) plus the live path in Nightly. Heavier (live runs need a model key and a browser), and #204/#209 already own the live suite.
Recommendation: (b) — it turns #358's matrix into something the merged runner and Nightly can execute, and leaves the live wiring to #204/#209. Axis choices (which journeys must be mock vs live) are the owner's call on the matrix.
Birfy commentedon Oct 21, 2026
Triage spec (2026-10-21, main
16f50984). The draft rows in the body stand as the proposal; the PR fills in the two missing coverage columns.Acceptance
docs/testing/user-scenarios.mdwith one table: ID / Scenario / Persona / Surfaces / Plugins / Today / Nightly target / Blocked by.blocked by #349/#352/#353/#355instead of guessed coverage.Todayis a concrete pointer — a test file,nightly.yml,live-e2e.yml, orcore/dev/eval/specs.mjs— ornone; a row may not claim coverage without a pointer.Nightly targetis one ofunit / E2E mock / live E2E / manual / none.docs/subsystems/live-e2e.mdlinks the matrix, and thecore/dev/eval/specs.mjsP-ids match the rows (its header already points at User-scenario matrix: map journeys to Nightly / live / perf coverage #358).Out of scope: rewriting
nightly.yml(follow-up once the matrix is signed) and implementing new tests.Scope:
docs/testing/user-scenarios.md(new),docs/subsystems/live-e2e.md(link),core/dev/eval/specs.mjs(id/comment sync only). Docs-only, T0; the evidence is the table plus working file pointers, withnpm run verifygreen.Decision made under the owner release (2026-10-21): the draft journey list is the proposal; corrections belong in PR review. Marking
agent-ready— of the six released items this is the routine one (a coverage map for Nightly/SelfTest) and needs no security or policy call.Birfy commentedon Oct 21, 2026
Triage (2026-10-21): the map is ready to be written down (the axes and 15 rows above, filled in by comments), but this issue's own output is a documentation change — the body itself says "Out of scope here: implementing the tests". So the PR would be docs-only. Whether a docs-only map counts as routine work ("例行业务自身需要") is with the Chief; asked in the delivery group. Until that answer it stays
needs-human— the open question is the Chief's call, not the reporter's. Once answered, the writing work is: fill the Today and Nightly target columns for all rows, add the "how a row is verified" line, updatedocs/where the matrix belongs, and leave the test work to #357.Birfy commentedon Oct 21, 2026
Final triage state:
agent-ready, second in the Dev order. The earlier "does a docs-only map count as routine work" question is answered by the Chief: use the existing SelfTest coverage (core/dev/eval/run.mjsbaseline + rotation focus) against the journey list in the body; produce the coverage matrix + gap list; add no test infrastructure. The spec above stands.Birfy commentedon Oct 21, 2026
The Chief answered the docs-only question (this round): a docs PR counts as "例行业务自身需要" when it is part of an approved delivery — #358's matrix qualifies. It stays
agent-ready(second in the Dev order); no needs-human needed.Birfy commentedon Oct 21, 2026
Dev claim (2026-10-21, 10:2xZ): claiming #358 as
agent-ready, branchdocs/358-scenario-matrix. Docs-only per the spec (6313169588): newdocs/testing/user-scenarios.md, a link fromdocs/subsystems/live-e2e.md, and a check thatcore/dev/eval/specs.mjs's P-ids match the rows (they do today —P3-tree-split,P4-invite-approve,P6-timeline— so that file may need no edit; the PR says which).Per the Lead's direction, the
Nightly targetcolumn is not decided in this PR: each cell is待用户定with the proposal from this issue kept in parentheses, so nothing the owner must choose is pre-empted.Evidence caveat stated up front: this host has no usable sandbox backend right now, so I cannot run
npm run verifylocally; CI runs it on this PR. Docs-only, no source change, so the "revert the source change and the new test goes red" rule does not apply here.Map user journeys to today's coverage (#358)
Map user journeys to their current coverage (ThinkFlowLab#358) (Think…