Skip to content

Performance baseline: measure current smoothness (help wanted) #356

Open
Open
Performance baseline: measure current smoothness (help wanted)#356
Labels
area:automationScheduled tasks and automationenhancementNew feature or requestneeds-humanThe agent loop stopped; a person must decide (reason in the last comment)

Description

@hsliuustc0106
hsliuustc0106
opened on Oct 18, 2026

Why

We do not know how smooth FlyBot actually is. A repo search found no performance baseline or perf issues (only E2E-related #204 and #324). Before optimizing anything, we need measured numbers on real machines. This issue asks for volunteers to measure the current state and report it.

What to measure (personal mode)

Record machine (OS, CPU, RAM), FlyBot version/commit, and model used, then:

  1. Cold start: launch to the first message being sendable.
  2. First-message latency: send to first streamed token (with a fast model, to separate FlyBot overhead from model time).
  3. Streaming: does the input box stay responsive while text streams; does scroll jump?
  4. Long session: a chat with 500+ messages and many tool calls: scroll smoothness, memory after 30 min, time to reopen it.
  5. Background tab: hide the page for 5 min, return: does it recover (cf. Timeline panel stays on Loading… in a background tab (hidden page never does the first fetch) #259)?
  6. Concurrency: run 3 and 6 Bots at once (tree example): total time, CPU/RAM, any stalls.
  7. Timeline on: enable the plugin and repeat 4 and 6; note the overhead.
  8. Slow or failing network: throttle or drop the model API: are waiting, timeout and retry messages clear, or is it a blank screen?

What to measure (team mode, two or more instances once available, #352)

  1. Approval latency: from trigger to the card appearing at the other person.
  2. Many tasks: 200+ cards on the board ([Feature RFC] Agent inbox board: one place for tasks, approvals and failures (personal first, team second) #349): load time and scroll.
  3. Cross-instance retry: kill one side mid-task: is work duplicated or lost ([Team] Recover task handoffs without duplicate work #55)?

How to report

Comment with a small table: scenario, number (ms, fps, MB or subjective), machine, and a screen recording or trace for anything that feels slow. Mark each item fine / noticeable / bad. No tokens or personal content in recordings.

Output of this issue

Process (2026-10-21)

Implemented by an agent from the spec in the triage comment of 2026-10-21; that comment is the spec of record and wins where it differs from the text above. Reviewed per .github/agent/review-policy.json — no Codex review. The earlier sentence about volunteers measuring manually and "cloud OpenCode with Codex review" is void; it predates the owner decision of 2026-10-19.

Activity

  1. Birfy
    added
    enhancementNew feature or request
    area:automationScheduled tasks and automation
    needs-humanThe agent loop stopped; a person must decide (reason in the last comment)
    on Oct 19, 2026

  2. Birfy commented on Oct 19, 2026

    @Birfy
    Birfy
    on Oct 19, 2026
    Collaborator

    Triage on main 7b9e90b, 2026-10-19 ~12:10 Asia/Shanghai: needs-human, not agent-ready. Question for the owner: which metrics and which machine define the baseline? Without that a measurement cannot be checked.

  3. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Triage (2026-10-21): agent-ready. The question that parked this ("which metrics, which machine") is now decided by the owner: measure on the maintenance box itself (Node 24, same container), four metrics only, and ship a reproducible script plus one baseline table. One item at a time — this is the first.

    Metrics (all on a mock bench via core/dev/bench.sh, DEEPSEEK_API_KEY=mock-key, no real model)

    • turn end-to-end p50/p95 — message POST → that turn's turn/end, from the same session events the activity feed folds (includes the mock delay; MOCK_DELAY_MS fixed).
    • tool-call latency p50/p95 — tool/call → tool/result per call.
    • resident RSS — VmRSS of the dsh web process (pid file in .dsh-test-<name>/web.pid): after ready, after warm-up, at the end; report the peak.
    • plugin load time — process start → the dsh web: http line in web.log (covers every plugin's install).

    Acceptance criteria

    1. node core/dev/perf-baseline.mjs <bench-name> <port> --out <file.json> on a freshly reset bench, TZ=UTC, 3 warm-up + 20 measured turns, each turn one message to the Chief's own chat that triggers one tool call. Exits non-zero if the bench is down or a turn times out, and leaves no process of its own behind.
    2. Output is JSON { machine, commit, node, scenarios: [{ metric, unit, samples, p50, p95, peak }] } plus one markdown table row, so two runs diff cleanly.
    3. docs/performance-baseline.md holds each metric's one-sentence definition, the exact command, and one real run on the maintenance box with machine line (OS, CPU, RAM), commit SHA, Node version, date. No thresholds yet: a baseline, not a gate.
    4. core/test/perf-baseline.test.js covers nearest-rank p50/p95 (n=1, odd and even n), the JSON schema and the markdown row, so the numbers are not untested code.
    5. Tooling only: no change under core/src/**, no DSH patch, writes only to .dsh-test-<name>/ and the paths below.

    Change scope

    • new core/dev/perf-baseline.mjs
    • new docs/performance-baseline.md
    • docs/subsystems/ds-bot-core.md (owns core/dev/, required by the arch-doc check)
    • new core/test/perf-baseline.test.js
    • no changelog fragment (not user-visible)

    Out of scope for this pass: the UI smoothness items (items 1–8) and team mode (9–11) — they stay follow-ups until there is a baseline.

    Reproduce

    dev/bench.sh perf 3097 8797 reset
node core/dev/perf-baseline.mjs perf 3097 --out /tmp/perf.json
dev/bench.sh perf 3097 8797 stop
  4. Birfy
    added
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    and removed
    needs-humanThe agent loop stopped; a person must decide (reason in the last comment)
    on Oct 21, 2026

  5. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Triage spec (2026-10-21, main 16f50984).

    Acceptance for a PR

    1. New core/dev/perf-baseline.mjs runs the measurable scenarios on the existing bench (core/dev/bench.sh + the scripted model): cold start, first-message latency (model isolated), streaming responsiveness, a 500+ message session reopen, background-tab recovery.
    2. Emits a markdown table plus JSON: commit, machine, per-scenario ms / MB / fps, and a fine / noticeable / bad mark.
    3. New docs/performance.md records the machine, the first baseline numbers, and how to rerun.
    4. Four or five key metrics are named there for recording on each daily tag (Release cadence: at least one tag per day, small and stable #351).

    Scope: core/dev/perf-baseline.mjs (new), docs/performance.md (new), one root package.json script. No product code.

    Gate: NOT agent-ready yet — the issue asks for numbers on real machines, and a container + mock-model run is a different baseline. Blocking: the owner decides whether the container + mock-model baseline counts as the engineering baseline (real-machine numbers stay welcome as comments). If yes, this becomes the first item for Dev.

  6. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Correction to the gate line above (concurrent maintenance call): the maintenance round recorded the owner decision at 09:53 — maintenance box / same container, Node 24, four metrics, reproducible script plus one baseline table. So this is agent-ready and ordered first for Dev; the "NOT agent-ready yet" sentence is superseded. Keep the acceptance and scope above, run it on the bench, and write the numbers into docs/performance.md without claiming a real-machine baseline.

  7. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    汇总（2026-10-21）：本 issue 现在保持 needs-human，不是 agent-ready。

    • 规格以 6313125530 为准：维护轮这台机器（Node 24、同一容器）、四项指标（turn 端到端 p50/p95、工具调用延迟、常驻 RSS、插件加载耗时）、验收 5 条、改动 5 个文件（core/dev/perf-baseline.mjs、docs/performance-baseline.md、core/test/perf-baseline.test.js、docs/subsystems/ds-bot-core.md）。
    • 09:55 那条 "Gate: NOT agent-ready yet" 是对放行状态的更正，不改变上面的规格内容。
    • 卡点只有一个，是用户的一次二选一：(a) 容器内 mock 基线算不算工程基线（可复现脚本 + 表，本规格）；(b) 必须真机人工测量（人会占用机器，脚本就不必交付）。Chief 不降门槛，走用户点选。
    • 拍板前 needs-human；拍板后要么按 (a) 转 agent-ready，要么按 (b) 改写测量方案。Dev 侧本期全机 shell 不可用，不催它。
  8. Birfy
    added
    needs-humanThe agent loop stopped; a person must decide (reason in the last comment)
    and removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 21, 2026

  9. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Final triage state (Chief decision this round, 2026-10-21): agent-ready, first in the Dev order. The earlier "needs-human pending the user's a/b choice" is superseded: the Chief fixed option (a) — run on the maintenance box (Node 24, same container) and deliver a reproducible script plus one baseline table. Metrics fixed by the Chief: turn end-to-end p50/p95, tool-call latency, resident RSS, plugin load time. The 09:53 spec is the spec of record; keep the acceptance/scope there. One item at a time: #358, #471, #355 follow.

  10. Birfy
    added
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    and removed
    needs-humanThe agent loop stopped; a person must decide (reason in the last comment)
    on Oct 21, 2026

  11. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    收口（2026-10-21，更正上一条的卡点口径）

    • 规格以 6313125530 为准。
    • 机器与四项指标按 Chief 群决定：维护轮这台、Node 24、同一容器；指标 = turn 端到端 p50/p95、工具调用延迟、常驻 RSS、插件加载耗时。
    • 当前门不是缺信息，是沙箱：Bot 侧 shell 不可用（bash 报 sandbox 后端不可用），要等用户批 danger-full-access / bubblewrap；在此之前 Dev 开不了工。
    • docs/performance-baseline.md 的基线表不设阈值（是基线，不是门）。
    • 保持 needs-human；沙箱放开后再放 agent-ready，不提前标。

  12. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Dev-side blocker (per the Lead): blocked: host has no usable workspace-write sandbox backend; Dev needs the user to approve danger-full-access (or install bubblewrap) before it can run the script/tests. The issue stays agent-ready; this is environmental, not missing information.

  13. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    Blocked 注记（2026-10-21）：本 issue 的规格已定（以 09:53 那条为准，机器 = 维护轮这台、Node 24、同容器，四项指标，基线表不设阈值），当前只卡沙箱：host 没有可用的 sandbox backend（bash 直接报 workspace-write 后端不可用），要等用户批准 danger-full-access / bubblewrap。批准前 Dev 只能做只读准备，不放 agent-ready。

  14. Birfy
    added
    needs-humanThe agent loop stopped; a person must decide (reason in the last comment)
    and removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 21, 2026

  15. Birfy commented on Oct 21, 2026

    @Birfy
    Birfy
    on Oct 21, 2026
    Collaborator

    State aligned with the Chief: needs-human while the user's sandbox authorization card is open — the spec and acceptance above stand. Blocking: the user (approve danger-full-access, or install bubblewrap); the moment the card is approved this flips back to agent-ready and Dev claims it. This is an environment block, not missing information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:automationScheduled tasks and automationenhancementNew feature or requestneeds-humanThe agent loop stopped; a person must decide (reason in the last comment)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions