Repository navigation
id: speech.* — client-side transcription by default, server capture per app (#2027) #2344
Description
Follow-up from the capability taxonomy v0 investigation (#2027, record: doc/2026-10-29T14-05-00Z_investigation_capability_taxonomy/, PR #2337).
Goal
speech.* for human-conversation mode: speech.transcribe (client-side STT, the default), speech.capture (audio to server, per-app opt-in), speech.synthesize.
Why client-side first
The world model only needs text, and chat.say/events already carry text: client-side transcription costs ~no bandwidth and keeps audio on the device. speech.capture exists for devices that cannot run STT (most ESP32s) and for apps that need a better model. Per-app, not global — consent and provider choice differ per app.
Scope
- Caps and their refusal vocabulary:
cap_denied+consent_required(no consent),unknown_cap+device_lacks_affordance(no mic),failed+asr_unavailable. speech.capturereuses themedia.streamframe ceiling and consent rules; audio is never retained unless the program asks for an artifact (link themedia.*follow-up).- Interaction with the form/chat path: a transcribed utterance becomes a
chat.say-shaped or form-value input, not a new message type. - Provider boundary: the taxonomy stays provider-neutral; which STT (browser engine, local WASM model, server ASR) is a runtime choice.
Acceptance
- Cap names in CATALOG with providers; a test for consent refusal and for a device without a microphone.
- Documented default (client-side) and the per-app opt-in path for server capture.
- No audio bytes in the journal by default.
Out of scope
Speaker diarisation, wake-word detection, continuous conversation loop, TTS voice selection.
Acceptance criteria
- speech.transcribe / speech.capture / speech.synthesize are in CATALOG with providers; transcription is client-side by default and server capture is per-app opt-in, documented
- Tests cover consent refusal (cap_denied + consent_required) and a device without a microphone (unknown_cap + device_lacks_affordance); asr_unavailable surfaces as failed
- speech.capture reuses the id: media.* — typed media artifacts by default, one bounded raw stream for realtime (#2027) #2342 media.stream frame ceiling and consent rules; no audio bytes in the journal by default (retention only via an explicit artifact request)
- A transcribed utterance enters as a chat.say-shaped or form-value input, not a new message type; the taxonomy stays provider-neutral
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Medium: expected under 1 day of workNew feature or capability requestIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Filed by the new-issue-creator jobQueue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Oct 29, 2026
CODE-1189
dezren39 commentedon Oct 29, 2026
Intake review
Related: parent investigation #2027 (record: PR #2337); siblings #2338-#2345 are the other taxonomy follow-ups
Notes: Filed 2026-10-29T14:04Z from #2027. Still live: no open or merged PR implements it. Depends on #2342 for the capture stream rules.
Risks: Per-app (not global) consent is the point of the design; a global capture toggle would undo it. Audio retention defaults matter for the same reason.
Suggested approach: Transcribe (client-side) first, synthesize alongside, capture last once #2342's stream exists.
🤖 Agent note · filed
meta/enrichedapplied. Still-live check this pass: no merged PR or fix commit explicitly resolving this issue was found; where a queue triage task owns verification, it is named in the notes.developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
🤖 Agent note · triaged — still live, not started; ordered behind #2342
repo-worker-code), OpenChamber scheduled lane335a5d1f-bfc1-4831-99ea-850c8939a727Fresh state at current main
22892c316: nothing from this issue is implemented yet, and that is the correct state — the work is a capability addition on top ofmedia.*(#2342), which is still open.speech.*entries.pkgs/id/crates/id-core/src/world_caps.rsregisters exactly ten providers:time.now,time.tick,players,players.list,chat.say,chat.history,records.read,schedule.at,schedule.cancel,world.info. Nospeech.transcribe,speech.captureorspeech.synthesize.cap_denied,unknown_cap,bad_args,failed,actor_denied,outside_ceiling,result_too_large.consent_required,device_lacks_affordanceandasr_unavailableare absent (the contract has a free-form advisoryreasonfield next to the code, which is where they would land — that shape is a design decision for the implementer, not an existing mechanism).git grepformicrophone/asr/speech.underpkgs/idfinds no provider, no journal retention path and no capability test.speech.capturereuse requirement depends on id: media.* — typed media artifacts by default, one bounded raw stream for realtime (#2027) #2342'smedia.streamframe ceiling and consent rules — which is why I recorded the ordering natively: id: speech.* — client-side transcription by default, server capture per app (#2027) #2344 is now natively blocked-by id: media.* — typed media artifacts by default, one bounded raw stream for realtime (#2027) #2342 (verified:GET /issues/2344/dependencies/blocked_by→[2342]).Fix path when it is picked up (unchanged from the packet, confirmed accurate): add the three provider entries + argument schemas to the registry in
world_caps.rs(CATALOG is derived from it), extend the refusal vocabulary with the three codes, add consent/affordance/no-asr tests beside the existingcap_denied/unknown_capasserts inworld_ws.rsandworld_dos.rs, and document the client-side default + per-app opt-in.Ordering: after #2342 (
media.*), before any app-side speech work; #2341 (loc.*) is the sibling slice and shares the consent rules.No branch, worktree or PR created — triage verdict only.
8 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: handed_off
task: task 7905 (triage, repo-health-code-pilot, lane code-l)
worker: repo-worker-code
issue: #2344
Execution lane assigned: opencode. Handed off to the laptop claimant, which launches the bound session to drive task 7905 to its terminal state. Resume context from the prior attempt: triage is done — #2344 still live/unstarted, blocked-by #2342 (media.stream); the prepared findings comment (session host: /tmp/opencode/note2344.md) still needs posting after the earlier GitHub secondary rate limit.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: done (triaged)
task: 7905
worker: repo-worker-code
lane: 335a5d1f-bfc1-4831-99ea-850c8939a727
origin: openchamber-scheduled-lane
issue: #2344
Verdict: still live and unstarted; ordered behind #2342. Fresh GitHub reads show #2344 open with no linked PR, and prerequisite #2342 is still open. The recorded native blocked-by relation remains part of the prior triage evidence.
Fresh code check: branch worktree is based on current origin/main e81e202. Searching pkgs/id for speech.*, microphone, and ASR found no implementation. The prepared triage also records the missing refusal codes and test surfaces. No behavior reproduction was attempted; this is a capability addition, not a present runtime regression.
Fix path: after #2342 lands, add speech.transcribe (client-side default), speech.capture (per-app opt-in), and speech.synthesize to the provider registry; add consent/device-affordance/ASR refusal coverage; preserve the no-audio-in-journal default; document the provider-neutral client/server boundary. #2341 is the consent-rules sibling.
Task branch/worktree: queue/7905-2344-speech at /home/user/git/fleet/7905-2344-speech. Triage only; no source edits or PR.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: picked_up
task: 7905 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane aea12c82-c5af-4e5a-9b66-f8e360f87177)
state: claimed; resuming the saved triage and terminal handling.
resume: prior triage and dependency ordering are recorded; prepared findings note exists at /tmp/opencode/note2344.md. Fresh issue history already includes prior triaged/done notes, so this run is checking whether the evidence is already published and whether current main changed the verdict before deciding whether to post anything.
next: verify current-main capability surface and #2342 dependency, then complete as already done if findings remain accurate; otherwise update the finding without duplicate comments.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: triaged (current-main recheck)
task: 7905 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane aea12c82-c5af-4e5a-9b66-f8e360f87177)
verified: #2344 remains OPEN with no linked PR. #2342 remains OPEN and the native blocked-by relation is still present. On fetched origin/main
a51be0e220c2,pkgs/id/crates/id-corestill has nospeech.*,consent_required,device_lacks_affordance, orasr_unavailablecapability entries. This is still an unimplemented capability request, not a runtime regression; no behavior reproduction applies.clarification:
pkgs/iddoes contain browser-side audio file detection/rendering (web/content_mode.rs,web/routes.rs,web/templates.rs); that is distinct from speech transcription/capture/ASR providers. This narrows the earlier shorthand that there was no audio surface.ordering: keep #2344 after #2342 (
media.*); no code changes, branch edits, or PR created.dezren39 commentedon Nov 4, 2026
agent-note:v1
event: done
task: 7905 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane aea12c82-c5af-4e5a-9b66-f8e360f87177)
outcome: triaged; #2344 remains OPEN, has no linked PR, and stays ordered after open prerequisite #2342. Current-main verification and the audio-rendering clarification are in #2344 (comment). No source changes or PR; no behavior repro applies.
queue: completed as triaged; work/assigned, queue/claimed, and status/in-progress labels removed.
bookkeeping: openchamber scheduled lane — sandbox bookkeeping deferred to fleet reconciliation.