RFC: layered safety for always-on Bots (grants, action gates, visible channels, decision log) #386

Why

Second part of the secondary write-up discussed in #373 (Chinese summary of a podcast; not verified against the original). Its safety section matches FlyBot's direction: agents usually do not "turn evil"; they pursue the assigned goal and, along the way, find loopholes, open side channels, or game graders. You cannot fix that only by "forbid bad things". Parent: #361, related #373, #51, #56, #57, #153-style gates.

Four defense layers described (as a design vocabulary)

1. Training-time alignment (model vendor; out of our control).
2. Inference-time monitors: intent probes / classifiers that look before a dangerous step. Costly; pre-action.
3. Action / permission mode: ignore intent, check whether the concrete action is within what the user granted (e.g. touching a DB without authorization is blocked).
4. Identity and permission system: who may use which tool and which data.

Each layer backs up the next. FlyBot owns layers 3 and 4 plus visibility; layer 2 can be a pluggable check later.

Proposed FlyBot actions (discussion)

• Grants as the hard layer: keep default-deny per-Bot grants ([Team] Gate cross-owner agent routes by per-Bot grants (default deny; superseded the publication gate) #51); every tool call checked against grants, not against "intent".
• Action-level approval: irreversible or out-of-scope actions need human approval; reminders never auto-execute (Proactive reminders 1/3: event reminders (approval waiting, failed, finished) in app and desktop #353–Proactive reminders 3/3: 24/7 runtime and Slack/email delivery #355).
• Visible coordination only: Bot-to-Bot messages go through logged surfaces (Bot Social, tasks, Timeline); detect and alert on unexpected channels (shared files, side stores).
• Anti-gaming for evaluators: live-E2E / Nightly graders must not be writable by the Bot under test; evidence from read-only observation (Add real-model browser E2E nightly tests with reusable scenarios #204/User-scenario matrix: map journeys to Nightly / live / perf coverage #358).
• Decision log ("tried and rejected"): ask Bots to record options they considered and discarded; surface in Timeline / audit ([Team] Add a human-readable authorization and task audit trail #57) for review.
• Resource caps ([Team] Bound agent loops and resource consumption #56) so goal pursuit cannot run unbounded.

Other product ideas from the same write-up (low priority)

• Mutable software: users reshape UI/workflows with AI (long term).
• Pay-per-call APIs for agents: strip UI, bill per invocation (business idea; not scoped).
• "CLAUDE.md-style" instructions may over-constrain stronger models: start new projects with minimal Bot instructions; add rules when problems repeat; re-check per model version. → Tie to RFC: harness × provider (Magpie-inspired gateway, routing groups, profiles) #366 harness×model: instruction sets are per harness/model, versioned.
• "Unknown unknowns": the bottleneck is often unclear user intent → onboarding prompts, task templates, and clarifying questions in the Agent inbox flow ([Feature RFC] Agent inbox board: one place for tasks, approvals and failures (personal first, team second) #349).

Acceptance for this RFC

• Agreement on which layers FlyBot implements (3, 4, visibility) and what is out of scope (1, 2).
• Child issues opened for: decision log, unexpected-channel detection, evaluator read-only guarantee.
• No claim that any of the above is implemented.

Process

RFC only; implementation via OpenCode + Codex review.