Expose custom interaction hooks and APIs for Codex desktop pets

What variant of Codex are you using?

Codex App (desktop), with custom pets generated/installed through Hatch Pet. This may also be relevant to CLI/SDK surfaces if they expose the same pet/action contract later.

What feature would you like to see?

Please expose a supported interaction/API contract for Codex desktop pets, beyond visual spritesheet customization.

Requested capabilities:

Let a custom pet declare safe interactions in pet.json, e.g. click, double-click, right-click/context menu, hover, drag/drop, and optional keyboard shortcut actions.
Let those interactions call first-party Codex actions or approved deep links, such as:
- open/tuck the pet;
- open the current or related Codex thread;
- create a new thread for a configured workspace/prompt;
- open a local project dashboard or app route through an allowlisted URL/deeplink;
- invoke a named skill/command where permissions allow.
Expose a small, documented read-only status/event API for pets to observe agent/thread state, e.g. idle, running, waiting for user, ready for review, failure, latest activity category, active workspace/thread id.
Keep the contract permissioned and safe: no arbitrary command execution by default; require user approval/allowlisting for external URLs, local apps, workspace commands, and any write-capable integration.
Document the interaction contract so third-party dashboards can integrate with the official Codex pet instead of building their own always-on launcher.

Additional information

Related but distinct: #20863 asks for configurable animation sequences and activity events. This request is about interaction hooks and open integration surfaces.

Use case: a local project dashboard or personal wiki may want the official Codex pet to act as a lightweight entry point and status surface. For example, clicking the pet could open the project dashboard, create a project-specific Codex thread, or show safe actions for maintaining the project. Without supported hooks, users either treat pets as visual-only status indicators or build a separate desktop companion/launcher, which duplicates the Codex pet's role.

A minimal first version could be a manifest-defined context menu with allowlisted codex:// actions and read-only status events. More advanced versions could expose a small SDK/API for registered local apps.

+1

I’d like to add a related use case that would benefit from a small plugin-published pet state API, not just read-only pet/status observation.

I’m building a local Codex plugin for a Steam-focused Windows setup. The plugin can already detect useful external state locally, for example:

Steam client open/closed
Steam Big Picture/controller UI active
installed Steam app/game process running
launch or validate action requested from a local dashboard
recent Steam app update/patch activity

Today the plugin can render a separate dashboard companion using the same Codex pet spritesheet contract, but it cannot drive the native floating Codex pet. That means the official pet and the plugin companion duplicate each other.

A minimal API that would solve this cleanly:

codex.pet.setState({
  source: "steam-command-center",
  animation: "running", // existing pet row id
  title: "Crimson Desert",
  detail: "Game running",
  ttlMs: 5000,
  priority: 40
})

codex.pet.clearState("steam-command-center")

Suggested constraints:

Only allow existing pet animation row IDs, e.g. idle, waving, jumping, failed, waiting, running, review, etc.
Require explicit user permission per plugin/source, e.g. “Allow Steam Command Center to temporarily control pet status.”
Require TTL expiry so stale external state does not leave the pet stuck.
Include source attribution so users know what is controlling the pet.
Let first-party Codex work states win over lower-priority plugin states, especially waiting-for-input, approval, failure, and ready-for-review states.
Avoid arbitrary code execution or arbitrary animation injection; this should be a small state handoff, not a general desktop automation surface.

The broader design ask: let trusted plugins publish ephemeral, permissioned pet state to the native floating pet overlay, while Codex remains the arbiter of priority and safety. This would let local integrations such as Steam, OBS, CI monitors, build dashboards, or hardware monitors use the official pet as the shared status surface instead of each building their own overlay.

I want to add a related product-design point from the CLI user side.

For my workflow, the main value of pets is not that they appear inside the current terminal surface. The value is that they are an ambient, always-visible status surface for Codex work: running, waiting for input or approval, ready for review, failed, idle, etc. If the pet is confined to a terminal pane, it only works while that terminal is visible and focused enough to notice. That makes it much less useful as a status indicator.

This is also why terminal pets feel like the wrong primary design direction to me, independent of tmux / Sixel / Kitty compatibility. Terminal rendering is useful as a fallback for SSH, headless, or no-Desktop environments, but the primary local experience should be:

CLI sessions publish lifecycle/status events locally.
Codex Desktop floating pet overlay consumes those events.
The Desktop pet stays above other apps and reflects active CLI work in the same way it reflects Desktop-created work.
Terminal pets remain an optional fallback, not the main status surface.

This is related to #22625, but I do not think the important framing is “the terminal pet does not work” or “tmux does not support images.” The more important issue is that pets are fundamentally ambient status UI, and ambient status UI should live outside the terminal container. A small permissioned state API like the one discussed above (codex.pet.setState(...), or a first-party CLI -> Desktop event bridge) would solve this cleanly without requiring every integration to build its own overlay.

+1 for a documented, read-only pet/task status API. I have a concrete external-monitoring use case that is not currently covered by hooks or by connecting to a separately managed app-server.

I want to run a small local background service that can reflect Codex activity while I continue using Codex normally across the Desktop app, CLI, and IDE. The minimum useful state model would be:

idle: no active turn
running: Codex is reasoning, calling tools, editing files, or streaming a reply
waiting_for_approval: the active turn is waiting for command/file/network/tool approval
error: the turn failed or the thread entered a system-error state

A minimal API could expose both a current snapshot and a stream of transitions, for example over a user-scoped local socket:

task/thread id
active turn id, when present
state
source surface (desktop, cli, ide)
timestamp
optional redacted error category

Important lifecycle semantics would include explicit approval-requested and approval-resolved transitions, plus turn-failed/turn-completed events. Prompt text, assistant messages, commands, and file contents should be excluded by default.

Today the available mechanisms leave gaps:

Global hooks provide useful events such as UserPromptSubmit, PermissionRequest, and Stop, but there is no complete approval-resolved or turn-error lifecycle for an external status service.
App-server exposes the right structured events, but Desktop/IDE clients own their app-server connection, often over private stdio, so an unrelated local process cannot subscribe to the same stream.
Parsing persisted transcripts would be brittle because that format is not a stable public interface.

The built-in floating pet already derives equivalent product states (Running, Needs input, Ready, Blocked). Exposing a privacy-preserving read-only view of that same derived state would let menu-bar apps, accessibility tools, hardware indicators, and custom desktop companions integrate without parsing conversations or forcing every Codex client through a custom shared app-server.

I opened #34145 as an implementation-focused child proposal for the interaction contract discussed here. It narrows the first deliverable to manifest-defined, permissioned bindings for actions already advertised by Codex or an installed app, with an illustrative music-control use case, a backwards-compatible pet.json shape, dispatcher pseudocode, security boundaries, staged implementation cost, and MVP acceptance criteria. It is intended to complement—not replace—this broader API request.

Add hatch, the last thing Codex pets do that this did not

不如设计成微信小程序那种模式，给一个空白的窗口，大家都会用ai，大家想怎么玩就怎么玩。

与codex的链接用还是接口来实现。

Adding a concrete custom v2 pet use case to this request: a quiet companion that notices the physical pointer when idle and gives a short affectionate response when hovered or clicked.

A small opt-in declarative animation interaction contract would cover this without arbitrary JavaScript or command execution:

Idle proximity gaze: use the existing 16 look-direction cells when the physical pointer is near the pet, with a center deadzone, distance/angle hysteresis, and return to idle on leaving. Define precedence when the Computer Use cursor is active. Related pointer-source report: Codex Desktop v2 pet look directions ignore the physical mouse and only follow the Computer Use cursor on Windows #33224.
Bounded reactions: map hover and click to an existing animation row, play once, then resume the appropriate state. Support a cooldown that consumes repeated inputs rather than queueing them or restarting the animation continuously.
Drag lifecycle: apply a movement threshold and pointer capture/cancellation; dragging wins over a reaction, and release after dragging must never become a click reaction.
Task/accessibility priority: keep waiting-for-user/error/task states legible, support reduced motion and keyboard activation, and retain current click-to-open behavior unless the user explicitly opts into a different mapping.
Preserve compatibility with existing v1/v2 pets. The manifest could describe supported events, existing row IDs, one-shot/loop behavior and timing; the host would own hit testing, scheduling and priorities.

I implemented a browser-only prototype in silent-pet, including an alpha-aware body hit test, 180/210 CSS-pixel proximity hysteresis, a 350 ms hover dwell, an 8 px drag threshold, a 2.4 s cooldown, and a 3-degree direction hysteresis. These are tunable prototype values, not proposed universal defaults. The repository includes 15 passing automated state-machine/animation tests and documents the boundary between sprite customization and host behavior.

Useful acceptance cases would be: entering/leaving the attention radius, movement across a direction boundary, a quick click, sustained hover, rapid repeated clicks, drag release outside the pet, pointer cancellation, task-state changes during a reaction, reduced motion, and physical-pointer coordinates across monitor/DPI changes.

Environment context: Windows AppX package version 26.915.4065.0, based on local static inspection. This is a feature proposal, not a claim that I reproduced #33224 in the native UI. The new browser input flow could not be automated in this session because the automation runtime failed to initialize; the unit tests and prototype are not native Codex acceptance evidence.

The installed pet remains a normal v2 sprite package: I can put a gentle expression in the existing hover animation slot, but richer pointer/click scheduling needs a supported host contract.

我也有相同需求，想补充一个具体的Windows桌面使用场景。

我在做个人桌宠“皎璘”。目标并不是创建另一个Codex客户端，而是让官方客户端继续正常升级，同时保留自己设计的高清角色与动画。例如：开始处理回合时取出并打开笔记本，任务持续时循环敲键盘，最终停止后合上收回；两个任务同时运行时，不应因其中一个结束就提前停止工作动画。

我希望支持两种实现方式：

1. 官方宠物的声明式扩展。 自定义帧数/时序、进入/循环/退出阶段与中断策略；支持高清或按动作资源；提供有安全边界的缩放、鼠标命中、拖动和控件锚点。已有宠物保留默认行为，新配置按需启用，不要求放开任意原生代码。

2. 独立本地伴随应用的只读事件接口。 能订阅主回合的开始、最终结束、中断、失败/等待及当前活动快照；带稳定的session/turn身份、必要的主/子关系与有序或可去重的事件信息。断线和接收器重启时能重新对齐当前状态，不把旧事件误当正在工作。用户明确同意后才能连接，默认不携带任何提示词、回复、工具输入输出或账户内容，也不授予任务控制权。

目前公开Hooks已经很有帮助。我想特别确认：

Windows桌面客户端中，哪些版本和任务模式支持相应生命周期事件及正常的hook审阅/停用？
Stop hook可能要求继续执行时，外部观察者如何判断“最终回合已结束”？
用户中断、等待审批、连接丢失、重试耗尽和进程异常退出，各有什么明确的投递保证或状态恢复办法？
本地任务、云任务和多个客户端的事件发生在哪里？是否有受支持的桌面端转发或状态快照，而不需要读取会话数据库或暴露本地服务到公网？
插件能否以清晰的一次安装/信任流程携带这些只读状态桥接配置，并且易于撤销？

对独立桌宠而言，即使官方暂时不开放完整播放器控制，只要提供稳定、最小化的状态接口，也能避免每个项目维护一套客户端改造分支。

本建议不要求关闭完整性保护、不请求OpenAI私钥，也不要求公开私有源码。希望的是一个受支持、版本清晰、权限最小的扩展约定与Windows示例，让Codex更新与个人桌宠设计可以各自继续。

相关请求：#21657（交互与状态接口）、#20863（动画序列与事件）。请按既有讨论归并这个场景；我理解功能请求不等于产品承诺。

Navigation Menu

Repository navigation

Expose custom interaction hooks and APIs for Codex desktop pets #21657

What variant of Codex are you using?

What feature would you like to see?

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Expose custom interaction hooks and APIs for Codex desktop pets #21657

Description

What variant of Codex are you using?

What feature would you like to see?

Additional information

Activity

mn8821236 commented on May 10, 2026

scottzzj commented on May 10, 2026

ajshpprd commented on May 23, 2026

xnng commented on Jun 17, 2026

RookieGz commented on Jul 10, 2026

hiroki-tamba-research commented on Jul 19, 2026

bru-ys commented on Aug 26, 2026

Quiyyy commented on Sep 19, 2026

firefly020725 commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions