Proposal: manifest-defined, permissioned app actions for custom Codex pets #34144

What variant of Codex are you using?

Codex App (Desktop), Microsoft Store/AppX package 26.715.4045.0, on Windows 11 x64 (build 26200), with a locally installed custom V2 pet.

What feature would you like to see?

Summary

Add a small, additive pet.json contract that lets a custom Codex pet bind gestures to actions already advertised by Codex or an installed/authorized app.

This is an implementation-focused child proposal for #21657, which requests broader custom-pet interaction hooks and APIs. It is complementary to #20863, which requests manifest-defined animation behavior.

The narrow first version proposed here is:

A pet may reference an existing action ID. It may not define executable code, request new connector scopes, store credentials, or bypass the existing approval system.

Concrete use case

I created and installed a valid custom V2 pet named Companion:

• 1536 × 1872 WebP atlas
• 8 × 9 grid
• RGBA transparency
• all required animation rows
• validator completed with zero errors and zero warnings

The pet works as an animated status surface, but the runtime owns every interaction. Clicking it performs the built-in Codex behavior; there is no supported way for pet.json to bind a gesture to an approved app action.

A concrete example would be music control:

• single click: play/pause
• double click: next track
• context-menu action: open the music app
• optional pet animation while playback is active

Apple Music is only an example. This proposal does not assume that the current Apple Music integration already exposes playback actions. A binding should become available only when an installed app explicitly advertises the corresponding action.

What this would enable

The same contract could support:

• play/pause or next-track controls for a music app;
• starting an approved timer;
• opening a project-specific Codex thread or dashboard;
• invoking a safe, named Codex action;
• showing a context menu of approved workflow actions;
• accessibility switches or hardware buttons using the same action registry;
• reusing the native floating pet instead of building a second desktop overlay.

Current behavior / reproducible gap

1. Install a valid V2 custom pet under the supported custom-pet directory.
2. Refresh custom pets and select it.
3. Wake the pet.
4. Connect an app that advertises one or more safe actions, if available.
5. Click or double-click the custom pet.
6. Observe that only the built-in pet interaction runs.
7. Add an interactions field to pet.json.
8. Observe that there is currently no documented schema, validator, dispatcher, permission prompt, or runtime API for it.

This is not an asset or atlas problem. The missing surface is the supported interaction contract.

Proposed additive manifest

Illustrative schema:

{
  "id": "companion",
  "displayName": "Companion",
  "description": "A custom Codex pet with optional approved actions.",
  "spritesheetPath": "spritesheet.webp",
  "interactionVersion": 1,
  "interactions": {
    "click": {
      "type": "appAction",
      "appId": "apple-music",
      "actionId": "music.playback.toggle"
    },
    "doubleClick": {
      "type": "appAction",
      "appId": "apple-music",
      "actionId": "music.playback.next"
    },
    "contextMenu": [
      {
        "label": "Open music app",
        "action": {
          "type": "appAction",
          "appId": "apple-music",
          "actionId": "app.open"
        }
      }
    ]
  }
}

The identifiers above are illustrative, not claims about current Apple Music action names.

Existing pets without interactionVersion or interactions must retain current behavior.

Suggested TypeScript contract

type PetGesture =
  | "click"
  | "doubleClick"
  | "contextMenu"
  | "keyboardShortcut";

type PetActionBinding =
  | {
      type: "codexAction";
      actionId: string;
    }
  | {
      type: "appAction";
      appId: string;
      actionId: string;
    };

interface PetInteractionManifestV1 {
  interactionVersion: 1;
  interactions?: Partial<
    Record<Exclude<PetGesture, "contextMenu">, PetActionBinding>
  > & {
    contextMenu?: Array<{
      label: string;
      action: PetActionBinding;
    }>;
  };
}

A V1 binding intentionally contains no shell command, executable path, OAuth token, arbitrary URL, or manifest-supplied payload.

Suggested dispatcher boundary

async function handlePetGesture(
  pet: InstalledPet,
  gesture: PetGesture,
): Promise<void> {
  const binding = resolvePetBinding(pet, gesture);

  if (!binding) {
    return runDefaultPetInteraction(gesture);
  }

  const action = actionRegistry.resolve(binding);

  if (!action || !action.advertised) {
    return showPetActionUnavailable(binding);
  }

  const decision = await permissionBroker.authorize({
    source: {
      type: "customPet",
      id: pet.id,
      displayName: pet.displayName,
    },
    action: action.descriptor,
  });

  if (!decision.allowed) {
    return runDefaultPetInteraction(gesture);
  }

  await action.invoke();
}

The existing app/connector action registry and approval broker should remain authoritative. The pet is only another user-interface entry point.

Security and permission requirements

• Installing a pet must not install an app or grant connector scopes.
• Only actions advertised by an installed and authorized app may resolve.
• Existing action side-effect annotations and approval policy must still apply.
• No arbitrary commands, executable paths, dynamic code, credentials, or raw OAuth tokens in pet.json.
• Prefer no manifest-supplied action arguments in V1.
• Attribute every approval prompt to the pet and target app.
• Rate-limit and debounce gesture actions.
• A missing, unavailable, or denied action must safely fall back to the existing pet behavior.
• Managed workspaces must be able to disable pet action bindings independently of ordinary pet display.
• Pet packages remain local and should not receive ambient access to conversation content.

Interaction conflicts to resolve

This proposal should compose with:

• Feature request: Add Stay and Draggable modes for Codex pets #32377: Stay and Draggable modes
• Add toggle behavior for desktop pet click to show/hide ChatGPT window #32776: click-to-toggle the Codex window
• Support configurable custom pet animation sequences and activity events #20863: configurable animation and activity events

A reasonable event priority would be:

1. pointer-down/drag threshold handles repositioning;
2. context-menu gesture opens the pet menu;
3. configured double-click wins over single-click after a short gesture window;
4. unbound or denied gestures use the current built-in behavior.

Relative implementation cost

These are relative estimates based only on public behavior; internal architecture may differ.

A useful MVP is Phases 0–2 with only click, doubleClick, and argument-free advertised actions.

Acceptance criteria for an MVP

• Existing pets behave exactly as before.
• Invalid bindings produce a per-pet validation message rather than being silently ignored.
• An authorized, advertised test action can be bound to click or double-click.
• A denied or unavailable action never executes and falls back safely.
• Tokens, secrets, prompts, messages, commands, and file contents are not exposed to the pet manifest.
• Dragging and existing overlay interactions remain functional.
• Windows and macOS use the same manifest contract.
• The feature can be disabled by policy.

Additional information

Related issues:

• Parent interaction/API request: Expose custom interaction hooks and APIs for Codex desktop pets #21657
• Animation manifest and proof of concept: Support configurable custom pet animation sequences and activity events #20863
• Stay/drag interaction modes: Feature request: Add Stay and Draggable modes for Codex pets #32377
• Click-to-toggle behavior: Add toggle behavior for desktop pet click to show/hide ChatGPT window #32776

The goal is not to turn pet packages into executable plugins. The goal is to let the native pet act as a small, permissioned front end for actions that Codex already knows how to authorize.