V2: route progressive AGENTS.md through durable Instructions #34341

Summary

Define and implement the user-visible semantics for progressive path-scoped AGENTS.md instructions in V2.

The current implementation injects discovered files as synthetic user messages. This gives them an accidental lifetime: they remain sticky until compaction, can disappear as authoritative instructions after compaction, and may be rediscovered only after deduplication state resets and another matching read occurs.

We should choose the lifecycle first, then choose persistence, watching, and Instructions admission to implement it.

Settled direction

• A successful local read(file) or directory listing discovers applicable path-local AGENTS.md files.
• Authoritative instructions must flow through Core Instructions and the durable instruction checkpoint, not raw synthetic user messages.
• Discovery and admission are separate facts:
  • discovered means the session knows an instruction path may apply;
  • admitted means the model has actually been told its current effective contents.
• Changes become model-visible at a safe provider-step boundary as chronological instruction updates. Ordinary updates do not rewrite the original top-level system prompt; completed compaction may rebuild the baseline.

Current user-visible semantics

These semantics apply to path-local files discovered below the Location root. Global and project-root instructions already belong to the ambient baseline.

This resembles “active for one attention epoch,” but the boundaries are accidental: token-pressure compaction expires authority, while process lifetime controls rediscovery.

Semantic choices to make

1. How long does discovery last?

Choose one:

• Session-sticky: once discovered, track the path for the rest of the session. Compaction and restart do not forget it.
• Instruction epoch: compaction intentionally expires path-local discoveries. Reading under that path later rediscovers them.
• Explicit working set: retain only recently relevant scopes, with a deliberate eviction rule independent of compaction.

Recommended starting point: session-sticky discovery. Compaction is a transcript/storage operation and should not silently change which authoritative project instructions apply. If long sessions accumulate too many files, add an explicit bounded-working-set policy based on observed usage rather than inheriting compaction's incidental boundary.

2. What happens when a discovered file changes?

Choose one:

• Freeze the discovery-time text.
• Re-observe the file and announce edits before the next provider step.

Recommended: re-observe tracked files. An edit becomes effective once, at the next safe provider-step boundary, as a per-file chronological Instructions update.

A filesystem watcher is optional implementation machinery, not required semantics. Re-reading tracked files while preparing each provider step provides the same model-visible behavior without waking an idle session or managing watcher lifetimes.

3. What happens when a discovered file disappears?

Choose one:

• Continue applying the last readable contents indefinitely.
• Mark the file inactive and announce that its instructions no longer apply.
• Forget the path entirely, requiring a future matching read to rediscover a recreated file.

Recommended: distinguish absence from transient failure.

• NotFound: mark the file inactive and announce removal.
• Other read failure: preserve the model's last-applied belief and retry at the next safe boundary.

Keep the path tracked after NotFound, so recreating the same file can reactivate it without requiring an unrelated read. If we instead want deletion to forget discovery, make that an explicit policy.

4. Must discovery persist file contents?

Choose one:

• Persist the path and discovery ownership only; the filesystem supplies current content.
• Also persist discovery-time content as a fallback.

Recommended: persist paths and lifecycle metadata, not another copy of file contents. The instruction checkpoint already stores the exact value last admitted to the model and can preserve that belief during transient unavailability or restate it during compaction. Treat NotFound as removal rather than silently applying a stale discovery snapshot.

5. What is the scope and precedence of accumulated files?

Choose one:

• Every discovered file remains globally active for subsequent session steps.
• Only instructions applicable to a current explicit working set remain active.

Whichever policy we choose must define deterministic ordering. The default should remain broad-to-specific: global, project, then path-local ancestors from root toward the target. More-specific instructions appear later.

Recommended initial slice: retain all discovered paths for the session and preserve broad-to-specific order. Measure actual instruction growth before introducing implicit eviction.

6. How do session lifecycle operations affect discovery?

Recommended:

• Fork copies discoveries owned by transcript boundaries inherited by the child.
• Committed revert removes discoveries first made after the revert boundary.
• Session move clears path discoveries so the destination rebuilds from its own filesystem.
• Session deletion removes all discovery state.

Recommended semantic state machine

unknown
  -- matching successful read --> tracked + active

tracked + active
  -- content edited -----------> tracked + active, announce new contents
  -- NotFound -----------------> tracked + inactive, announce removal
  -- transient read failure ---> tracked + active, retain last-applied belief
  -- compaction ---------------> tracked + active, restate in new baseline

tracked + inactive
  -- file recreated -----------> tracked + active, announce current contents

Implementation consequences

If the recommended semantics are accepted:

1. Record a durable discovery fact containing canonical paths and transcript ownership, not model-facing synthetic text.
2. Project the active tracked paths per session.
3. Include those paths in the session's normal core/instructions source.
4. Re-observe tracked files before each provider step; a watcher is not needed for the first implementation.
5. Let the instruction checkpoint compare the observed files with the last-applied snapshot and emit additions, edits, and removals.
6. Do not persist discovery-time file contents unless we explicitly choose stale-content fallback semantics.

PR #35497 currently implements session-durable tracking, live re-observation, per-file deltas, and explicit fork/revert/move handling. It also stores discovery-time content as an unreadable-file fallback. Review that fallback against choice 4 before merging.

Needed plugin/core seams

• A post-settlement tool hook containing at least:
  • sessionID
  • effective agent
  • tool name
  • decoded input
  • structured output/result
  • resolved/canonical filesystem resources
• A session-scoped Instructions contribution because discovered files belong to a sessionID, not only a Location.

Provider follow-ups

• Replace the hardcoded Anthropic native chronological-system support check, currently tied to claude-opus-4-8, with route/provider capability metadata.
• Verify Anthropic cache behavior for explicit block cache_control versus top-level automatic caching.
• Coordinate with the broader V2 tool plugin architecture work in V2: add tool registration to Promise plugins #34489.

Related work

• Current synthetic implementation: d972aa9d83, followed by directory-read support in bea6e1499d.
• Durable discovery proposal: feat(core): make path-local instruction discovery durable #35497.
• Vocabulary rename from System Context to Instructions: refactor(core): rename system context to instructions #35583.
• Original design discussion: https://discord.com/channels/1391832426048651334/1521953168899182612