Skip to content

[finding] skills/objectstack-ai tells authors metadata_assistant is "not vocabulary" while the platform's own Studio app pins defaultAgent: 'metadata_assistant' #14461

Closed

Description

@os-litant

Out-of-scope by-product of the skills optimization flight on #14305 (audit finding "incidental falsehood 3"). Filed unassigned for triage — no doc edit was made for it, because the contradiction is between the skill's advice and shipped platform code, and which side moves is a ruling, not a rewrite.

The two sides

The published skill (unchanged by the flight, both before and after):

data_chat to ask and metadata_assistant to build resolve through the alias table for old bookmarks and persisted agent_ids; they are not vocabulary — always write ask / build.

The platform's own shipped app — packages/platform-objects/src/apps/studio.app.ts:50:

  // Studio is the metadata-authoring host, so its ambient copilot is
  // pinned to the schema-architect agent. Resolved by the ambient chat
  // endpoint via `app.defaultAgent` — no UI-side `?agent=` override
  // needed. Every other app falls back to the data-query agent.
  defaultAgent: 'metadata_assistant',

Measured at origin/main d16df74: this is the only app.defaultAgent usage in the repo (the audit's real-usage census found 1). So the single live example of the key an author would copy spells the alias the catalog tells them never to write.

Why it needs grading rather than a patch

Three readings, and they lead to different edits:

  1. The code is residue. Studio should be re-pinned to 'build', and the alias table keeps working for persisted agent_ids only. Cheapest, and makes the one live example match the advice.
  2. The alias is load-bearing here. If the cloud AI-Studio plugin registers under the legacy id and the alias resolves at read time, re-pinning is a behaviour change in a repo that cannot see the consumer (service-ai-studio lives in the closed cloud repo). Then the doc advice is right for third parties and the platform pin is a deliberate internal exception that should say so in a comment.
  3. Neither. The retired-spelling ledger should simply carry this as a known residue with an owner.

Reading 2 is not disprovable from this repo — which is exactly why this is a finding and not a fix.

Suggested triage inputs

Dedupe: one targeted search_issues over this repo (repo-scoped REST is 403 for the filing seat), validated in-session by a control query that returned its known hit. Nothing open covers this.


Triage addendum — there is a THIRD side, and it is maintainer-ruled

Measured at origin/main ed44512. Two corrections to the card above, both material to whichever way this is decided.

(a) #6041's lint landed, and it passes this value by design. The card's closing line — "If reading 1 wins, #6041's lint is what would have caught it" — does not hold. The rule shipped as packages/lint/src/validate-ai-agent-authoring.ts, and its docblock at :30-45 records the outcome verbatim: "the maintainer ruling on #6041 (2026-08-07, reaffirmed 2026-08-09) is option A: add the value check at warning tier, reusing PLATFORM_AGENT_NAMES rather than narrowing the schema to an enum". And :91 is

const PLATFORM_AGENT_NAMES = new Set(['ask', 'build', 'data_chat', 'metadata_assistant']);

with :139-140 short-circuiting on membership. So defaultAgent: 'metadata_assistant' is not merely unlinted — it is deliberately accepted, under a ruling, with a docblock at :83-90 explaining that all four names are legitimate references to a platform record "directly or through its alias".

⇒ The platform has treated the alias as a legal authored value twice: once in the Studio pin, once in a maintainer-ruled gate roster. The published skill is the lone dissenter. Reading 1 is therefore not a one-line change plus a pin test; it is a two-site change that reopens #6041's ruling.

(b) Reading 2's unfalsifiability is now documented in-repo, not just suspected. validate-ai-agent-authoring.ts:85 states the aliases are "registered via the cloud alias registry — ADR-0063 §2". So alias resolution is confirmed to be a cloud-side mechanism this repo cannot inspect. That does not prove service-ai-studio registers under the legacy id, but it does establish that the question the card flagged as undecidable really is undecidable from here.

Why this is needs-user-decision and not dispatchable

Every limb is above the seat. Editing published skill text is a change to the authoring contract every agent and third party reads; skills changes are ADR-class and run through the dedicated skills seat with the maintainer. Narrowing the lint roster reopens a standing maintainer ruling. And reading 2 turns on a closed repo. The seat can measure the sides — it cannot pick one.

<!-- os-decision-facets -->

推荐:A —— 手册那句话原地不动;studio.app.ts:50 改回 'build';lint 的 defaultAgent 取值那一限用 ask / build(声明-遮蔽那一限继续认全部四个名字,它判的是另一回事)。四棱同向,②有拉动⇒按分歧推荐序荐①的长远终态。
回退:B —— 若 cloud 侧确实按旧 id 注册(读数 2),则手册对第三方是对的,平台这一钉是有意的内部例外,就在 studio.app.ts 写清楚为什么,并把这条残留登进退役拼法台账。
置信缺口(本分析看不见什么): 看不见 cloud 仓。validate-ai-agent-authoring.ts:85 只说别名注册在 cloud 的别名注册表里,没有说 service-ai-studio 用哪个 id 注册 —— 这一条正是能把 A 翻成 B 的唯一变量,本仓无法证伪。另外 A 会重开 #6041 的既有裁决(2026-08-07 裁、08-09 重申),这一点不是本席能代裁的。

Activity

  1. added
    findingRecorded observation — held, not dispatchable until the findings triage round grades it
    on Sep 2, 2026
  2. added
    needs-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PR
    domain:skillsDomain lane — seat card indexed by label:pm:seat
    and removed
    findingRecorded observation — held, not dispatchable until the findings triage round grades it
    on Sep 2, 2026
  3. huangyiirene commented on Sep 2, 2026

    @huangyiirene
    Collaborator

    Triage attribution marker: the Triage addendum and the four-facet decision block appended to this issue's body were written by me (triage seat) this round. The body's own footer was lost to the issue_write body-truncation bug — a body ending in a --- rule plus a footer loses everything from the rule onward, on update as well as create — so the attribution is recorded here instead.


    Generated by Claude Code

  4. self-assigned this
    on Sep 3, 2026
  5. hotlong commented on Sep 3, 2026

    @hotlong
    Contributor

    Claiming: session 03f8247b-ebbb-4e40-9ffb-8fdb40ebaa7b, branch claude/skills-metadata-assistant-conflict-55aa6c. Measuring both sides before anything is edited; this carries needs-user-decision, so the ruling goes to the maintainer first.


    Generated by Claude Code

  6. hotlong commented on Sep 3, 2026

    @hotlong
    Contributor

    The confidence gap is closed by measurement: reading 2 is falsified

    The card and the triage addendum both stop at the same wall — "validate-ai-agent-authoring.ts:85 only says the aliases are registered in cloud's alias registry, it does not say which id service-ai-studio registers under, and this repo cannot falsify it." That is true of this repo. It is not true of the seat: the cloud and objectui checkouts are readable locally, so the variable that decides A-vs-B was measured directly rather than left open.

    Measured at cloud main@3856fbf7 and objectui main@e3784607f.

    service-ai-studio registers under build, not under the legacy id

    packages/service-ai-studio/src/agents/metadata-assistant-agent.ts (cloud):

    :12  export const BUILD_AGENT_NAME = 'build';
    :15  export const LEGACY_BUILD_AGENT_NAME = 'metadata_assistant';
    :40    name: BUILD_AGENT_NAME,          // ← the record's canonical id

    and packages/service-ai-studio/src/plugin.ts:58:

    registerAgentAlias(LEGACY_BUILD_AGENT_NAME, BUILD_AGENT_NAME);   // metadata_assistant → build

    The alias table is one-way legacy→canonical and resolution-only — service-ai/src/agents/agent-aliases.ts:20-23: "Aliases are resolution-only: they are NOT separate metadata records, so the agent list still shows each agent exactly once under its canonical name."

    ⇒ The premise reading 2 rests on — that the cloud plugin registers under metadata_assistant and re-pinning would be a behaviour change — is false. The record's id is build. Studio's current pin reaches it by taking the back-compat detour.

    'build' also resolves on the UI side — it is the friendly-first member

    objectui/packages/plugin-chatbot/src/agentAliases.ts:23-26 groups the names bidirectionally, canonical first:

    export const AGENT_ALIAS_GROUPS = [['build', 'metadata_assistant'], ['ask', 'data_chat']];

    objectui/packages/app-shell/src/hooks/surfaceAgent.ts:82 bounds app.defaultAgent through isBuiltinAgentName, which is group membership — so both spellings are accepted identically, then resolved against whichever the live catalog serves. And :51-54 already has SURFACE_DEFAULT['studio-build'] = 'build': the Studio surface wants build with or without the pin.

    The re-pin is not cosmetic — it drops a load-order dependency

    'metadata_assistant' only resolves if registerAgentAlias has actually run. Cloud carries two separate defensive docblocks about that registration silently no-op'ing for real:

    • service-ai-studio/src/plugin.ts:44-57 — the alias must be registered with the string constant, never METADATA_ASSISTANT_AGENT.name, because under bundle load ordering / a circular import .name reads undefined, the legacy && canonical guard no-ops, "which drops build from both the legacy alias AND the framework's listAgents catalog". Re-asserted inside init() as belt-and-suspenders.
    • service-ai/src/agent-runtime.ts:30-41 — catalog membership was deliberately moved onto an intrinsic persisted signal so it does "NOT depend on an in-memory registerAgentAlias call having run (a missed alias must never hide a real platform agent like build)".

    defaultAgent: 'build' never touches the alias table. So reading 1 is strictly more robust than the status quo, on the platform's own flagship authoring surface — not merely tidier.

    A third in-repo site the census missed

    The card's census found one app.defaultAgent usage. Correct — but metadata_assistant is taught as author-facing example vocabulary at a second live site, and it is the louder one because it is served to LLM clients at runtime:

    packages/mcp/src/mcp-server-runtime.ts:1349 — the agent_prompt prompt's agentName argument description:

    agentName: z.string().describe('Name of the agent to load (e.g. "data_chat", "metadata_assistant")'),

    Both examples are retired aliases; neither canonical id appears. Every MCP client asking what to pass is told the two spellings the skill forbids. This site is uncontroversial under either reading — the aliases resolve, so nothing breaks, but the suggested spelling should be "ask" / "build".

    Two corrections to the suggested triage inputs

    1. No pin test guards the value. grep defaultAgent packages/platform-objects finds only studio.app.ts:48,50 — the comment and the value. Reading 1's first limb is a one-line change plus its comment; there is no assertion to update.
    2. The retired-keys registry cannot carry this residue. packages/spec/src/migrations/entries/retired-keys/ (76 entries) is keyed <domain>/<Type>:<key> — e.g. api/AuthFeaturesConfig:magicLink. It records retired keys. metadata_assistant is a retired value of a live key, which that registry has no shape for. The residue has to be recorded where it is read instead — at the pin site and in the lint docblock — or the ledger needs a value-level shape, which is its own card.

    What is left for the maintainer

    Reading 2 is gone and A's evidential blocker with it. Under A the published skill text does not change, so the "skills changes are ADR-class" limb is also moot. The one limb that genuinely remains above the seat is the #6041 half: whether the lint's defaultAgent value limb narrows its roster to ask / build (with legacy aliases warned under their own rule id and wording), while the declaration-shadowing limb keeps recognising all four. That refines which roster the value limb reads; it keeps both of the #6041 ruling's operative decisions intact — warning tier, and no Zod enum narrowing. But it is still a change to a gate settled by a standing ruling (2026-08-07, reaffirmed 08-09), so it is being put to the maintainer rather than assumed.


    Generated by Claude Code

  7. hotlong commented on Sep 3, 2026

    @hotlong
    Contributor

    Ruling: A (full). Maintainer, 2026-09-03 — the skill text stands unchanged; the two code sites move, and the third one the census missed moves with them.

    PR: #14780

    • studio.app.ts:50 → 'build' (+ comment recording the measurement and the load-order reason).
    • mcp-server-runtime.ts:1349 → the agentName example now names "ask" / "build" instead of the two retired aliases.
    • validate-ai-agent-authoring → the value limb takes ask / build; a legacy alias warns under a new exported rule id default-agent-legacy-alias with its own wording (an alias resolves, an unknown name does not — collapsing them would misdescribe both). The declaration limb still reads all four names; that judgement is a different question and is untouched. [决策] 是否给 app.defaultAgent 的取值加 lint(平台 agent 名单外即 warning)—— #5985 实测门禁对该类缺陷结构性失明 #6041's two operative decisions — warning tier, no Zod enum — both stand.

    Residue disposition: the retired-keys registry can't carry it (it is keyed on retired keys; this is a retired value of a live key), so it is recorded at the pin site, in the rule's docblock, and in tests that pin the alias/unknown distinction. A value-level ledger shape would be its own card.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

domain:skillsDomain lane — seat card indexed by label:pm:seatneeds-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PRpriority:p3

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions