lint: no reference-integrity for AI metadata (agent skills, skill tools) — R7, blocked on D1/D2

Split out of #3583, which is otherwise closed: eight of its nine bug categories are now gated on validate/lint/compile, and the ninth (hook body write sets) is an accepted, documented gap. This is the one category with resolvable references and no rule — assessment R7 in docs/audits/2026-07-app-metadata-reference-integrity-assessment.md §3.

The gap

packages/lint/src contains zero AI rules. Nothing checks that:

agent.skills[] names a skill the stack declares (ai/agent.zod.ts)
skill.tools[] — wildcard-aware — names a declared tool (ai/skill.zod.ts)
agent.tools[].name resolves to an action/flow by its declared type

Evidence on the real corpus (2026-07-28, hotcrm @ `8b28fa2`, v2.2.2)

Surface	Count
agents	2
skills	6
`agent.skills` references	8
`skill.tools` references	16
tools declared by the stack	0

So all 16 skill→tool references are dead today — the original audit's "agent skills listing 11 tools with no definitions", still shipping. Every one passes objectstack validate and objectstack lint cleanly.

Two decisions come first (neither is a lint question)

D2 — runtime-registered kernel skills/tools vs. static lint. The ask/build kernel agents register their skills/tools via service at boot, so a stack can legitimately reference names no static analysis can see (same invisibility as plugin objects, which feat(lint): reference-integrity validation for object and action names (#3583) #3657 solved with the curated PLATFORM_PROVIDED_OBJECT_NAMES registry). Either extend that precedent with official skill/tool names, or run R7 at advisory severity. Small; take it when the rule starts.
D1 — agent.knowledge.indexes has no definition site. KnowledgeSourceSchema is never referenced by stack.zod.ts, so the namespace is unresolvable by construction — parsed, unmarked, and unenforceable, which is the state ADR-0049's trichotomy prohibits. Either add a knowledgeSources slot to the stack (spec change, ADR-worthy) or mark the fields [EXPERIMENTAL — not enforced]. Linting a namespace with no definition site would institutionalise the gap, so the knowledge half of R7 stays blocked until this lands. HotCRM carries 2 such references.

Out of scope

The "hand-off to a nonexistent skill" instance from the original audit: hand-off has no schema field anywhere — it lives in free-form prompt text, which has no structure to check (§7 non-goal).

Done looks like

validate-ai-references in @objectstack/lint, appended to REFERENCE_INTEGRITY_RULES (one line — validate/lint/compile pick it up automatically), with fixtures from the HotCRM instances and a zero-false-positive pass over examples/. Note the verification lesson from #3806: the false-positive floor must be read per branch — an example corpus that declares no agents proves nothing about the agent branch.

Refs #3583 · ADR-0049 · ADR-0072 · ADR-0078

refactor(lint): one reference-integrity entry point + the HotCRM resi…

Positioning check before D1/D2: the agent branch lints a surface ADR-0063 closed, and the tool branch has no registry to resolve against

Read of the three repos at today's HEAD (objectstack @ 87aca93, cloud, hotcrm). The conclusion is that R7 should not be built as specced — not because the references don't matter, but because two of its three branches are checking namespaces whose authoring status is itself unresolved. There is a decision upstream of D1/D2.

1. Agents are already a closed surface — but only half-closed, and not at the authoring layer

ADR-0063 §2 decided it: "third parties author *.skill.ts and tools / MCP connectors. *.agent.ts is closed to third parties", and ADR-0040 §3 (tenant custom agents) is withdrawn. That landed in two places:

spec/src/kernel/metadata-plugin.zod.ts:697 — agent carries allowRuntimeCreate: false, allowOrgOverride: false.
cloud/packages/service-ai/src/agent-runtime.ts:199-208 — listAgents() filters the catalog to platform-owned records.

It did not land at the authoring layer, and the withdrawal has holes:

	state
`stack.agents` slot	still open — `spec/src/stack.zod.ts:303`
`stack.zod.ts:295-298` doc	still describes the withdrawn ADR-0040 model ("Persona-bearing copilots (1-3 per app)… the active app's `defaultAgent` is selected automatically")
`app.defaultAgent`	still in spec (`ui/app.zod.ts:675`), still an editable field in `ui/app.form.ts:39`, and its doc example is literally `defaultAgent: 'sales_copilot'` — the withdrawn pattern
`loadAgent()` (chat route)	no platform filter — `agent-runtime.ts:244`
`resolveDefaultAgent()`	reads `app.defaultAgent` and loads it unfiltered — `agent-runtime.ts:496-502`

So a stack-authored agent today is hidden from the picker but still reachable — via POST /agents/:name/chat and via app.defaultAgent. HotCRM's two agents are exactly that state: parsed, built into the artifact, invisible, reachable.

This is the ADR-0078 prohibited state, and the remedy ADR-0078 prescribes is not a lint rule about the slot's contents — it is closing or marking the slot. Writing validate-ai-references over agent.skills[] would institutionalise an authoring surface the platform decided not to have — the same objection the issue itself raises for the knowledge half in D1.

Corroborating: on the motivating corpus the agent branch finds nothing. HotCRM's 8 agent.skills references (sales_copilot 5, service_copilot 3) all resolve against its 6 declared skills. 8/8 clean.

2. `stack.tools` has zero readers — the tool half cannot resolve because there is no tool authoring story yet

Searched both repos for any read of the tool metadata type:

grep -rE "(list|get|register|load)\(\s*'tool'" objectstack/packages cloud/packages
→ cloud/packages/service-ai-studio/src/plugin.ts:538   register('tool', …)   ← write only

One write, no reads. The plugin's own comment says it: "Studio visibility: persist tool/agent/skill metadata… nothing subscribes to tool/agent/skill events" (plugin.ts:518-524). The executable path is the in-process ToolRegistry (aiService.toolRegistry.getAll(), routes/agent-routes.ts:582), which the metadata service never feeds.

Consequences:

A tool declared in an app stack registers a record, never a callable — ToolSchema is pure declaration with no handler binding. "Third parties author tools" (ADR-0063 §2) is not currently possible; today the real third-party primitive is skills alone, and a skill's tools[] can only name runtime-registered tools it does not own.
'tools' is missing from CONCAT_ARRAY_FIELDS (stack.zod.ts:1161-1188, which does list 'agents' and 'skills'), so composeStacks silently drops it — consistent with having no consumer.
plugin.ts:527-531 skips registration if (exists), so a tenant-declared tool of the same name would suppress the platform's own metadata mirror. Shared namespace, no prefixing.

3. Re-measuring the corpus: the "all 16 dead" figure is an artifact of resolving against the wrong registry

The issue reads 16 dead skill.tools references because the stack declares 0 tools. Resolved against the registry that actually decides the call — the in-process one — the split is:

	refs
resolve today (`describe_object`, `list_objects`, `query_records`, `get_record`, `aggregate_data`, `search_knowledge`)	6
genuinely fictional (`triage_case`, `generate_email_copy`, `optimize_subject_line`, `personalize_content`, `generate_email`, `analyze_lead`, `suggest_next_action`, `analyze_pipeline`, `identify_at_risk`, `forecast_revenue`)	10

A rule resolving against stack.tools alone would therefore emit 6 false positives out of 16 (37.5%) on its own motivating corpus — a rate that gets a rule disabled rather than fixed. And the runtime treats the miss as tolerable by design: "Tools that are declared by a skill but missing from the available tool registry are silently dropped — this is intentional so a skill can be authored before all its underlying tools are registered" (skill-registry.ts:196-199, same wording at agent-runtime.ts:376-379).

Also worth noting for the verification plan: examples/ declares zero agents, skills and tools (grep -rl "defineSkill|defineAgent|defineTool" examples/ → 0 files). Per the #3806 lesson the issue already cites, the false-positive floor for R7 is unmeasurable on examples/ — every branch, not just the agent one.

4. What is actually worth building — and it is not in R7's list

resolveActiveSkills() throws on a skill↔agent surface mismatch (agent-runtime.ts:468-477, ADR-0064 §3). That is a statically decidable, runtime-fatal condition surfacing as a 500 at chat time. Zero-false-positive, cheap, and it protects the invariant that keeps ask from authoring. This is the AI rule with the best evidence behind it, and the issue does not mention it.

Two adjacent findings from the same read, offered as separate work:

ADR-0064's "no global fall-through" is not enforced at the agent.tools[] seam. buildRequestOptions() resolves agent.tools[].name against the full registry passed in (agent-runtime.ts:409-424), with no surface check — while the route passes toolRegistry.getAll(), which includes AI Studio's authoring tools when that plugin is loaded. Combined with §1's unfiltered loadAgent/app.defaultAgent, a stack-authored surface:'ask' agent can name build-surface tools. Whether that is exploitable depends on the per-tool authz interceptors, which I did not audit — but the scoping invariant ADR-0064 states is not enforced here.
skill.permissions is silently stripped. HotCRM's live_data skill declares permissions: ['crm:read'] (hotcrm/src/skills/live-data.skill.ts); SkillSchema has no such field, so Zod drops it. A permission-shaped declaration that does nothing — ADR-0049 territory.

5. Proposed disposition

Insert D0 ahead of D1/D2: decide whether tool is a third-party-authorable metadata type or in-process registration only. Until that is answered, stack.tools is inert metadata and no reference rule over it can be correct in either direction. It is also the decision that determines whether ADR-0063 §2's "skills and tools" is a shipped contract or an aspiration.

Then, in dependency order:

Finish the ADR-0063 withdrawal instead of linting around it. Close stack.agents to app packages, or mark it [EXPERIMENTAL — INTERNAL] per ADR-0078; fix the unfiltered loadAgent/resolveDefaultAgent paths; retire or re-document app.defaultAgent and the stale stack.zod.ts:295-298 prose. HotCRM's two agents then get deleted or become legitimate — either way the R7 agent branch has nothing left to check.
Ship the surface-affinity rule (§4) — it stands on its own regardless of D0/D1/D2.
R7's skill.tools branch, only after D0, at advisory severity (S2) and resolving against a published tool-name registry. Feasible: the platform already publishes its tool names to the metadata service at boot (plugin.ts:538), so the D2 choice is between snapshotting that into a PLATFORM_PROVIDED_TOOL_NAMES set (the PLATFORM_PROVIDED_OBJECT_NAMES / feat(lint): reference-integrity validation for object and action names (#3583) #3657 precedent) or staying advisory. Without one of those, the 6/16 false-positive rate above stands.
D1 — agree with the issue, take the marker, not the slot. Add [EXPERIMENTAL — not enforced] to agent.knowledge.indexes/sources. Adding a knowledgeSources stack slot would create a second namespace with no reader, i.e. the §2 problem again.

Net: R7 as scoped is ~M of work that would find 10 real problems and 6 false ones, over two namespaces whose authoring status the platform has not settled. Settling D0 and finishing the §1 withdrawal is smaller, removes the false positives by construction, and is a prerequisite either way.

Generated by Claude Code

chore: changeset + regenerate app reference for defaultAgent doc (#3820)

Disposition update (per maintainer direction): the knowledge half (D1) is deferred entirely — not a current priority, no marker work either. The rest of the assessment above is now in flight as three PRs:

feat(lint,spec): AI surface-affinity rule, ADR-0063 doc completion, ADR-0109 draft (#3820) #3871 (this repo) — validate-ai-surface-affinity in REFERENCE_INTEGRITY_RULES (the statically provable, runtime-fatal AI check; zero FP by construction), the ADR-0063 doc-truth fixes (stack.agents marked platform-internal, app.defaultAgent re-documented as a surface-binding knob, SkillSchema documents that per-skill permissions doesn't exist), and ADR-0109 (Proposed) — the D0 decision this issue is blocked on: third-party tools are bindings to actions/flows; platform tool names become a conformance-tested registry. CI green.
objectstack-ai/cloud#904 — closes the "hidden from the picker but still reachable" hole: loadAgent() now carries the same platform-ownership gate as listAgents(), so a withdrawn tenant agent 404s on direct chat and app.defaultAgent falls back to ask.
fix(skills): drop the per-skill permissions keys — the field does not exist in SkillSchema hotcrm#511 — deletes the six dead skill.permissions keys (ADR-0049 shape).

R7's remaining buildable scope after this: the skill.tools branch only, blocked on ADR-0109 acceptance (which replaces D0/D2 here), warning-first per the ADR-0078 ratchet.

Generated by Claude Code

feat(lint,spec): AI surface-affinity rule, ADR-0063 doc completion, A…

feat(spec,lint): ADR-0109 revised + Phase 1 — skills-only default pat…

feat(spec,lint)!: accept ADR-0109, remove agent.tools[], lint agent a…

fix(spec): tombstone agent.tools instead of deleting it — unbreak main (

Batch complete — 10 PRs merged across three repos

objectstack

feat(lint,spec): AI surface-affinity rule, ADR-0063 doc completion, ADR-0109 draft (#3820) #3871 — validate-ai-surface-affinity rule; ADR-0063 doc completion (stack.agents marked platform-internal, app.defaultAgent re-documented as a surface-binding knob); ADR-0109 drafted
feat(spec,lint): ADR-0109 revised + Phase 1 — skills-only default path, tool-name registry, advisory skill.tools lint (#3820 R7) #3885 — ADR-0109 revised + Phase 1: PLATFORM_PROVIDED_TOOL_NAMES registry, validate-ai-tool-references (advisory), composeStacks stops dropping tools
feat(spec,lint)!: accept ADR-0109, remove agent.tools[], lint agent authoring, model AI exposure in tool resolution (#3820) #3894 — ADR-0109 accepted; agent.tools[] removed; validate-ai-agent-authoring rule; exposure-aware action_* resolution; docs teach the Action → skill path
fix(spec): tombstone agent.tools instead of deleting it — unbreak main #3904 — follow-up fix, see below

cloud

🐛 [Bug] pnpm dev 启动失败：plugin.app.dev-workspace failed to start — 三重问题合并修复 #904 — loadAgent / resolveDefaultAgent platform filtering
feat: promote i18n to core built-in service with in-memory fallback #908 — PLATFORM_PROVIDED_TOOL_NAMES conformance suites
i18n 路由应由 service-i18n 自注册，RestServer 应移除统一注册职责，保持架构一致性 #910 — runtime stops reading agent.tools
fix: move i18n routes from RestServer to service-i18n self-registration + dispatcher bridge #911 — framework pin → 11949fc

hotcrm

Add comprehensive strategic analysis reports for business model and product design #511 — dead skill.permissions key removed
[WIP] Add detailed cloud project implementation plan #512 — ported to the ADR-0109 model: 16 tool references (10 dead) → 22 references, 0 findings

Disposition

R7 delivered — surface-affinity at error severity (statically provable, runtime-fatal), skill.tools at advisory per the ADR-0078 ratchet. The 37.5% false-positive rate that originally blocked this branch is gone by construction: on the HotCRM corpus the rule yields exactly the 10 fictional references and 0 false positives.
D0 answered by ADR-0109 — Accepted, Phase 1 implemented. The default third-party path needs no tool records at all; a skill's tools[] names a platform-registered tool or a materialised action_<name>. Tool records are demoted to an optional AI-presentation refinement layer (Phase 2, unscheduled).
D1 (knowledge half) deferred per maintainer direction on this thread.

One correction worth recording

#3894 removed agent.tools and AIToolSchema outright. Its 16 checks were green, but that was a turbo cache restore on Build Core — a cold build of main failed, because the authorable-surface ratchet (ADR-0104 / #3733) rejects an authorable key disappearing without a tombstone. It was right to: these schemas are not .strict(), so Zod silently strips an unknown key, and an author who kept writing tools: would have gotten a clean parse and an agent reaching none of the tools they listed — the same silent-capability-loss shape this issue exists to eliminate, reintroduced one layer down by the PR meant to eliminate it.

#3904 fixes it properly: agent.tools is now retiredKey() (authoring it throws with the fix in the message), plus a D2 conversion agent-tools-to-skills and its D3 chain step so the removal reaches spec-changes.json and os migrate meta. Unlike the protocol-17 renames beside it, this one has no lossless target — each entry must become a reference inside a skill, a human decision — so the conversion drops the dead key and emits a notice per agent rather than guessing a destination. main builds cold again.

Two things that follow from it and are worth a maintainer's attention independently:

Build-time gates can be skipped entirely by a turbo cache hit. That is not specific to this ratchet.
feat(spec,lint)!: accept ADR-0109, remove agent.tools[], lint agent authoring, model AI exposure in tool resolution (#3820) #3894's changeset claimed authoring tools "remains a silent no-op rather than a parse error." That is now false and fix(spec): tombstone agent.tools instead of deleting it — unbreak main #3904's changeset supersedes it.

Open findings — maintainer decisions, not cleanups

1. ASK_AGENT.skills is a hardcoded allow-list, so third-party skills cannot attach to ask. ADR-0063 §2 names skills the only third-party extension primitive, but there is no path by which a third-party skill reaches the ask agent:

ask-agent.ts:82 — skills: ['schema_reader', 'data_explorer', 'actions_executor', 'builder_handoff']
skill-registry.ts:128-138 — listActiveSkills(context, restrictTo) intersects the registry with that array (if (allowList && !allowList.has(skill.name)) return false), and agent-runtime.ts:467 passes agent.skills as restrictTo
ask-agent.ts:105-108 — protection: { lock: 'full' }, so tenants cannot edit the persona to add a name

A third-party surface: 'ask' skill registers successfully, passes validation, and is silently never activated — it does not even hit the ADR-0064 §3 affinity error, since that check only runs on skills that already cleared the allow-list. The comment directly above the lock states the promise this blocks: "Tenants extend the platform with skills + tools, never by editing this persona." Three possible fixes (drop the intersection when skills[] is absent; a skill-side attachTo; a persona extraSkills override) all move the ADR-0063/0064 boundary, so this needs a call before implementation. Filing separately against cloud.

2. Nothing in the open distribution consumes skills at all — filed as #3905. The open framework is deliberately MCP-only (cloud ADR-0025), which is fine; the gap is that skill / tool / mcp metadata still ships in the open spec, is authorable, is actively linted by the rules added in this batch, and reaches no runtime there.

Both are pre-existing and neither blocks this batch.

Generated by Claude Code

关闭核验(devx 车道 PM,session_01GX3sL71LFq8m2usg6VqTSE,2026-08-05):本单交付物已全部在 main 上 —— packages/lint/src/validate-ai-surface-affinity* 与 validate-ai-tool-references*(含测试)存在且接入 reference-integrity-suite.ts(本 PM 对 origin/main 实测命中);07-28 批次评论记录 10 个 PR 合并、D0/D2 由 ADR-0109(Accepted)回答、R7 的 skill.tools 分支按 ADR-0078 棘轮以 advisory 落地。D1(knowledge 半边)维护者已裁定整体搁置(2026-07-28 评论在案),不留在本单——届时重启是新立项。

按发现分诊纪律关闭(completed)。维护者可否决重开。

Generated by Claude Code

test(spec): security, ai, identity, integration, migrations, marketpl…

Navigation Menu

Repository navigation

lint: no reference-integrity for AI metadata (agent skills, skill tools) — R7, blocked on D1/D2 #3820

The gap

Evidence on the real corpus (2026-07-28, hotcrm @ `8b28fa2`, v2.2.2)

Two decisions come first (neither is a lint question)

Out of scope

Done looks like

Positioning check before D1/D2: the agent branch lints a surface ADR-0063 closed, and the tool branch has no registry to resolve against

1. Agents are already a closed surface — but only half-closed, and not at the authoring layer

2. `stack.tools` has zero readers — the tool half cannot resolve because there is no tool authoring story yet

3. Re-measuring the corpus: the "all 16 dead" figure is an artifact of resolving against the wrong registry

4. What is actually worth building — and it is not in R7's list

5. Proposed disposition

Batch complete — 10 PRs merged across three repos

Disposition

One correction worth recording

Open findings — maintainer decisions, not cleanups

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

lint: no reference-integrity for AI metadata (agent skills, skill tools) — R7, blocked on D1/D2 #3820

Description

The gap

Evidence on the real corpus (2026-07-28, hotcrm @ 8b28fa2, v2.2.2)

Two decisions come first (neither is a lint question)

Out of scope

Done looks like

Activity

os-zhuang commented on Jul 28, 2026

Positioning check before D1/D2: the agent branch lints a surface ADR-0063 closed, and the tool branch has no registry to resolve against

1. Agents are already a closed surface — but only half-closed, and not at the authoring layer

2. stack.tools has zero readers — the tool half cannot resolve because there is no tool authoring story yet

3. Re-measuring the corpus: the "all 16 dead" figure is an artifact of resolving against the wrong registry

4. What is actually worth building — and it is not in R7's list

5. Proposed disposition

os-zhuang commented on Jul 28, 2026

os-zhuang commented on Jul 28, 2026

Batch complete — 10 PRs merged across three repos

Disposition

One correction worth recording

Open findings — maintainer decisions, not cleanups

os-zhuang commented on Aug 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Evidence on the real corpus (2026-07-28, hotcrm @ `8b28fa2`, v2.2.2)

2. `stack.tools` has zero readers — the tool half cannot resolve because there is no tool authoring story yet