Six more skills/objectui/evals assertions grade tokens their own guide never teaches, and one passes only by accidental substring match

Filed out-of-scope from the #7251 skills-optimization flight, which corrected three evals of this class (OUI-H-01, H-03, H-04) and left these. Unassigned.

Every must_contain entry is a claim that a correct answer will contain that literal substring. When the token appears nowhere in the guide the eval is scoring, the eval grades a decision the package never taught: a correct answer fails, and an answer that invents the token passes. That is the failure mode #7251 fixed in evals/mobile.json, where the eval demanded useSwipe / useLongPress — zero occurrences repo-wide.

Remaining rows

Untaught tokens (OUI-H-05). Four assertions require tokens absent from the guide they score:

Eval	Token	Occurrences in its guide
`evals/mobile.json` eval 1	`sidebar`	0 in `guides/mobile.md`
`evals/project-setup.json` eval 3	`ComponentRegistry`	0 in `guides/project-setup.md`
`evals/testing.json` eval 2	`edit`	0 in `guides/testing.md`
`evals/page-builder.json` eval 3	`refresh`	0 in `guides/page-builder.md`

(evals/page-builder.json eval 2 also requires section, which is likewise absent; #7251 removed the props token from that same assertion but left section alone, being out of its funded list.)

Untaught API, plus a substring accident (OUI-H-02). Three assertions across evals/plugin-development.json and evals/page-builder.json require registerComponent. The API is real — PluginScope.registerComponent(type, component, meta?), documented in its own docblock at packages/types/src/plugin-scope.ts — but every guide teaches registration only as ComponentRegistry.register(...), so the package's own eval contradicts its own guide. Two of those evals also require FieldWidgetProps, whereas the guide and the implementation both use FieldWidgetComponentProps; the assertion passes only because FieldWidgetPropsSchema happens to contain the shorter string.

A second substring accident, found while working #7251 and not in the audit. evals/mobile.json eval 2 requires useOffline. packages/mobile/src/index.ts exports useOfflineSync, not useOffline — the assertion passes on the substring, and would go on passing if the real hook were renamed to something else entirely that happened to start the same way.

Two ways to close it, and they are not equivalent

Fix the assertion where the eval is testing a real decision with the wrong token — the FieldWidgetProps → FieldWidgetComponentProps and useOffline → useOfflineSync rows are this, and are mechanical.
Teach the thing where the token names a real surface the package should cover. registerComponent is this: it is the API a plugin author actually gets, and guides/plugin-development.md never mentions it. Adding a short row there makes three evals honest at once.

Deciding which per row is the work; a blanket delete of the tokens would lose case 2.

The structural half

None of this is catchable today: nothing validates an eval fixture against the guide it names, and evals/*.json is checked by no gate at all. A ~30-line check that every must_contain token occurs somewhere in skills/objectui/** would have caught every row above, including the two substring accidents if it compared whole tokens. Worth considering alongside #7359.

Claim: PM loop round 5 (graded by the skills lane seat as a follow-up of its own programme card #7251 — priority:p2, pm:dispatched, assignee and needs:contract-review set in the same stroke)
Session: session_01LraLgQVGq8egUwfYZpbYt1
Branch: claude/issue-7360-evals-untaught-tokens
Worktree: objectui-issue-7360
Domain: domain:skills
Scope, exactly the card's two-way close: fix the assertion where the eval tests a real decision with the wrong token (FieldWidgetProps → FieldWidgetComponentProps; useOffline → useOfflineSync; the four untaught tokens sidebar / ComponentRegistry / edit / refresh and section — each either re-pointed at a token the guide does teach or dropped, per row, with the guide's own text as the oracle), and teach the thing for registerComponent: one short row in guides/plugin-development.md naming PluginScope.registerComponent(type, component, meta?) beside ComponentRegistry.register(...), verified at packages/types/src/plugin-scope.ts. ⛔ No blanket token deletion; ⛔ no new section; the structural gate (a must_contain ↔ guide check) is #7359's, not this card's.
File surface: skills/objectui/evals/{mobile,project-setup,testing,page-builder,plugin-development}.json and guides/plugin-development.md (one row). Token ratchet: objectui has no skills ratchet (measured in #7251); still shrink-or-neutral per file, additions paid in the same file.
Container & model: S, mode:subagent, model: opus; floor sonnet · default opus · ceiling fable.
Clause-②: yes — the registerComponent row states a plugin-API contract; needs:contract-review on this card and on the PR; the seat reviews in-seat at CONTRACT_REVIEW_TIER.
Serial constraints: no open PR touches skills/objectui/** (#7370 merged 14:37Z). Heavy slots: 4 of 5 after this dispatch.
Landing form: ONE draft PR against main, first line Fixes #7360; changeset per check-changeset-presence's own verdict (expected: none owed; ⛔ never request skip-changeset here — objectui has no such mechanism); governed (skills/**, ui#6866 批 #7) ⇒ stays draft, human merge; review requests are the seat's step.

Generated by Claude Code

Dev claim (implementing seat, same session as the dispatch claim above).

Session: session_01LraLgQVGq8egUwfYZpbYt1
Branch: claude/issue-7360-evals-untaught-tokens
Worktree: objectui-issue-7360, cut from origin/main@0614b6d

File surface, as dispatched and unchanged: skills/objectui/evals/{mobile,page-builder,plugin-development,project-setup,testing}.json and skills/objectui/guides/plugin-development.md (one bullet, paid in the same file). Six commits, one per eval file plus one for the guide, each individually droppable. No file under packages/ touched; the structural gate stays #7359's.

One row of the card's list is skipped as premise_false rather than edited — useOffline in evals/mobile.json eval 2 is taught by its guide and exported by the package the guide names. Evidence in the PR body.

Generated by Claude Code

os-dev-report

{
  "issue": 7360,
  "status": "done",
  "branch": "claude/issue-7360-evals-untaught-tokens",
  "pr": "https://github.com/objectstack-ai/objectui/pull/7408",
  "premise_still_valid": true,
  "summary": "Seven funded rows closed per row against the guide each eval actually scores, in six droppable commits at f6c2f1b. Re-pointed: FieldWidgetProps to FieldWidgetComponentProps in evals/plugin-development.json evals 1 and 3 (the short name counted 1 in the guide, and only as the head of FieldWidgetPropsSchema); sidebar to ResponsiveContainer in evals/mobile.json eval 1 (0 to 4 occurrences); section to card in evals/page-builder.json eval 2 (0 to 4). Dropped, nothing in the guide corresponding: ComponentRegistry from evals/project-setup.json eval 3, edit from evals/testing.json eval 2, refresh from evals/page-builder.json eval 3 - each verified real at source but taught in a different package or guide, and each assertion still grading its decision through the tokens that stay. Taught: one bullet in the Namespace system section of guides/plugin-development.md naming scope.registerComponent(type, component, meta?) beside ComponentRegistry.register, signature verified at packages/types/src/plugin-scope.ts:54, paid in the same file (14335 to 14325 bytes, 3584 to 3582 tokens) by deleting comments the ComponentMeta full-reference table two lines below already restates. One funded row was NOT edited: useOffline is real and taught, detailed under premise_false below. PR is DRAFT and stays draft - governed surface, check-governed-queue-guard --test returns exit 3. Label needs:contract-review written as a union over an observed-empty set and read back present. No file under packages/ touched; the structural gate remains #7359's.",
  "tests": "All at head f6c2f1b, exit codes captured before any pipe, each quoted from the gate's own verdict line. node scripts/check-skills-paths.mjs EXIT=0 'check-skills-paths: OK (28/28 stated path(s) resolve across 16 guide file(s))' and --list confirms the newly stated packages/types/src/plugin-scope.ts is among the 28. node scripts/check-doc-links.mjs EXIT=0 'Links are valid across 17 scan roots.' node scripts/check-control-bytes.mjs EXIT=0 'OK (scanned 6103 tracked text file(s); skipped 85 binary)'. node scripts/check-shell-escape-residue.mjs EXIT=0 'OK (4/4 root(s) resolved ... skills: 16 file(s), 196 fence(s))'. node scripts/check-changeset-presence.mjs EXIT=0 'No source or published contract of a released package changed in this range, so no changeset is owed' - its own verdict, so none added and no label requested. node scripts/check-governed-queue-guard.mjs --self-test EXIT=0 '132 cases pass'; --test over the six changed paths EXIT=3 'One governed path governs the WHOLE pull request' = GOVERNED, the correct verdict. pnpm exec vitest run over four files EXIT=0 'Test Files 4 passed (4)' / 'Tests 91 passed (91)' - run under the shared verify lock (VERDICT command-exit 0, held 17s, waited 154s, slot objectui-7360-dev). pnpm type-check:scripts EXIT=0, no diagnostics. JSON.parse on all 11 evals/*.json: all parse. TWO DISPATCH PREMISES MEASURED FALSE, both adding gates: (1) 'evals/*.json is read by no gate or test' - the stated grep does return nothing, but packages/components/src/__tests__/skill-guide-provider-envelope.test.tsx:248 globs the published tree, filters /evals/, and asserts over every must_contain array that no entry matches /^props./ with a counter-probe against vacuous passing; added to the run and green. (2) 'guides/plugin-development.md is read by no executable-fixture test' - correct as to fixtures, but it is pinned twice in scripts/__tests__/doc-version-claims.test.ts (KNOWN_CLAIMS lines 1167 and 1173), which is why the added bullet deliberately states no version literal; that test was already dispatched and is green. Also added and green: skill-guide-data-table-binding.test.tsx. NOT RUN, narrowing proven rather than omitted: pnpm lint (eslint . --no-inline-config) is repo-wide and CI-owned; asked about the two edited file types, eslint's own configuration answers 'File ignored because no matching configuration was supplied' for both the edited .md and the edited .json (read from --format json; 2 files, 0 configured), so a config matching neither extension cannot move its verdict on any untouched file. No ablation was performed - none of the six edits is a guard whose failure needed demonstrating.",
  "mcp_calls": "7 - one search_issues (dedup, non-zero with issue 7360 itself as the in-session control hit), two issue_write create (7405, 7406), one issue_write update (PR labels), one create_pull_request, two add_issue_comment (claim on 7360, design input on 7359). This report comment makes 8. All card and PR reads went through the zero-quota public-repo payload channel.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #7405: three more assertions grade nothing - `create` and `view` in evals/testing.json eval 2 pass by accidental substring (a mocked `create: vi.fn()` dataSource method, and the comment `// vite preview`), which a grep -c == 0 audit cannot see by construction; `visible` in evals/page-builder.json eval 2 is untaught outright (0 in its guide, which teaches `hidden`). Not folded in: per-row disposition is the judgement this card funds by row and none of these rows was funded.",
    "filed as #7406: guides/page-builder.md states 'Five registered types do all the structural work' and omits `section`, which is registered at packages/components/src/renderers/layout/containers.tsx:988 and authored in examples/schema-catalog. This is why the `section` eval row was re-pointed at `card` rather than closed by teaching - teaching it is an addition to a second guide, outside this card.",
    "recorded as a comment on #7359 (not a new card): the structural check's oracle choice is load-bearing and the two candidates disagree on a live row - after this PR `registerComponent` counts 1 in guides/plugin-development.md but still 0 in guides/page-builder.md, so evals/page-builder.json eval 1 is honest bundle-wide and red per-guide. Same comment carries the whole-token-versus-substring evidence and a note that must_not_contain needs the opposite treatment."
  ]
}

premise_false detail, since the field above is a boolean and this row needs its evidence: useOffline in evals/mobile.json eval 2 was NOT edited. The card reads it as a substring accident because packages/mobile/src/index.ts exports useOfflineSync. True of @object-ui/mobile and irrelevant — the guide imports the hook from @object-ui/react, not from that package. At 0614b6d: packages/react/src/hooks/useOffline.ts:256 declares export function useOffline(config: OfflineConfig = {}): OfflineResult; it reaches the entry point through packages/react/src/hooks/index.ts:30 and packages/react/src/index.ts:11 (export * from './hooks/index.js'); and guides/mobile.md:123 teaches exactly import { useOffline } from '@object-ui/react';, destructuring { isOnline, pendingCount, syncState }, which matches the implementation. The token counts 2 in the guide it scores, not 0. useOfflineSync is a different hook in a different package. premise_still_valid is reported true because the card's other seven rows all held and the PR stands; this one row is the exception and no edit was forced.

Generated by Claude Code

Seat review — ACCEPT; contract review (clause ②) — PASS

PR #7408, head f6c2f1b, Fixes #7360. Verified on the branch head, not from the PR body:

Six files, +21/−23, one droppable commit each. The seven funded rows, per row: re-pointed FieldWidgetProps → FieldWidgetComponentProps (plugin-development evals 1 and 3 — the short name matched only as the head of FieldWidgetPropsSchema), sidebar → ResponsiveContainer (mobile eval 1; 0 → 4 in the guide), section → card (page-builder eval 2; 0 → 4); dropped ComponentRegistry (project-setup eval 3), edit (testing eval 2), refresh (page-builder eval 3) — each real elsewhere, none taught by the guide the eval scores, each assertion still grading its decision through the tokens that stay; taught scope.registerComponent(type, component, meta?) as one bullet in the Namespace system section, paid in the same file (14,335 → 14,325 B, −2 tok) by deleting comments the ComponentMeta table two lines below restates; left alone, correctly useOffline — the card's premise was wrong there: packages/react/src/hooks/useOffline.ts:256 is a real export of @object-ui/react and guides/mobile.md teaches exactly that import, twice.
Clause ② at CONTRACT_REVIEW_TIER (the served model read via get_session equals the constant): the one contract claim the PR adds — that scope.registerComponent applies the scope's own name, stores pluginName:type, and so takes no namespace — is true at source: packages/types/src/plugin-scope.ts:54 (signature), packages/core/src/registry/PluginScopeImpl.ts:104-110 (sets namespace: this.name on the registry meta) and :118 (reads ${this.name}:${type}); @object-ui/types base.ts declares zero namespace keys while core's Registry.ts:173 declares namespace?: string. PASS ⇒ under the 2026-08-31 ruling the same seat strips needs:contract-review from this card and from PR skills(objectui): evals grade what the guides teach — six assertions re-pointed or dropped, two substring accidents fixed, registerComponent taught (#7360) #7408.
Two dispatch premises measured false and both added gates rather than removed them (skill-guide-provider-envelope.test.tsx:248 globs every must_contain; doc-version-claims.test.ts pins the guide twice, which is why the bullet states no version literal) — accepted. eslint narrowing measured from the config (.md / .json not in its population).
CI: every check on f6c2f1b green by 18:29Z (Lint, Type Check, Test 4/4 shards, Build & E2E, the doc gates, Governed Surface Queue Guard, Changeset Declaration).

Out of scope, graded by the seat and queued behind this merge (same files): #7405 (p2) and #7406 (p2); the whole-token-vs-substring oracle note is on #7359.

Landing: governed skills/objectui/** ⇒ PR stays draft; review requested from os-zhuang + hotlong. The base has moved (behind); the human merge carries it. At merge the seat strips pm:dispatched.

Generated by Claude Code

fix(skills): testing eval 2 stops grading two accidental substring ma…

fix(skills): retract one false sentence from the previous commit message

feat(scripts): gate every eval must_contain token against the guides …

skills(objectui): evals grade what the guides teach — six assertions …

skills(objectui): two eval assertions stop grading accidental substri…

feat(scripts): gate every eval must_contain token against the guides …

Navigation Menu

Repository navigation

Six more `skills/objectui/evals` assertions grade tokens their own guide never teaches, and one passes only by accidental substring match #7360

Remaining rows

Two ways to close it, and they are not equivalent

The structural half

Seat review — ACCEPT; contract review (clause ②) — PASS

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Six more skills/objectui/evals assertions grade tokens their own guide never teaches, and one passes only by accidental substring match #7360

Description

Remaining rows

Two ways to close it, and they are not equivalent

The structural half

Activity

os-litant commented on Sep 2, 2026

os-litant commented on Sep 2, 2026

os-litant commented on Sep 2, 2026

os-litant commented on Sep 2, 2026

Seat review — ACCEPT; contract review (clause ②) — PASS

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Six more `skills/objectui/evals` assertions grade tokens their own guide never teaches, and one passes only by accidental substring match #7360