Repository navigation
QA run · area:ai (9/9) · 94114c1f · 2026-10-16 · 8 PASS / 0 PARTIAL / 1 FAIL / 0 BLOCKED / 0 NOT-RUN #23915
Description
Run record for the full-checklist wave anchored on #23878 (selector all, run per area). Runner session session_01K3qUoyh1xFGoWzAACPMbXJ, dispatched by the domain:devx seat 1 PM. Protocol: .claude/skills/checklist-test/SKILL.md + docs/qa/platform-checklist/RUNNER.md.
Environment fingerprint
- framework:
94114c1f269f6b3d885f24b6f4689dce29504574(origin/main at run start;pnpm install --frozen-lockfile && pnpm build, 75/75 green) - resolver:
node scripts/checklist-select.mjs allat that sha → 272 runnable · 2 blocked hidden · 1 planned .objectui-sha:daeb77fc84dd824e74c95b87cdede85afdb248e7. ⚠pnpm objectui:buildFAILS on this sha (Console pin does not build on main: objectui daeb77fc imports checkPageComponentViewData, deleted from spec by #23784 #23908). Browser/mixed verdicts ran on a console built at that same pin against@objectstack/specfroma35f7a67^(the commit before the deletion), served from a linked worktree at94114c1fwith identical framework dists.check:console-shamatches.- app: showcase · boot
OS_PORT=P objectstack dev --ui --seed-admin -p P -d file:…/data.db· per-runner port and DB · seeded admin + demo personas - browser: Playwright 1.63 on
/opt/pw-browsers/chromium(Chromium 141), one context per persona - timestamp: 2026-10-16 (UTC) · runners:
- G3 (ai+search): ports [41060,41061,41062,41063,41064,41065,41066,41067] · sequential · env 22 min / test 26 min
Scope
Selector area:ai (from the all resolution) · 9 runnable item(s):
ai.agent-tool-skill-metadata-roundtripr4 · P1 · api → passai.mcp-http-surfacer2 · P1 · api → passai.mcp-stdio-fail-closedr4 · P0 · cli → failai.mcp-run-action-exposure-gater2 · P1 · api → passai.mcp-validate-expressionr2 · P2 · api → passai.skill-instructions-mcp-promptsr1 · P1 · api → passai.open-edition-honest-degradationr1 · P2 · api → passai.console-ai-surface-gatingr4 · P2 · browser → passai.mcp-oauth-private-host-transportr3 · P1 · api → pass
Per-clause verdicts
| item | # | clause (short) | verdict | evidence |
|---|---|---|---|---|
| ai.agent-tool-skill-metadata-roundtrip | 0 | valid probes build; tombstones rejected loudly with prescriptions | pass | scratch-copy build of agent+tool+skill trio: Build complete exit 0; 10 tombstone builds exit 2, each names the key (tool.permissions "promised a capability gate…nothing ever enforced"; agent.tools prescribes skills+ADR-0 |
| ai.agent-tool-skill-metadata-roundtrip | 1 | registry serves agent/tool/skill singular | pass | GET /meta and /meta/types: 34 types, agent/tool/skill present, agents/tools/skills absent |
| ai.agent-tool-skill-metadata-roundtrip | 2 | authored items list + read back with defaults | pass | GET /meta/skill lists qa_probe_skill+3 prompt skills; get qa_probe_skill = authored fields + surface:ask, active:true; agent gets surface:ask active:true; tool field-identical |
| ai.agent-tool-skill-metadata-roundtrip | 3 | runtime write door matches registry posture (all sides) | pass | PUT skill+tool qa_runtime_* ?mode=draft 200, ?state=draft read 200; PUT package-shipped skill/tool 403 NOT_OVERRIDABLE; PUT agent (runtime name and shipped) 403 code-only (allowRuntimeCreate=false) |
| ai.agent-tool-skill-metadata-roundtrip | 4 | anonymous-deny on meta AI kinds | pass | unauthenticated GET /meta/agent, /meta/tool, /meta/skill all 401 UNAUTHENTICATED |
| ai.agent-tool-skill-metadata-roundtrip | 5 | tool metadata is read-only projection | pass | POST /ai/tools/qa_probe_tool/execute 501 NOT_IMPLEMENTED with the service-ai Cloud/EE remedy, with qa_probe_tool authored |
| ai.mcp-http-surface | 0 | opted-out boot: 404 on /mcp and /mcp/skill (not 501) | pass | OS_MCP_SERVER_ENABLED=false boot: GET+POST /mcp/skill and POST /mcp all 404 RESOURCE_NOT_FOUND "MCP server is not enabled for this environment" — the 404 gate refusal, not the 501 |
| ai.mcp-http-surface | 1 | GET /mcp/skill public SKILL.md headers + own URL | pass | anon 200 text/markdown; charset=utf-8, inline; filename="SKILL.md", no-store; body carries http://localhost:41060/api/v1/mcp |
| ai.mcp-http-surface | 2 | non-GET /mcp/skill 405 Allow: GET | pass | POST /mcp/skill 405 Allow: GET, METHOD_NOT_ALLOWED "Method not allowed — use GET" |
| ai.mcp-http-surface | 3 | anonymous /mcp 401 before any tool | pass | anon initialize 401 "Unauthorized: a valid OAuth access token or API key is required" + WWW-Authenticate resource_metadata=.../.well-known/oauth-protected-resource; anon tools/call 401 |
| ai.mcp-http-surface | 4 | keyed client initialize + tools/list full set | pass | osk_ key: init 200, tools/list = 11 documented tools incl aggregate_records, plus resume_run (registered in mcp-http-tools.ts:1299, newer than the item list) |
| ai.mcp-http-surface | 5 | scopeless OAuth 403 insufficient_scope; data:read narrows | pass | real auth-code+PKCE token, scope "openid profile": 403 "grants none of the MCP scopes (data:read, data:write, actions:execute)" + error="insufficient_scope" scope="data:read data:write actions:execute"; data:read token t |
| ai.mcp-stdio-fail-closed | 0 | enabled + keyless refuses, names var | pass | os start OS_MCP_STDIO_ENABLED=true no key: exit 1, "[MCP] The stdio transport is enabled … but OS_MCP_STDIO_API_KEY is not set … Refusing to start an unscoped stdio server (ADR-0101)"; health never 200 |
| ai.mcp-stdio-fail-closed | 1 | unknown key rejected up front | pass | OS_MCP_STDIO_API_KEY=osk_unknown: exit 1, "did not resolve to a valid identity (unknown / revoked / expired / owner-less). Refusing to start stdio (ADR-0101)." |
| ai.mcp-stdio-fail-closed | 2 | valid key principal-bound log line | pass | --log-level info: "[MCP] stdio transport principal-bound to OS_MCP_STDIO_API_KEY identity OkCFyhQm… (RLS/FLS/tenant applied)" = contributor member uid |
| ai.mcp-stdio-fail-closed | 3 | member MCP reads = member REST (RLS rows, FLS fields); admin full | pass | contributor: MCP query 3 own invoices = REST 3; get INV-1003 not found on both; admin MCP 15 + INV-1003. liaison: budget/spent absent on MCP query/get and REST, admin REST shows them |
| ai.mcp-stdio-fail-closed | 4 | member MCP aggregate = member REST, not admin | pass | count by status: MCP member [draft 3] = REST member [draft 3]; admin REST/MCP [draft 5, paid 5, sent 5] |
| ai.mcp-stdio-fail-closed | 5 | aggregate over FLS-masked field fails closed | pass | liaison sum(budget), groupBy budget, count_distinct(spent): MCP error PERMISSION_DENIED "[Security] Field read denied: not permitted to aggregate [budget/spent] on showcase_project", no number; control count by status + |
| ai.mcp-stdio-fail-closed | 6 | revocation honored on next read with the pinned message | fail | refusal holds (revoked mid-session ⇒ next query_records isError), but the text is API_KEY_REVOKED "This API key (osk_…) was revoked…", not "MCP stdio identity is no longer valid (key revoked or expired)"; reproduced on 2 |
| ai.mcp-stdio-fail-closed | 7 | stdio disabled ⇒ no key demanded | pass | os start no stdio env, --log-level info: clean boot, "[MCP] Transport not auto-started (HTTP is served per-request at /api/v1/mcp regardless)…" |
| ai.mcp-run-action-exposure-gate | 0 | undeclared action refused with prescription, no side effect | pass | run_action showcase_mark_done: "Action 'showcase_mark_done' on 'showcase_task' is not exposed to AI — the app author must opt it in with ai: { exposed: true, description: … }"; task status in_review + updated_at unchan |
| ai.mcp-run-action-exposure-gate | 1 | list_actions = exposed+permitted only | pass | list_actions: totalCount 1 = showcase_portfolio_snapshot; showcase_mark_done absent |
| ai.mcp-run-action-exposure-gate | 2 | exposed action runs, counts reconcile | pass | result {ok, scope global, accounts 14, projects 5, invoices 15} = REST totals 14/5/15 |
| ai.mcp-run-action-exposure-gate | 3 | trusted-elevation audit line | pass | boot log: "[action-audit] MCP run_action 'showcase_portfolio_snapshot' on 'global' — body executes TRUSTED (system-elevated context, RLS/FLS-bypassing) for user 'X7sRjx…'" (admin) |
| ai.mcp-run-action-exposure-gate | 4 | system objects refused | pass | run_action revoke_api_key: "is on a system object and is not exposed via MCP"; with objectName sys_api_key: "Object sys_api_key is a system object…"; key still revoked:false |
| ai.mcp-validate-expression | 0 | sound formula ok + inferredType + inScope | pass | record.estimate_hours * 1.1 formula: ok true, inferredType number, inScope cel, 18 fields incl estimate_hours, 37 functions |
| ai.mcp-validate-expression | 1 | error vs warning tiers | pass | bare estimate_hours: error bare-reference "Write record.estimate_hours"; record.estimate_hourz: error "did you mean estimate_hours?"; record.title * 2: ok true + WARNING type mismatch string * int |
| ai.mcp-validate-expression | 2 | site flips binding rules | pass | "estimate_hours > 8" flow_condition ok true; same under formula ok false bare-reference error |
| ai.mcp-validate-expression | 3 | unknown object + system object refused | pass | qa_no_such_obj: 'Object "qa_no_such_obj" not found'; sys_user: 'is a system object and is not exposed via MCP' (text only, no schema) |
| ai.skill-instructions-mcp-prompts | 0 | qa_prompt_full listed with name/title/description | pass | prompts capability declared; prompts/list has {name qa_prompt_full, title "QA Prompt Full", description "full prompt skill"} |
| ai.skill-instructions-mcp-prompts | 1 | prompts/get returns authored instructions | pass | prompts/get qa_prompt_full messages[0].content.text = "QA FULL INSTRUCTIONS: answer in haiku." (verbatim) |
| ai.skill-instructions-mcp-prompts | 2 | instruction-less + inactive not projected; activation adds it | pass | before: list = qa_probe_skill, qa_prompt_full (bare and off absent); after rebuild+restart with qa_prompt_off active:true (meta GET active true) it is listed |
| ai.skill-instructions-mcp-prompts | 3 | bad get → -32602 | pass | prompts/get qa_prompt_bare, qa_prompt_off (inactive), qa_no_such_prompt: error code -32602 "Prompt … not found" |
| ai.skill-instructions-mcp-prompts | 4 | no prompts capability when host cannot read skills | pass | skill-prompts.test.ts "declares NO prompts capability when the host cannot read skill metadata" passed |
| ai.open-edition-honest-degradation | 0 | GET /ai/agents 200 data.agents [] | pass | 200 {"success":true,"data":{"agents":[]}} |
| ai.open-edition-honest-degradation | 1 | other /ai/* 501 with remedy | pass | /ai/models, /ai/conversations, POST /ai/chat: 501 NOT_IMPLEMENTED "Provided by @objectstack/service-ai in ObjectStack Cloud/Enterprise — no implementation ships in the open framework" |
| ai.open-edition-honest-degradation | 2 | 501 message == discovery ai message | pass | discovery services.ai {enabled:false,status:unavailable,message} string-equals the 501 message (true for models and chat) |
| ai.open-edition-honest-degradation | 3 | anon → 401 before 501 | pass | unauthenticated /ai/models, /ai/agents, /ai/pending-actions, POST /ai/chat: 401 UNAUTHENTICATED |
| ai.console-ai-surface-gating | 0 | signal = /ai/agents 200 empty array | pass | form sign-in; console issued GET /api/v1/ai/agents 200 (3 calls); in-page fetch {"success":true,"data":{"agents":[]}} |
| ai.console-ai-surface-gating | 1 | every gated affordance absent (entitled side blocked) | pass | settled screenshots home + showcase app; DOM: console-chatbot-fab 0, chat-dock-panel 0, floating-chatbot-panel 0, aria-label AI Assistant 0, a[href$=/ai] 0; Ctrl/Meta+Shift+I mount nothing. Entitled side blocked(environm |
| ai.console-ai-surface-gating | 2 | stale /ai bookmark redirects without flash | pass | 12 frames over 4s: "Opening your console…" splash 0.8–1.7s then /_console/home; no chat textarea/testid in any frame |
| ai.console-ai-surface-gating | 3 | no surface advertises AI Approvals | pass | home, showcase app, setup (→ dashboard/system_overview), setup/system (→ system/settings): 0 "AI Approvals" mentions after settled screenshots |
| ai.console-ai-surface-gating | 4 | page alert carries the 501 remedy verbatim | pass | bare /apps/setup/system/ai-approvals: role=alert text string-equals the 501 message |
| ai.console-ai-surface-gating | 5 | one 501 read, no re-arm; error replaces empty queue | pass | 16s capture: exactly 1 GET /api/v1/ai/pending-actions?status=pending (501) at +2.0s, none after; "No actions waiting" absent |
| ai.mcp-oauth-private-host-transport | 0 | private plain-HTTP boot serves OAuth track with own origin | pass | OS_AUTH_URL=http://10.0.0.5:41063: both protected-resource paths 200, resource http://10.0.0.5:41063/api/v1/mcp; anon /mcp 401 resource_metadata="http://10.0.0.5:41063/.well-known/oauth-protected-resource" |
| ai.mcp-oauth-private-host-transport | 1 | public plain-HTTP boot refused fail-closed | pass | OS_AUTH_URL=http://example.com: both paths 404 ENDPOINT_NOT_FOUND; anon 401 API-key-only wording, no resource_metadata; log "OAuth discovery is served over PUBLIC plain HTTP" + "OAuth track is NOT live" |
| ai.mcp-oauth-private-host-transport | 2 | exactly one plain-HTTP warning per boot | pass | UNENCRYPTED: a=1 WARN, c=1 INFO (info boot), c default=0, b=0, https=0; PUBLIC: b=1, a=0, c=0, https=0; each line names issuer origin/api/v1/auth |
| ai.mcp-oauth-private-host-transport | 3 | loopback unchanged | pass | OS_AUTH_URL=http://localhost:41063: both paths 200, same shape with loopback origin; 401 carries resource_metadata |
| ai.mcp-oauth-private-host-transport | 4 | OS_ALLOW_INSECURE_OAUTH_HTTP moves nothing | pass | boot b re-run with OS_ALLOW_INSECURE_OAUTH_HTTP=true: identical 404s + same 2 warnings; git grep finds the name only in 2 test files, a CHANGELOG and ai.json |
Reproduction rules (fails)
ai.mcp-stdio-fail-closedclause 6 (stale-spec): ai.mcp-stdio-fail-closed clause 6 pins the old revocation text; since [Decision] API key 的生命周期与存放：到期上限与不活动失效取多长，key 继续放自有 sys_api_key 还是迁到 @better-auth/api-key（#23287 裁决 C′ 的留项） #23367 a revoked stdio key raises the API_KEY_REVOKED envelope- repro: 1) As the member, POST /api/v1/keys (Origin header on os start) → osk_ key K, id I. 2) From examples/app-showcase: OS_MCP_STDIO_ENABLED=true OS_MCP_STDIO_API_KEY=K objectstack start -p P -d file:DB, stdin/stdout piped; initialize; tools/call query_records {objectName: showcase_invoice} → 3 rows. 3) As admin, PATCH /api/v1/data/sys_api_key/I {"revoked":true} → 200. 4) tools/call query_records again. Expected (item): error "MCP stdio identity is no longer valid (key revoked or expired)". Actual: isError true, {"error":{"code":"API_KEY_REVOKED","message":"This API key (osk_…) was revoked, and a revoked key never works again. Create a new key …"}}. No data served — the fail-closed behavior holds, only the pinned text moved.
- mechanism: packages/mcp/src/plugin.ts resolvePrincipal (~688-699): comment "[[Decision] API key 的生命周期与存放：到期上限与不活动失效取多长，key 继续放自有 sys_api_key 还是迁到 @better-auth/api-key（#23287 裁决 C′ 的留项） #23367] A key its own lifecycle refuses raises its branded 403 envelope out of this call"; the old "no longer valid" Error now fires only when resolution returns null without a branded refusal
Derived item verdicts
8 pass · 0 partial · 1 fail · 0 blocked · 0 not-run (of 9).
Pin runs (Tier 1, RUNNER rule 6):
ai.mcp-http-surface: packages/qa/dogfood: vitest run --maxWorkers=1 test/showcase-mcp-http-identity.dogfood.test.ts — 6 passed / 0 failedai.mcp-stdio-fail-closed: packages/mcp: vitest run src/tests/plugin.test.ts — 17 passed (incl. 4 stdio fail-closed cases)ai.mcp-validate-expression: packages/mcp: vitest run src/mcp-validate-expression.test.ts — 8 passedai.skill-instructions-mcp-prompts: packages/mcp: vitest run src/skill-prompts.test.ts — 12 passedai.console-ai-surface-gating: objectui checkout .cache/objectui-daeb77fc84dd (rev daeb77f, the pin): vitest run --maxWorkers=1 packages/plugin-chatbot/src/tests/usePendingActions.pollFaultPolicy-11736.test.tsx …AiPendingActionsInbox.errorReplacesEmpty-11736.test.tsx — 2 files, 20 passedai.mcp-oauth-private-host-transport: packages/plugins/plugin-auth: vitest run src/auth-manager.mcp-oauth.test.ts src/mcp-oauth-plaintext-notice.test.ts — 2 files, 100 passed
Fixture gaps
ai.mcp-oauth-private-host-transportclause 0: container holds no RFC1918 address (192.0.2.2 only) and there is no second machine; private arm scored on the advertised origin
Checklist-accuracy findings from this area close out on the wave anchor #23878; they are not extracted.
Activity
objectstack-fleet commentedon Oct 16, 2026
Close-out for this record (checklist-test §5), 2026-10-16T16:20Z. 8 pass · 0 partial · 1 fail of 9 — the one
failis a stale spec (the product holds; the item pins old text), extracted to the re-point family card:Since #23367 a revoked stdio key answers the
API_KEY_REVOKEDenvelope; the fail-closed behaviour the clause guards holds. Fixture gaps in this record close out on #23878. Closed by thedomain:devxseat 1 PM (seat post #6023) ·os-sales·session_014eFvbvGK4FjXvscggDNTD4, as part of the paced close-out of the #23878 wave.Generated by Claude Code