feature: Introduce azure-app-onboard skill along with azure-app-onboard-prereq + vally tests - #2938
Merged
JasonYeMSFT (JasonYeMSFT) merged 33 commits intoJul 23, 2026
Conversation
…2232) * feat: AppOnboard skill and test files * update trigger snapshots for pr gate * fix dead references and remove view call * migration from waza to fully integration tests * fix failing unit tests for CI checks * fix failing unit tests for CI checks * remove redundant integration tests --------- Co-authored-by: Vaibbavi SK <vaibbavi@Gmail.com>
…ig.ts and update-snapshots.js. Fixture repos (e.g. deploy-test-repo-app-service) contain real .test.js files that are part of the simulated user project — they shouldn't be discovered by the parent Jest runner.
…ssertions and ensure no post deploy execution
…ed and e2e deploy evals
… add evals
Skill changes (azure-app-onboard):
- Free-tier degradation: when F1 quota and SWA Free cap are exhausted, degrade
to the cheapest AVAILABLE tier instead of assuming a paid SKU, and record an
assumptions[] note so the approval gate surfaces WHY cost/SKU differs.
- SKU quota validation: treat the limit=0/used=-1 API response as the "Free tier
not offered here" sentinel (clamp so it no longer reads as available=1); add a
Static Web Apps Free cap check (~10 apps/sub) since it has no Quota provider.
- Existing IaC: never delete or overwrite user files - move displaced files to
.copilot-azure/sessions/<id>/replaced-files/ (mirror path). Determine file
ownership via any session's scaffold-manifest.json, not git commit status.
- Routing: scope the azd-template file-system check to repo root + infra/ only
(never scan .copilot-azure/).
- Handoff: suggest post-deploy skills (azure-reliability, azure-cost, azure-rbac)
but never self-execute hardening; add an explicit verbatim completion line.
- Session: ensure repo .gitignore contains .copilot-azure/ before writing any
session artifact (may hold deploy secrets).
- Schema: SelfReviewFinding.layer is now a string ("L1"-"L4").
Test infra (vally):
- shell-command-invoked-grader now scans run_in_terminal/run_command/shell/
terminal by default (not just bash/powershell/pwsh) so a banned command like
'azd up' can't slip through ungraded; add coverage.
Evals:
- Add/update app-onboard + prereq eval suites (onboard, prepare, scaffold,
seeded-deploy, e2e app service/container apps) and results.
vaibbavis
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
JasonYeMSFT (JasonYeMSFT)
marked this pull request as ready for review
July 21, 2026 22:45
JasonYeMSFT (JasonYeMSFT)
requested a review
from Tom Meschter (tmeschter)
as a code owner
July 21, 2026 22:45
JasonYeMSFT (JasonYeMSFT)
requested review from
a team and
Rick Winter (RickWinter)
as code owners
July 21, 2026 22:45
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the new Azure App Onboard skill family (orchestrator + phase sub-skills + prereq) along with extensive reference documentation, test registration, and new Vally grading infrastructure to support the pipeline.
Changes:
- Introduces new skills under
plugin/skills/azure-app-onboard/andplugin/skills/azure-app-onboard-prereq/with progressive-disclosure reference docs. - Registers new skills for testing and schedules them in
tests/skills.json. - Adds a new reusable Vally grader (
shell-command-invoked) and registers it.
Show a summary per file
| File | Description |
|---|---|
| tests/vally/vally-graders.ts | Registers the new ShellCommandInvokedGrader in the global grader registry. |
| tests/vally/shell-command-invoked-grader.ts | New Vally grader that matches/blocks shell commands after stripping non-executable content. |
| tests/vally/tests/shell-command-invoked-grader.test.ts | Unit tests for the new shell-command-invoked grader. |
| tests/skills.json | Registers azure-app-onboard + azure-app-onboard-prereq and adds them to the integration test schedule. |
| plugin/skills/azure-app-onboard/version.json | Adds NBGV per-skill version configuration. |
| plugin/skills/azure-app-onboard/SKILL.md | Adds the orchestrator skill definition and 10-step workflow. |
| plugin/skills/azure-app-onboard/references/pipeline-rules.md | Defines cross-cutting pipeline rules (gates, lifecycle, prohibited patterns). |
| plugin/skills/azure-app-onboard/references/pipeline-rules-runtime.md | Captures known runtime/platform issues and workarounds. |
| plugin/skills/azure-app-onboard/references/session-protocol.md | Defines session creation/resume protocol and auth gate behavior. |
| plugin/skills/azure-app-onboard/references/session-schemas.ts | Adds shared TypeScript interfaces for session artifacts used across phases. |
| plugin/skills/azure-app-onboard/references/approval-gates.md | Defines scaffold/deploy approval gate presentation requirements. |
| plugin/skills/azure-app-onboard/references/intent-gathering.md | Defines scope triage + intent gathering flow around prereq outputs. |
| plugin/skills/azure-app-onboard/references/azd-template-routing.md | Specifies detection and routing behavior for existing azd templates. |
| plugin/skills/azure-app-onboard/references/subscription-resolution.md | Provides subscription resolution fallback procedure. |
| plugin/skills/azure-app-onboard/references/mcp-tool-reference.md | Shared MCP tool index + global parameter guidance. |
| plugin/skills/azure-app-onboard/references/iac-resources.md | Reference links for IaC authoring and troubleshooting. |
| plugin/skills/azure-app-onboard/references/handoff-protocol.md | Defines required handoff content (identity, cleanup, redeploy, recommendations). |
| plugin/skills/azure-app-onboard/prepare/SKILL.md | Defines prepare phase workflow, quota validation and pricing guidance hooks. |
| plugin/skills/azure-app-onboard/prepare/references/mcp-tools.md | Prepare-phase MCP tool parameter reference and tool map. |
| plugin/skills/azure-app-onboard/prepare/references/validation-rubric.md | Defines the multi-dimension plan validation rubric. |
| plugin/skills/azure-app-onboard/prepare/references/subagent-quota.md | Quota validation subagent template and output schema. |
| plugin/skills/azure-app-onboard/prepare/references/subagent-pricing.md | Pricing subagent template and output schema. |
| plugin/skills/azure-app-onboard/prepare/references/sku-matrix.md | Budget-tier SKU selection matrix and fast-track rules. |
| plugin/skills/azure-app-onboard/prepare/references/pricing-guide.md | Cost estimation methodology and Retail Prices API rules. |
| plugin/skills/azure-app-onboard/prepare/references/naming-patterns.md | Resource naming rules used by prepare and consumed by scaffold. |
| plugin/skills/azure-app-onboard/scaffold/references/iac-generation-rules.md | IaC generation rules and mandatory tagging/security patterns. |
| plugin/skills/azure-app-onboard/scaffold/references/validation-and-manifest.md | Validation + scaffold-manifest + deploy-gate bridging guidance. |
| plugin/skills/azure-app-onboard/scaffold/references/scaffold-schemas.ts | TypeScript interface definitions for scaffold-manifest.json. |
| plugin/skills/azure-app-onboard/scaffold/references/scaffold-healing-rules.md | Self-healing loop rules, escalation cadence, and plan-level change process. |
| plugin/skills/azure-app-onboard/scaffold/references/self-healing.md | Error classification table for scaffold validation failures. |
| plugin/skills/azure-app-onboard/scaffold/references/subagent-review.md | Subagent template for layered L1–L4 adversarial review. |
| plugin/skills/azure-app-onboard/scaffold/references/self-review-procedure.md | Procedure for dispatching and consuming self-review results. |
| plugin/skills/azure-app-onboard/scaffold/references/waf-checklist.md | WAF alignment checklist for scaffold review. |
| plugin/skills/azure-app-onboard/scaffold/references/rbac-roles.md | Role assignment GUID reference and Cosmos DB RBAC caveats. |
| plugin/skills/azure-app-onboard/scaffold/references/env-var-secrets.md | Rules for env var derivation and Key Vault secret wiring. |
| plugin/skills/azure-app-onboard/scaffold/references/dockerfile-generation.md | Dockerfile generation and BuildKit-stripping guidance. |
| plugin/skills/azure-app-onboard/scaffold/references/error-handling.md | Scaffold-phase error handling reference. |
| plugin/skills/azure-app-onboard/scaffold/references/cicd-pipelines.md | Defers CI/CD generation and points to MCP guidance tool. |
| plugin/skills/azure-app-onboard/scaffold/references/bicep-app-service.md | App Service Bicep module patterns and deploy-strategy rules. |
| plugin/skills/azure-app-onboard/scaffold/references/bicep-patterns-data.md | Data service module patterns (Postgres/MySQL/Redis). |
| plugin/skills/azure-app-onboard/scaffold/references/bicep-swa.md | Static Web Apps Bicep module rules for detached deployments. |
| plugin/skills/azure-app-onboard/deploy/SKILL.md | Defines deploy phase workflow, preflight delegation, and artifact finalization. |
| plugin/skills/azure-app-onboard/deploy/references/subagent-preflight.md | Subagent template for generating deploy-checklist.md. |
| plugin/skills/azure-app-onboard/deploy/references/preflight-checks.md | Preflight procedure including mandatory what-if and RBAC checks. |
| plugin/skills/azure-app-onboard/deploy/references/approval-gate-template.md | Deploy approval gate display template and response handlers. |
| plugin/skills/azure-app-onboard/deploy/references/portal-links.md | Portal link generation rules to avoid broken encoding. |
| plugin/skills/azure-app-onboard/deploy/references/mcp-tools.md | Deploy-phase MCP tool parameter reference and tool map. |
| plugin/skills/azure-app-onboard/deploy/references/deploy-schemas.ts | TypeScript interface definitions for deploy-result.json. |
| plugin/skills/azure-app-onboard/deploy/references/deploy-safety.md | Safety rules including 403 scope fallback procedure. |
| plugin/skills/azure-app-onboard/deploy/references/blocked-patterns.md | Hard-block list for destructive/imperative commands. |
| plugin/skills/azure-app-onboard/deploy/references/health-check-patterns.md | Post-deploy health verification patterns and false-positive detection. |
| plugin/skills/azure-app-onboard/deploy/references/error-classification.md | Deploy-time error classification and healing rules. |
| plugin/skills/azure-app-onboard/deploy/references/database-post-deploy.md | Migration discovery and execution guidance post-deploy. |
| plugin/skills/azure-app-onboard/deploy/references/code-deployment-appservice.md | App Service/Functions code deploy procedures (Oryx/Kudu/SCM toggling). |
| plugin/skills/azure-app-onboard/deploy/references/code-deployment-swa.md | Static Web Apps deploy procedures and Windows path workaround. |
| plugin/skills/azure-app-onboard-prereq/version.json | Adds NBGV per-skill version configuration. |
| plugin/skills/azure-app-onboard-prereq/SKILL.md | Adds prereq skill definition and evaluation workflow. |
| plugin/skills/azure-app-onboard-prereq/references/prereq-schemas.ts | TypeScript interface definitions for prereq-output.json. |
| plugin/skills/azure-app-onboard-prereq/references/prereq-artifacts.md | Artifact write requirements and phase exit checklist. |
| plugin/skills/azure-app-onboard-prereq/references/readiness-gate.md | Readiness scoring rules + remediation decision gates. |
| plugin/skills/azure-app-onboard-prereq/references/remediation-protocol.md | Remediation scope, loop limits, and build-validation consent gate. |
| plugin/skills/azure-app-onboard-prereq/references/session-protocol.md | Direct-entry session handling and resume/fresh gating. |
| plugin/skills/azure-app-onboard-prereq/references/session-schemas.ts | Local copy of shared session interfaces for prereq. |
| plugin/skills/azure-app-onboard-prereq/references/subagent-starter-scaffold.md | Subagent template for zero-code starter app scaffolding. |
| plugin/skills/azure-app-onboard-prereq/references/zero-code-path.md | Zero-code workflow path and build-validation gate. |
| plugin/skills/azure-app-onboard-prereq/references/build-check.md | Static build-health evaluation rules and optional execution gate. |
| plugin/skills/azure-app-onboard-prereq/references/completeness-check.md | Completeness checks (entry point, config, ports, container readiness). |
| plugin/skills/azure-app-onboard-prereq/references/deployability-check.md | Deployability and specialized-skill routing rules. |
| plugin/skills/azure-app-onboard-prereq/references/component-mapping.md | Component mapping + existing infra detection + compose extraction. |
| plugin/skills/azure-app-onboard-prereq/references/cloud-sdk-migration.md | Classification rules for non-Azure cloud SDK dependencies. |
| plugin/skills/azure-app-onboard-prereq/references/subscription-resolution.md | Subscription resolution fallback procedure. |
Review details
Comments suppressed due to low confidence (3)
plugin/skills/azure-app-onboard/scaffold/references/validation-and-manifest.md:45
- This phase-exit guidance mentions
validationResult.status: 'Passed'|'Failed'|'Skipped', but scaffold-schemas.ts definesstatusas"Validated" | "Partial" | "Failed". Keeping these inconsistent will lead to invalidscaffold-manifest.jsonoutputs.
plugin/skills/azure-app-onboard/scaffold/references/validation-and-manifest.md:31 - The
validationResult.checks[]example uses{ result: "PASS" }, butValidationCheckin scaffold-schemas.ts definespassed: boolean(and optionaldetail). As written, agents will produce a manifest that doesn't match the documented interface.
plugin/skills/azure-app-onboard/SKILL.md:23 - This SKILL.md is missing a dedicated
## MCP Toolssection. Most service skills in this repo include one (e.g., azure-app-onboard/prepare and azure-app-onboard/scaffold), and reviewers/tests may expect it as part of the standard skill structure.
## Quick Reference
| Property | Value |
|----------|-------|
| Best for | Developers who know what to build but not which Azure services to use |
| Inputs | Business idea or existing codebase, budget/scale preferences (optional) |
| Outputs | Architecture plan, cost estimate, IaC files, deployed Azure resources |
| Phases | Discover → Architect → Scaffold → Deploy (self-contained, no external skill calls) |
## When to Use This Skill
- Files reviewed: 98/98 changed files
- Comments generated: 5
- Review effort level: Low
Member
|
The CI pointed out a few invalid patterns in your eval suites. Completed grader and early terminate condition cannot coexist because these tests will always fail when they trigger the early terminate condition. |
JasonYeMSFT (JasonYeMSFT)
previously approved these changes
Jul 22, 2026
Contributor
Author
Removed the completed grader |
JasonYeMSFT (JasonYeMSFT)
previously approved these changes
Jul 22, 2026
JasonYeMSFT (JasonYeMSFT)
approved these changes
Jul 23, 2026
Tom Meschter (tmeschter)
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduces the App Onboard skill family — a self-contained pipeline that takes a user from a business/app idea or an existing codebase all the way to a running Azure deployment, with live cost estimates and approval gates along the way. All net-new; no existing skills are modified.
This is a parallel path, not a replacement for
azure-prepare → azure-validate → azure-deploy. That chain already goes end-to-end, but its default recipe forks an azd starter template into the workspace and assumes the user already knows their target architecture. App Onboard is for the user who doesn't: it analyzes the app, picks services, shows live cost, gates on approval, and generates production-shippable custom Bicep — starting from a prompt as vague as "I want to build a marketplace app." Across today's 28-skill catalog, none estimate cost before deploy, none produce a halting code-readiness verdict, and none handle "I don't know where to start."Two new top-level skills, five SKILL.md files total: an orchestrator (
azure-app-onboard) with three phase sub-skills (prepare,scaffold,deploy), plus a standaloneazure-app-onboard-prereq.azure-app-onboardazure-app-onboard/prepareazure-app-onboard/scaffoldaz bicep build, max 3 attempts)azure-app-onboard/deployaz deployment, and runs post-deploy health checks on every endpointazure-app-onboard-prereqHow this stays distinct from
azure-prepare → azure-validate → azure-deployreferences/pipeline-rules.md): the pipeline never callsazure-validate/azure-deploy/azure-preparemid-flow — validation is directaz bicep build, cost/quota run as inline sub-agents, deployment is directaz deployment. No azd dependency.$X/mofigure from the Retail Prices API..copilot-azure/sessions/{uuid}/(context.json,prereq-output.json,prepare-plan.json,scaffold-manifest.json,deploy-result.json) survive context compaction; a mid-flow "use App Service instead" re-runs only the affected phases.azure-prepare; non-Azure cloud SDKs (aws-sdk,boto3,google-cloud,firebase) →azure-cloud-migrate;azd up/ existing-spend cost work →azure-deploy/azure-cost. If the user just wants to ship infra they already have, App Onboard hands off instead of engaging.Prereq has no equivalent in the catalog
No existing skill answers "is my code ready to deploy?" with a halting verdict before infrastructure work begins.
azure-validatevalidates infrastructure (Bicep/Terraform/RBAC), not source code;azure-prepareanalyzes the stack to pick services but produces no readiness verdict the pipeline can halt on. Prereq catches broken builds, EOL runtimes, intentionally vulnerable apps, and non-Azure cloud SDKs before any infra is generated.What's included
skills.json.Test infrastructure
shell-command-invoked(+ its own test suite), registered in the existingvally-graders.ts: flags disallowed shell commands across all shell tool names, and strips non-executable content (heredoc bodies, PowerShell here-strings, comments) before matching so command-like text inside generated files doesn't cause false positives.0.0.0-placeholder(NBGV-stamped at build);version.jsonadded per new skill.Vally eval results
Deploy group — real Azure provisioning (billed) — 3 e2e specs + seeded-deploy,
--workers 3,claude-sonnet-4.6, Playground-01:deploy-depth(App Service via Bicep)fasttrack(static HTML → App Service F1 / SWA Free)wetty(Node/TS → ACR build → Container Apps)seeded-deploy(App Service + Container Apps pipelines)133/134 graders passed (99.3%) — all four eval specs clear the 80% threshold. The single grader miss is a regex false-negative in
shell-command-invoked(the live HTTP health check did run — the FQDN was assigned to$URLbeforecurl, so the ordering-sensitive regex missed it), not a deploy regression:deploy-result.jsonfinalized withstatus:"succeeded"+healthStatus:"healthy"and a liveazurecontainerapps.ioendpoint.Prereq — free suite — 22 stimuli:
prereqSafe group (non-billed pipeline suites) — all clear the 80% threshold:
onboard94.0%,prepare86.1%,scaffold93.5%. Remaining fails are lower-severity — probabilisticskill-invocationrouting misses (flip to PASS on retry), async session-artifact flush timing, and one phrasing false-positive — not wrong content.Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:Related Issues