AppOnboard skill and test files - Existing skills testing - #2232
Merged
JasonYeMSFT (JasonYeMSFT) merged 8 commits intoMay 12, 2026
Merged
Conversation
vaibbavis
changed the title
JasonYeMSFT (JasonYeMSFT)
merged commit May 12, 2026
b7c75c2
into
microsoft:features/azure-bya
2 of 4 checks passed
samcdonald-ms
pushed a commit
to vaibbavis/GitHub-Copilot-for-Azure
that referenced
this pull request
Jun 30, 2026
…2232) * feat: AppOnboard skill and test files * update trigger snapshots for pr gate * fix dead references and remove view call * migration from waza to fully integration tests * fix failing unit tests for CI checks * fix failing unit tests for CI checks * remove redundant integration tests --------- Co-authored-by: Vaibbavi SK <vaibbavi@Gmail.com>
JasonYeMSFT (JasonYeMSFT)
pushed a commit
that referenced
this pull request
Jul 23, 2026
…rd-prereq + vally tests (#2938) * AppOnboard skill and test files - Existing skills testing (#2232) * feat: AppOnboard skill and test files * update trigger snapshots for pr gate * fix dead references and remove view call * migration from waza to fully integration tests * fix failing unit tests for CI checks * fix failing unit tests for CI checks * remove redundant integration tests --------- Co-authored-by: Vaibbavi SK <vaibbavi@Gmail.com> * fix: update AppOnboard skill and test files * delete old reference files * Test infra fix: Added fixtures to testPathIgnorePatterns in jest.config.ts and update-snapshots.js. Fixture repos (e.g. deploy-test-repo-app-service) contain real .test.js files that are part of the simulated user project — they shouldn't be discovered by the parent Jest runner. * further improve integration * fix: exclude fixtures from Jest test discovery * call pricing mcp tool with the right params * fix integration test flakiness caused by timeouts * fix integration test flakiness caused by mismatch in skill body and assertions and ensure no post deploy execution * remove deleted test * minor assertion hardening and test improvements * Add eval.yaml files from samcdonald/evals-transition * Increasing Vally coverage for BYA * Refactoring test fixtures and improving Vally evals * Editing deployment-summary reference * Vally: modified folllow-up turns and "file-matches" checks * skill changes * add new custom grader, split vally evals into multipe files, and seeded and e2e deploy evals * feat: harden app-onboard free-tier, existing-IaC, and handoff safety; add evals Skill changes (azure-app-onboard): - Free-tier degradation: when F1 quota and SWA Free cap are exhausted, degrade to the cheapest AVAILABLE tier instead of assuming a paid SKU, and record an assumptions[] note so the approval gate surfaces WHY cost/SKU differs. - SKU quota validation: treat the limit=0/used=-1 API response as the "Free tier not offered here" sentinel (clamp so it no longer reads as available=1); add a Static Web Apps Free cap check (~10 apps/sub) since it has no Quota provider. - Existing IaC: never delete or overwrite user files - move displaced files to .copilot-azure/sessions/<id>/replaced-files/ (mirror path). Determine file ownership via any session's scaffold-manifest.json, not git commit status. - Routing: scope the azd-template file-system check to repo root + infra/ only (never scan .copilot-azure/). - Handoff: suggest post-deploy skills (azure-reliability, azure-cost, azure-rbac) but never self-execute hardening; add an explicit verbatim completion line. - Session: ensure repo .gitignore contains .copilot-azure/ before writing any session artifact (may hold deploy secrets). - Schema: SelfReviewFinding.layer is now a string ("L1"-"L4"). Test infra (vally): - shell-command-invoked-grader now scans run_in_terminal/run_command/shell/ terminal by default (not just bash/powershell/pwsh) so a banned command like 'azd up' can't slip through ungraded; add coverage. Evals: - Add/update app-onboard + prereq eval suites (onboard, prepare, scaffold, seeded-deploy, e2e app service/container apps) and results. * remove integration tests and add app-onboard and prereq to skills.json * remove fixtures references * remove duplicate tests * skill refinemenets for robustness and clear inconsistencies * docker gen update * remove unit and trigger tests * add deterministic script check * add skill.json to scheudle * address pr comments * eval hardening and skill fixes * swa deploy result hardening * remove completed grader in prereq * harden prereq npm install gate --------- Co-authored-by: Vaibbavi SK <vaibbavi@Gmail.com> Co-authored-by: Samantha McDonald <samcdonald@microsoft.com> Co-authored-by: Vaibbavi Senthil Kumar <vaibbavis+microsoft@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Checklist
cd tests && npm test)npm run test:skills:integration -- <skill>)USE FOR/DO NOT USE FOR/PREFER OVERclauses: confirmed no routing regressions for competing skillsRelated Issues