eval: migrate azure-cloud-migrate integration tests to vally eval suites - #3005
Merged
JasonYeMSFT (JasonYeMSFT) merged 4 commits intoAug 5, 2026
Conversation
Adds evals/azure-skills/azure-cloud-migrate/eval.yaml with 8 stimuli
migrated from tests/azure-cloud-migrate/integration.test.ts:
Brownfield (tier: full, runs: 1):
- Face-blur Lambda migration — clones aws-samples/serverless-face-blur-service,
runs migration prompt + follow-up, checks skill invocation and output files
(migration-status.md, migration-assessment-report.md, src/{app,detectFaces,blurFaces}.js)
- Webapp Lambda migration — clones aws-samples/lambda-refarch-webapp, checks
skill invocation and migration-status.md / migration-assessment-report.md
Routing (tier: smoke/full, runs: 5, earlyTerminate on skill-call):
- Spring Boot to ACA compatibility assessment (smoke)
- Spring Boot JAR containerization to ACA (full)
- AWS Fargate ECS tasks to ACA (smoke)
- ECS Fargate containers to ACA (full)
- GKE Kubernetes workloads to ACA (smoke)
- Kubernetes manifest conversion to ACA config (full)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
use vally test in CI
JasonYeMSFT (JasonYeMSFT)
changed the title
JasonYeMSFT (JasonYeMSFT)
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the azure-cloud-migrate skill’s integration validation from Jest-based tests to a Vally eval suite, and updates CI so the skill runs under the Vally integration path instead of the Jest integration path.
Changes:
- Removed
azure-cloud-migrateJest trigger tests, integration tests, and their snapshot file. - Added a new Vally eval suite at
evals/azure-skills/azure-cloud-migrate/eval.yamlcovering brownfield migration + routing scenarios. - Updated the all-integration workflow to route
azure-cloud-migrateaway from the Jest integration job and into the Vally job.
Show a summary per file
| File | Description |
|---|---|
| tests/azure-cloud-migrate/triggers.test.ts | Deletes Jest trigger test coverage for the skill. |
| tests/azure-cloud-migrate/integration.test.ts | Deletes Jest integration tests previously used for live-agent validation. |
| tests/azure-cloud-migrate/snapshots/triggers.test.ts.snap | Removes snapshots tied to the deleted trigger tests. |
| evals/azure-skills/azure-cloud-migrate/eval.yaml | Adds the Vally eval suite intended to replace the deleted Jest integration coverage. |
| .github/workflows/test-all-integration.yml | Routes azure-cloud-migrate to run under Vally instead of Jest integration tests in CI. |
Review details
Suppressed comments (1)
evals/azure-skills/azure-cloud-migrate/eval.yaml:103
- The webapp migration stimulus currently checks for
migration-status.md/migration-assessment-report.mdanywhere under the workspace. To ensure we’re validating the skill’s required output directory (<workspace>-azure/) and not an accidentally-written file in the source tree, scope these globs to**/*-azure/...like the other migration assertions.
- type: file-exists
config:
path: "**/migration-status.md"
- type: file-exists
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Lite
msalaman
previously approved these changes
Aug 5, 2026
msalaman
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Migrate azure-cloud-migrate skill's integration tests to use vally eval suites.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:integration -- <skill>ornpm run test:vally -- --skill <skill>)Related Issues
#2902