Feature/azure quotas by rakal-dyh · Pull Request #1137 · microsoft/GitHub-Copilot-for-Azure

rakal-dyh · 2026-03-05T20:14:49Z

In progress. Test need to be added

…r-Azure into feature/azure-quotas

…itHub-Copilot-for-Azure into feature/azure-quotas

rakal-dyh · 2026-03-06T06:47:15Z

@microsoft-github-policy-service agree [company="Microsoft"]

rakal-dyh · 2026-03-06T06:51:02Z

@microsoft-github-policy-service agree company="Microsoft"

rakal-dyh · 2026-03-06T22:02:29Z

Add test

…r-Azure into feature/azure-quotas

JasonYeMSFT (JasonYeMSFT) · 2026-03-09T23:35:25Z

Please bump the version number of azure-prepare. I'll see if I can reduce thec char count of some other skills to let your skills plus everything else fit in the budget.

JasonYeMSFT (JasonYeMSFT) · 2026-03-10T18:28:55Z

I merged a change to reduce the char count of other skills' descriptions. Please rebase to the latest main and check the total char count again.

rakal-dyh · 2026-03-10T18:31:59Z

Add a note here, the 'creates correct files for Terraform recipe' integration test under azure-prepare seems flaky.

I run this test under origin code and it can still fail. The bug analysis shows that it failed because the azure.yaml file generated against the assertion. But the Terraform recipe README says "default to azd+Terraform", so there is a chance it will generate azure.yaml some times

rakal-dyh · 2026-03-10T18:43:59Z

Add note, the azure-prepare integration test 'generates Entra-only SQL auth for ASP.NET Core EF Core app (not SQL admin password)' seems flaky as well which failed in my local with origin code,

and test failure // Must NOT use legacy SQL admin login/password auth
> 830 | expect(/administratorLoginPassword/i.test(bicepContent)).toBe(false);

analysis shows 'The agent tried to load services/sql/README.md but the correct path is services/sql-database/README.md. The view call returned "Path does not exist", so the agent never
saw the Entra-only auth guidance and fell back to generating legacy SQL admin password auth.'

…r-Azure into feature/azure-quotas

Sai Koumudi Kaluvakolanu (saikoumudi)

rakal-dyh could you also add the screenshots in the PR

* Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule #7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule #8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (microsoft#1200) * Collapse token analysis comment (microsoft#1147) * update region-availability in prepare/validate/deploy skill (microsoft#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (microsoft#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (microsoft#1204) * Add AVM (Azure Verified Modules) integration tests (microsoft#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (microsoft#1149) * Early terminate azure-deploy, azure-validate tests (microsoft#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (microsoft#1209) * Early terminate azure-deploy tests on deploy link (microsoft#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (microsoft#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (microsoft#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (microsoft#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (microsoft#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (microsoft#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

…and more. (#1219) * Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule #7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule #8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (#1200) * Collapse token analysis comment (#1147) * update region-availability in prepare/validate/deploy skill (#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (#1204) * Add AVM (Azure Verified Modules) integration tests (#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (#1149) * Early terminate azure-deploy, azure-validate tests (#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (#1209) * Early terminate azure-deploy tests on deploy link (#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

* Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule #7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule #8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (#1200) * Collapse token analysis comment (#1147) * update region-availability in prepare/validate/deploy skill (#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (#1204) * Add AVM (Azure Verified Modules) integration tests (#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (#1149) * Early terminate azure-deploy, azure-validate tests (#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (#1209) * Early terminate azure-deploy tests on deploy link (#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version * dataset fix (#9) * dataset fix * fix: restore foundry dataset guidance Restore the explicit seed dataset registration guidance in the deploy skill and align dataset docs with the current evaluation_dataset_create MCP surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add codeowner for microsoft-foundry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address foundry PR review comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: normalize foundry dataset versioning docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Deduplicate Foundry dataset registration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * use sdk for eval results * Fix comments * Align observe test with SDK guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant Foundry MCP tips Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dataset manifest version examples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix * Align eval-datasets metadata docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge remote-tracking branch 'upstream/main' --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

* update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

…and more. (microsoft#1219) * Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (microsoft#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (microsoft#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule microsoft#7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule microsoft#8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (microsoft#1200) * Collapse token analysis comment (microsoft#1147) * update region-availability in prepare/validate/deploy skill (microsoft#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (microsoft#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (microsoft#1204) * Add AVM (Azure Verified Modules) integration tests (microsoft#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (microsoft#1149) * Early terminate azure-deploy, azure-validate tests (microsoft#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (microsoft#1209) * Early terminate azure-deploy tests on deploy link (microsoft#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (microsoft#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (microsoft#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (microsoft#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (microsoft#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (microsoft#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

* Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (microsoft#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (microsoft#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule microsoft#7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule microsoft#8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (microsoft#1200) * Collapse token analysis comment (microsoft#1147) * update region-availability in prepare/validate/deploy skill (microsoft#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (microsoft#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (microsoft#1204) * Add AVM (Azure Verified Modules) integration tests (microsoft#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (microsoft#1149) * Early terminate azure-deploy, azure-validate tests (microsoft#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (microsoft#1209) * Early terminate azure-deploy tests on deploy link (microsoft#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (microsoft#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (microsoft#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (microsoft#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (microsoft#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (microsoft#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version * dataset fix (microsoft#9) * dataset fix * fix: restore foundry dataset guidance Restore the explicit seed dataset registration guidance in the deploy skill and align dataset docs with the current evaluation_dataset_create MCP surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add codeowner for microsoft-foundry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address foundry PR review comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: normalize foundry dataset versioning docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Deduplicate Foundry dataset registration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * use sdk for eval results * Fix comments * Align observe test with SDK guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant Foundry MCP tips Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dataset manifest version examples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix * Align eval-datasets metadata docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge remote-tracking branch 'upstream/main' --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

@jongio

* Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule #7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule #8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (#1200) * Collapse token analysis comment (#1147) * update region-availability in prepare/validate/deploy skill (#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (#1204) * Add AVM (Azure Verified Modules) integration tests (#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (#1149) * Early terminate azure-deploy, azure-validate tests (#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (#1209) * Early terminate azure-deploy tests on deploy link (#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version * dataset fix (#9) * dataset fix * fix: restore foundry dataset guidance Restore the explicit seed dataset registration guidance in the deploy skill and align dataset docs with the current evaluation_dataset_create MCP surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add codeowner for microsoft-foundry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address foundry PR review comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: normalize foundry dataset versioning docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Deduplicate Foundry dataset registration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * use sdk for eval results * Fix comments * Align observe test with SDK guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant Foundry MCP tips Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dataset manifest version examples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix * Align eval-datasets metadata docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge remote-tracking branch 'upstream/main' * feat(microsoft-foundry): refresh agent evaluation workflows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enforce LF line endings for Markdown files Windows clones with core.autocrlf=true convert SKILL.md to CRLF in the working tree, breaking unit tests like airunway-aks-setup's 'frontmatter has no tabs' check whose regex (/^-{3}\n.../) does not match across \r\n. Forcing eol=lf for *.md keeps line endings consistent across platforms and prevents test failures on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(microsoft-foundry): rename "test suite" to "evaluation suite" and add regression purpose Addresses PR #2092 review comments from @jongio: 1. Standardize terminology on "evaluation suite" to match the evaluationSuites[] schema field. Renames human-language "test suite" and "test-suite" mentions across deploy.md, observe.md, analyze-results.md, evaluate-step.md, compare-iterate.md, cicd-monitoring.md, and the agent-metadata-contract.md guidance section/heading. 2. Add `regression` to the `purpose` values table in agent-metadata-contract.md so the suggested values match the `trace-regression-suite` example (`purpose: regression`) and the analyze-results.md prioritization rule (`purpose=regression`). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(microsoft-foundry): document metadata-file fallback when default is absent Addresses PR #2092 Copilot review (#pullrequestreview-4195041793, discussion r3159385842) by extending Workflow Rule #3 in agent-metadata-contract.md to describe what happens when .foundry/agent-metadata.yaml is missing: - Use the only matching .foundry/agent-metadata.<env>.yaml sidecar when exactly one exists. - Require the user to choose the metadata file when multiple sidecar files exist and no explicit file/path was provided. Keeps the contract symmetric with Workflow Rule #2 (multiple agent roots also require a user choice) and removes the previous ambiguity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>

Yinghui Dong and others added 7 commits March 4, 2026 16:25

update for using azure-quotas in skill

551f066

Merge branch 'main' of https://github.com/microsoft/GitHub-Copilot-fo…

e7244e8

…r-Azure into feature/azure-quotas

Merge branch 'main' of https://github.com/microsoft/GitHub-Copilot-fo…

14a70e2

…r-Azure into feature/azure-quotas

test update

f9ab9c0

Merge branch 'main' of https://github.com/microsoft/GitHub-Copilot-fo…

850f799

…r-Azure into feature/azure-quotas

Merge branch 'microsoft:main' into feature/azure-quotas

3009dff

Merge branch 'feature/azure-quotas' of https://github.com/rakal-dyh/G…

495ea8b

…itHub-Copilot-for-Azure into feature/azure-quotas

github-actions Bot mentioned this pull request Mar 6, 2026

[repo-status] Weekly Repo Status — Feb 27 – Mar 6, 2026 #1154

Closed

github-actions Bot mentioned this pull request Mar 6, 2026

[repo-status] Weekly Repo Status — Feb 27 – Mar 6, 2026 #1173

Closed

unit test update

8cabc36

rakal-dyh changed the title ~~[WIP] Feature/azure quotas~~ Feature/azure quotas Mar 6, 2026

Yinghui Dong added 11 commits March 6, 2026 14:53

path update

f3b9915

add skill in skills.json

da927bb

skills.json update

33f5463

reduce the text

14f88ec

version update

fbeec83

skill version

18fd553

skill description update

2ad62b9

reduce text size

3229488

Merge branch 'main' of https://github.com/microsoft/GitHub-Copilot-fo…

088c680

…r-Azure into feature/azure-quotas

1.0.4 for next prepare version

9e7110a

upload snap shot

3a5222e

update version

20c9cbc

JasonYeMSFT (JasonYeMSFT) mentioned this pull request Mar 9, 2026

Reduce char count of existing skills #1210

Merged

Yinghui Dong added 2 commits March 10, 2026 11:52

test update

1c89b0b

Merge branch 'main' of https://github.com/microsoft/GitHub-Copilot-fo…

cc20850

…r-Azure into feature/azure-quotas

JasonYeMSFT (JasonYeMSFT) approved these changes Mar 10, 2026

View reviewed changes

JasonYeMSFT (JasonYeMSFT) merged commit 701fcca into microsoft:main Mar 10, 2026
12 checks passed

Sai Koumudi Kaluvakolanu (saikoumudi) approved these changes Mar 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

Feature/azure quotas - #1137

Feature/azure quotas#1137
JasonYeMSFT (JasonYeMSFT) merged 22 commits into
microsoft:mainfrom
rakal-dyh:feature/azure-quotas

rakal-dyh commented Mar 5, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

JasonYeMSFT (JasonYeMSFT) commented Mar 9, 2026

Uh oh!

JasonYeMSFT (JasonYeMSFT) commented Mar 10, 2026

Uh oh!

rakal-dyh commented Mar 10, 2026

Uh oh!

rakal-dyh commented Mar 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Sai Koumudi Kaluvakolanu (saikoumudi) left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rakal-dyh commented Mar 5, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

rakal-dyh commented Mar 6, 2026

Uh oh!

JasonYeMSFT (JasonYeMSFT) commented Mar 9, 2026

Uh oh!

JasonYeMSFT (JasonYeMSFT) commented Mar 10, 2026

Uh oh!

rakal-dyh commented Mar 10, 2026

Uh oh!

rakal-dyh commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Sai Koumudi Kaluvakolanu (saikoumudi) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rakal-dyh commented Mar 10, 2026 •

edited

Loading