feature: multi plugin project structure by JasonYeMSFT · Pull Request #2872 · microsoft/GitHub-Copilot-for-Azure

JasonYeMSFT (JasonYeMSFT) · 2026-07-17T18:30:07Z

Description

Transform the repository to support development of multiple plugins. Skills in one plugin share the same context window, which makes it infeasible to fit every Azure skill into one plugin given the wide feature area of Azure. We must have a multi-plugin model if we want to ship more skills.

Key Concept

Skills used to be uniquely identified by their names with the assumption that they all belong to the azure plugin. Now skills are uniquely identified by their plugin directory name and their name. Note that the plugin directory name can be different from the plugin name (e.g. the plugin directory name for the azure plugin is azure-skills). Our development tools consistently use plugin directory names to identify plugins/skills for convenience.

Folder structure changes

A new top level plugins/ directory is created. This directory will contain files for all the plugins.
Existing files in plugin/ are moved to plugins/azure-skills/.
Existing files in evals/ are moved to evals/azure-skills/.
An example plugins/cat/ plugin is created to demonstrate how a 2nd plugin can coexist with the azure plugin. It has its eval suites in evals/cat/. I will remove them before merging this PR.
The build output now writes to output/<plugin-dir>/ instead of directly under output/. With the additional cat plugin, build output now includes output/azure-skills/ and output/cat/.

Development Tool adjustments

The following adjustments are made in the tools for skill development:

These local validation scripts are extended to work on skills for all plugins by default.
- frontmatter
- references
- tokens
- plugin version
- copilot cli char budget
- vally stimulus
Vally test runners are adjusted in these ways
- It loads eval suites from the new locations
- It requires the users to use plugin directory and skill name to identify which plugin to test
- The custom executor's skill loading logic is extended to support loading skills from any combination of plugins
The build script is extended to support building all plugins. A workaround is implemented to preserve changelog generated from past git commit history that are lost due to file movement.

Workflow adjustments

All workflows are adjusted to read files from the new locations and work for skills in all plugins.
The skills.json that stores the automated integration test schedule is extended to support storing automated integration test schedule for multiple plugins. The workflow step that reads it is adjusted accordingly.
The file structure of published artifacts to microsoft/skills and microsoft/azure-skills stays as is. This means files for the azure plugin will be synced to the same locations as before, both in .github/plugins/ and at the microsoft/azure-skills repo root. Any new plugins will be synced to be sibling plugins in the .github/plugins/ directory in the target repo without being replicated at any repo root.

User expectations

The skill authors will need to do the following things once the changes are merged.

Develop skills at the new plugin and eval locations.
When manually queuing integration test jobs, the string for identifying the skill to test must carry the plugin directory name. For example, the input must be azure-skills/azure-ai for testing azure-ai skill in the azure-skill plugin directory.

As plugin owners, we have one more responsibility:

Maintain the legacy changelog file. I named the legacy changelog file for the azure plugin changelog-{date}.md that has all the legacy changelog entries. If we ever need to move the plugin and thus lose its git commit history again, we need to update the file with the up-to-date changelog before the movement.

Checklist

Tests pass locally (cd tests && npm test)
Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:integration -- <skill> or npm run test:vally -- --skill <skill>)

Related Issues

#2781

adapt build script

fix pr.yml for multi-plugin model

fix sync-to-azure-mcp.yml fix publish-to-marketplace.yml

Alex Okonechnikov (okonech) · 2026-07-23T19:54:59Z

nit: should "eval:lint": "vally lint plugin/skills/ --eval evals/ --strict" be changed? In GitHub-Copilot-for-Azure\tests\package.json

chandraneel

LGTM

…te eval to requiredSkills The multi-plugin restructure (microsoft#2872) moved plugin content to plugins/azure-skills/ and changed the eval executor to load skills only for stimuli declaring a per-stimulus requiredSkills tag. This merge: - Moves the new aks-baseline.sh/.ps1 scripts, orphaned by the merge at the old plugin/ path, to plugins/azure-skills/skills/azure-diagnostics/scripts/. - Adds requiredSkills: [azure-diagnostics] to all azure-diagnostics eval stimuli so the skill is loaded into the agent (without it, the agent gets no skills and every run reports 'Skills used 0'). Verified: build, frontmatter, references, validate-stimulus, typecheck, lint all pass; the aks-baseline script eval passes 3/3 (100%). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72

The multi-plugin restructure (microsoft#2872) changed the vally skill-loading model: without a requiredSkills tag every plugin skill is loaded and competes for routing, so the behavior probe test dropped to 0/5 (agent hand-rolled its own DNS probe instead of loading azure-diagnostics). Pin azure-diagnostics so its description survives char-budget truncation, matching the sibling-eval convention. Back to 5/5 skill-invocation + shell-command-invoked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c

…script (#2936) * feat: replace AKS baseline diagnostic sweep with aks-baseline script Adds cross-platform aks-baseline.sh/.ps1 that runs the read-only AKS cluster baseline sweep once and prints a single labeled digest (provisioning state, node pools, activity log, node readiness, unhealthy pods, kube-system health, warning events). Replaces the duplicated inline command blocks in general-diagnostics.md, aks-troubleshooting.md, and command-flows.md with script references. Fixes #2506 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * fix: address PR review feedback on aks-baseline scripts - sh: usage() accepts an exit code (--help exits 0); validate a value exists before consuming each option so a missing value no longer loops - sh: set pipefail inside the warning-events subshell so kubectl failures are surfaced by run() - ps1: drop mandatory params (avoids interactive prompts in non-interactive runs); explicitly validate ResourceGroup/Cluster and exit early with a clear message - ps1: remove redundant \Continue assignment (default is Continue) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * test: add eval that exercises the aks-baseline script Adds a response-quality stimulus to the azure-diagnostics eval suite that supplies concrete cluster identity and asserts (via the tool-calls grader) that the agent actually runs aks-baseline.sh/.ps1 rather than pasting the individual az/kubectl commands. No real cluster is required — each script step is guarded, so the sweep completes and emits its digest regardless. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * test: early-terminate aks-baseline eval when the script tool call starts Switches the script-execution stimulus to earlyTerminate on tool-call-match so the run aborts the moment the aks-baseline shell tool call starts, before the script actually executes. The triggering start event is still recorded, so the tool-calls grader confirms the agent chose to run the script; drops the completed grader per the early-terminate convention. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * fix: make aks-baseline eval gradable under early termination The eval early-terminates the moment the aks-baseline script tool call starts, so no tool_result is ever emitted. The built-in tool-calls grader only counts a required tool on its tool_result, so it could never match an early-terminated call. Its inline (?i) regex flag also crashed the custom executor's tag-helpers, which compile earlyTerminate patterns with plain new RegExp(). - Drop (?i) / anchors from the earlyTerminate patterns (JS RegExp has no inline flags), matching the azure-deploy convention. - Add a small custom tool-call-started grader that matches the tool-call start event, and register it in vally-graders.ts. - Switch the stimulus to the new grader and make the prompt directive so the agent routes straight to the script (no real cluster exists in the test subscription). Verified locally: all 3 graders pass (100%); the run terminates at the script tool-call start with no tool_result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * refactor: grade aks-baseline eval with built-in tool-calls grader Replace the custom tool-call-started grader with the built-in tool-calls grader, matching the pattern used by the azure-validate e2e suite. The built-in grader only matches a tool call once its result is recorded, so switch earlyTerminate from tool-call-match (aborts at the call's start, before completion) to tool-call-result (fires once the call completes). The aks-baseline script is read-only and fully guarded, so against a missing cluster every step fails fast and the sweep still exits cleanly in seconds. The call completes -> its tool_result is recorded -> tool-calls equired matches; termination then stops the run before any follow-up turns. No custom grader needed. Verified locally: all 3 graders pass (100%); the script tool call has a matching tool_result and the run ends at 2 turns. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * fix: resolve upstream merge — relocate aks-baseline scripts and migrate eval to requiredSkills The multi-plugin restructure (#2872) moved plugin content to plugins/azure-skills/ and changed the eval executor to load skills only for stimuli declaring a per-stimulus requiredSkills tag. This merge: - Moves the new aks-baseline.sh/.ps1 scripts, orphaned by the merge at the old plugin/ path, to plugins/azure-skills/skills/azure-diagnostics/scripts/. - Adds requiredSkills: [azure-diagnostics] to all azure-diagnostics eval stimuli so the skill is loaded into the agent (without it, the agent gets no skills and every run reports 'Skills used 0'). Verified: build, frontmatter, references, validate-stimulus, typecheck, lint all pass; the aks-baseline script eval passes 3/3 (100%). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 * fix: broaden aks-baseline unhealthy-pod check to catch container-level failures Address PR review: the phase-based field selector (status.phase!=Running,!=Succeeded) missed pods that stay in phase Running while a container fails — CrashLoopBackOff, ImagePullBackOff, readiness-probe failures, and high restart counts. Step 5 now lists all pods (kubectl get pods -A -o wide) and flags a pod when it is not Ready, has a non-Running/non-terminal STATUS, or has a high restart count. Terminal pods (Completed/Succeeded) are excluded so finished jobs are not falsely reported. Both the bash and PowerShell scripts use the same READY/STATUS/RESTARTS logic and were verified to flag identical rows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 56abebc8-418a-4684-ae16-b22c86d83e72

…cript (#2932) * feat: replace azure-diagnostics messaging connectivity probe with a script Adds cross-platform test-messaging-connectivity.{sh,ps1} scripts that probe DNS resolution, HTTPS reachability, and messaging TCP ports (AMQP 5671/5672, HTTPS 443, optional Kafka 9093) for a Service Bus / Event Hubs namespace, emitting one normalized report. Replaces the inline curl/nslookup probe in service-troubleshooting.md with markdown-linked references and examples. Fixes #2511 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * test: add eval covering messaging connectivity probe script Adds an output-content stimulus to evals/azure-diagnostics/eval.yaml that presents a 'cannot connect at all' Service Bus namespace scenario and asserts the skill surfaces the test-messaging-connectivity probe script, exercising the new reference added for #2511. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * fix: address PR review comments on messaging connectivity probe - bash --help no longer prints the shebang line (filter out #! lines) - PowerShell HTTPS check drops -SkipHttpErrorCheck (unavailable in Windows PowerShell 5.1) and extracts the HTTP status from the caught exception across both PS editions - service-troubleshooting.md links the scripts as markdown links and states the working directory (skill root) for the example commands - eval stimulus adds an earlyTerminate to cap cost and documents why area:output is intentional Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * fix: address PR review comments (Round 2) Remove [Parameter(Mandatory)] from the PowerShell script's Namespace param and replace it with an explicit empty check that exits 2, avoiding interactive prompts in non-interactive scenarios. Remove the global \Continue = 'Stop' so a stray non-terminating error no longer aborts the whole probe; the DNS/HTTPS/TCP blocks already use explicit -ErrorAction Stop inside try/catch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * test: assert agent attempts to run messaging connectivity probe Rework the azure-diagnostics connectivity-probe eval so it verifies the agent *attempts* to run test-messaging-connectivity (a shell tool call), rather than merely mentioning the script name in prose. Uses a tool-call-match early-terminate that fires on the attempt and the built-in tool-calls grader, mirroring the azure-validate pattern. No live namespace is required since we only observe the attempt, not the result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * test: make connectivity-probe eval drive the agent to run the script Validated locally against a live agent (5 runs, 93%). Two fixes from the local run: (1) reframe the prompt as a production-incident triage so it reliably routes to azure-diagnostics (5/5) instead of azure-messaging or no skill; (2) early-terminate on tool-call-result rather than tool-call-match, because vally's tool-calls grader only matches completed tool calls — terminating on the call start dropped the script's result before it could be graded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * chore: relocate messaging probe scripts after upstream tree reorg Upstream/main moved plugin/ -> plugins/azure-skills/ and evals/<skill>/ -> evals/azure-skills/<skill>/. The merge relocated modified files via rename detection, but the two new probe scripts (pure additions) had to be moved manually. Also update stale plugin/skills/azure-diagnostics path text in service-troubleshooting.md and migrate the connectivity-probe eval to the newly-added shell-command-invoked grader (terminating on the tool-call start). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * test: pin azure-diagnostics via requiredSkills in probe eval The multi-plugin restructure (#2872) changed the vally skill-loading model: without a requiredSkills tag every plugin skill is loaded and competes for routing, so the behavior probe test dropped to 0/5 (agent hand-rolled its own DNS probe instead of loading azure-diagnostics). Pin azure-diagnostics so its description survives char-budget truncation, matching the sibling-eval convention. Back to 5/5 skill-invocation + shell-command-invoked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c * fix: make messaging probe sh executable and invoke via bash in docs Address PR review: the committed .sh was mode 100644, so the documented ./scripts/test-messaging-connectivity.sh invocation would fail with a permission error on Linux/macOS after checkout. Set the executable bit (100755) and invoke the script through �ash in both bash examples so the quick connectivity test runs regardless of how the file mode survives the build copy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 08574dee-8dd0-4771-98ad-934a5553060c

github-advanced-security Bot found potential problems Jul 17, 2026

View reviewed changes

Comment thread gulpfile.ts Fixed

Comment thread gulpfile.ts Fixed

Comment thread scripts/src/frontmatter/helpers.ts Fixed

Comment thread scripts/src/references/cli.ts Fixed

JasonYeMSFT (JasonYeMSFT) added 29 commits July 17, 2026 13:18

move azure plugin files

c3b65a5

move azure eval files

af20fbe

2nd plugin for testing

7a19fc4

adapt build script

extend skill object in test code to carry plugin info

34e9e22

commit plan

480fd78

preserve changelog after moving plugin file during build

7b2789c

feature: cat emoji

83463bf

Keep azure-skills directory name

8f7fe1d

update plan

aa88a7a

plugin output dirname override

5849f9b

fix plugin selection in agent-runner

6848325

fix pr.yml for multi-plugin model

fix eval.yml

5d66b34

fix pr-plugin-version-check.yml

2913ab2

fix test-all-skills.yml

80abf31

fix sync-to-azure-mcp.yml fix publish-to-marketplace.yml

test scripts and notes

fdf0032

llm generated script todo

f1cc4d0

fix check-plugin-version-pr script

2ed4751

remove dead update-plugin-version script

37bf8a6

fix copilot cli char budget check script

3c5ec5b

use plugin dirname in code

27bbc7a

fix references check script

abb749f

fix tokens script

a4adb8c

fix frontmatter script

4bc70f4

move new foundry files

ecbdf91

give frontmatter cli a default check all option

b6da523

fix lint

1da9489

Remove invalid param in .vally.yaml

b4b1d71

clean up unused script

34cf0e5

rename eval dirname

c8ae35e

JasonYeMSFT (JasonYeMSFT) dismissed msalaman’s stale review via 2b88c72 July 23, 2026 18:08

JasonYeMSFT (JasonYeMSFT) added 2 commits July 23, 2026 11:36

remove cat plugin

2810c45

move azure-validate test fixture

0f5d1ab

Sai Koumudi Kaluvakolanu (saikoumudi) previously approved these changes Jul 23, 2026

View reviewed changes

Alex Thompson (alex-thompson) previously approved these changes Jul 23, 2026

View reviewed changes

Kashif Khan (kashifkhan) previously approved these changes Jul 23, 2026

View reviewed changes

Alex Okonechnikov (okonech) approved these changes Jul 23, 2026

View reviewed changes

remove unneeded npm scripts

725c450

JasonYeMSFT (JasonYeMSFT) dismissed stale reviews from Kashif Khan (kashifkhan), Alex Thompson (alex-thompson), and Sai Koumudi Kaluvakolanu (saikoumudi) via 725c450 July 23, 2026 20:08

chandraneel previously approved these changes Jul 23, 2026

View reviewed changes

Michael (micha31r) approved these changes Jul 23, 2026

View reviewed changes

Merge branch 'main' into chuye/multi-plugin-folder-structure

8d3c535

JasonYeMSFT (JasonYeMSFT) dismissed chandraneel’s stale review via 8d3c535 July 24, 2026 16:44

JasonYeMSFT (JasonYeMSFT) added 3 commits July 24, 2026 09:47

Merge branch 'main' into chuye/multi-plugin-folder-structure

6212471

remove stale file

7d90580

sync legacy changelog

74d58ee

Rick Winter (RickWinter) reviewed Jul 24, 2026

View reviewed changes

Comment thread tests/skills.json Outdated

Rick Winter (RickWinter) reviewed Jul 24, 2026

View reviewed changes

Comment thread .github/skills/vally-eval/references/ci-test.md

add trailing new line

360286c

Tom Meschter (tmeschter) approved these changes Jul 27, 2026

View reviewed changes

Rick Winter (RickWinter) approved these changes Jul 28, 2026

View reviewed changes

JasonYeMSFT (JasonYeMSFT) merged commit d3959f3 into microsoft:main Jul 28, 2026
13 checks passed

github-actions Bot mentioned this pull request Jul 30, 2026

[repo-status] Weekly Repo Status — July 23–30, 2026 #2994

Closed

Navigation Menu

Uh oh!

feature: multi plugin project structure - #2872

feature: multi plugin project structure#2872
JasonYeMSFT (JasonYeMSFT) merged 58 commits into
microsoft:mainfrom
JasonYeMSFT:chuye/multi-plugin-folder-structure

JasonYeMSFT (JasonYeMSFT) commented Jul 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alex Okonechnikov (okonech) commented Jul 23, 2026

Uh oh!

chandraneel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Uh oh!

Conversation

JasonYeMSFT (JasonYeMSFT) commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Concept

Folder structure changes

Development Tool adjustments

Workflow adjustments

User expectations

Checklist

Related Issues

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alex Okonechnikov (okonech) commented Jul 23, 2026

Uh oh!

chandraneel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

JasonYeMSFT (JasonYeMSFT) commented Jul 17, 2026 •

edited

Loading