feature: multi plugin project structure

.github/CODEOWNERS

@@ -7,7 +7,7 @@
 # Workflow files — explicit protection (changes require team review)
 /.github/workflows/ @microsoft/github-copilot-for-azure-writers
 
-# Plugin skills owners
+# Plugin skills owners (kept for reference)
 /plugin/skills/ @tmeschter @RickWinter
 /plugin/skills/airunway-aks-setup/ @tmeschter @RickWinter
 /plugin/skills/appinsights-instrumentation/ @JasonYeMSFT @RickWinter
@@ -43,3 +43,40 @@
 /plugin/skills/microsoft-foundry/foundry-agent/routine/ @anchenyi @XiaofuHuang @swatDong @RickWinter
 /plugin/skills/microsoft-foundry/foundry-agent/invocations-ws/ @anchenyi @XiaofuHuang @swatDong @RickWinter
 /plugin/skills/python-appservice-deploy/ @glaming1 @tmeschter @RickWinter
+
+# Plugin skills owners (multi-plugin)
+/plugins/azure-skills/skills/ @tmeschter @RickWinter
+/plugins/azure-skills/skills/airunway-aks-setup/ @tmeschter @RickWinter
+/plugins/azure-skills/skills/appinsights-instrumentation/ @JasonYeMSFT @RickWinter
+/plugins/azure-skills/skills/azure-ai/ @JasonYeMSFT @RickWinter
+/plugins/azure-skills/skills/azure-aigateway/ @azaslonov @RickWinter
+/plugins/azure-skills/skills/azure-cloud-migrate/ @saikoumudi @MadhuraBharadwaj-MSFT @RickWinter
+/plugins/azure-skills/skills/azure-compliance/ @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-compute/ @alex-thompson @rakal-dyh @joybb @rmmue21 @RickWinter
+/plugins/azure-skills/skills/azure-cost/ @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-deploy/ @microsoft/github-copilot-for-azure-writers @paulyuk
+/plugins/azure-skills/skills/azure-diagnostics/ @tmeschter @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-enterprise-infra-planner/ @Jbrocket @micha31r @arunrab @RickWinter
+/plugins/azure-skills/skills/azure-kubernetes/ @saikoumudi @chandraneel @gambtho @RickWinter
+/plugins/azure-skills/skills/azure-kusto/ @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-messaging/ @kashifkhan @RickWinter
+/plugins/azure-skills/skills/azure-prepare/ @microsoft/github-copilot-for-azure-writers
+/plugins/azure-skills/skills/azure-quotas/ @rakal-dyh @RickWinter
+/plugins/azure-skills/skills/azure-reliability/ @MadhuraBharadwaj-MSFT @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-resource-lookup/ @JasonYeMSFT @RickWinter
+/plugins/azure-skills/skills/azure-resource-visualizer/ @tmeschter @RickWinter
+/plugins/azure-skills/skills/azure-storage/ @JasonYeMSFT @RickWinter
+/plugins/azure-skills/skills/azure-upgrade/ @MadhuraBharadwaj-MSFT @saikoumudi @RickWinter
+/plugins/azure-skills/skills/azure-validate/ @microsoft/github-copilot-for-azure-writers
+/plugins/azure-skills/skills/entra-agent-id/ @ArLucaID @RickWinter
+/plugins/azure-skills/skills/entra-app-registration/ @JasonYeMSFT @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/ @ankitbko @tendau @XOEEst @anchenyi @XiaofuHuang @jugonzales @vebudumu @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/cicd/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/create/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/deploy/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/invoke/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/toolbox/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/troubleshoot/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/routine/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/microsoft-foundry/foundry-agent/invocations-ws/ @anchenyi @XiaofuHuang @swatDong @RickWinter
+/plugins/azure-skills/skills/python-appservice-deploy/ @glaming1 @tmeschter @RickWinter

.github/skills/vally-eval/SKILL.md

@@ -17,31 +17,31 @@ Vally eval suites are written as yaml documents. All eval suites share eval spec
 
 Refer to the official documentation on the schema of the spec and the schema of the eval suites [writing-eval-specs](https://microsoft.github.io/vally/guides/writing-eval-specs/).
 
-Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at `<repo-root>/.vally.yaml`. The eval suites are categorized by skills. The eval suites for each skill are located at `<repo-root>/evals/<skill-name>/*.yaml`, e.g. `<repo-root>/evals/azure-ai/eval.yaml`. 
+Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at `<repo-root>/.vally.yaml`. The eval suites are categorized by plugin and skills. The eval suites for each skill are located at `<repo-root>/evals/<plugin-dirname>/<skill-name>/*.yaml`, e.g. `<repo-root>/evals/azure-skills/azure-ai/eval.yaml`.
 
-Use meaningful file names to categorize tests. If a skill needs fixture files for its eval suites, it should organize such fixture files in a `fixture` directory under its directory, e.g. `<repo-root>/evals/azure-ai/fixture/`. The [vally test runner](/tests/run-vally-test.ts) and [stimulus validation script](/scripts/src/vally/validate-stimulus.ts) will load all `*.yaml` files except for those under a `fixture/` directory. Make sure to put all fixture files under the `fixture/` directory.
+Use meaningful file names to categorize tests. If a skill needs fixture files for its eval suites, it should organize such fixture files in a `fixture` directory under its directory, e.g. `<repo-root>/evals/azure-skills/azure-ai/fixture/`. The [vally test runner](/tests/run-vally-test.ts) and [stimulus validation script](/scripts/src/vally/validate-stimulus.ts) will load all `*.yaml` files except for those under a `fixture/` directory. Make sure to put all fixture files under the `fixture/` directory.
 
 ## Migrate integration tests
 
 azure-skills plugin have implemented JavaScript integration test using Jest as the underlying test runner. All such integration tests are under `tests/**/integration.test.ts` files.
 
-To migrate integration test for a skill to vally suites, create its eval suite spec at `<repo-root>/evals/<skill-name>/eval.yaml`, add a suite that runs the same prompt and uses vally's built-in graders to grade the trajectory of the agent run. If the integration test grades the agent run in a way that vally's built-in graders don't support, refer to the official documentation on how to create a custom grader [writing-custom-grader](https://microsoft.github.io/vally/guides/writing-custom-graders/).
+To migrate integration test for a skill to vally suites, create its eval suite spec at `<repo-root>/evals/<plugin-dirname>/<skill-name>/eval.yaml`, add a suite that runs the same prompt and uses vally's built-in graders to grade the trajectory of the agent run. If the integration test grades the agent run in a way that vally's built-in graders don't support, refer to the official documentation on how to create a custom grader [writing-custom-grader](https://microsoft.github.io/vally/guides/writing-custom-graders/).
 
 ## Why is there a custom executor
 
-The legacy Jest based integration test framework implemented features that vally doesn't support yet, such as early termination, follow up, system prompt modification, screenshot taking, etc. Besides, the azure-skills plugin runs automated integration tests, collects its exported data and feeds the data to a dashboard web app under `<repo-root>/dashboard/` to monitor skill integration test results.
+The legacy Jest based integration test framework implemented features that vally doesn't support yet, such as early termination, follow up, system prompt modification, screenshot taking, etc. Besides, [test-all-integration](/.github/workflows/test-all-integration.yml) runs automated integration tests, collects its exported data and feeds the data to a dashboard web app under `<repo-root>/dashboard/` to monitor skill integration test results.
 
 If you intend to have your vally suites use any of the extended features or have their results be consumed by the dashboard, you **MUST** use the custom executor in your vally suites.
 
 ### Use tags to control the custom executor
 
-The custom executor in azure-skills plugin uses special tag values to control the behavior of the custom executor. See [tag-helpers.ts](../../../tests/vally/tag-helpers.ts) to learn what special tags are supported.
+The custom executor uses special tag values to control the behavior of the custom executor. See [tag-helpers.ts](../../../tests/vally/tag-helpers.ts) to learn what special tags are supported.
 
 > Note: If an eval suite specifies an earlyTerminate condition, the suite MUST NOT use the `completed` grader because early terminated runs will always fail the `completed` grader by design.
 
 ## Validate vally eval suites
 
-Vally eval suites for azure-skills plugin follow certain conventions. For example, all eval suites must have a `type`, `tier`, `cost` and `area` tag so they can be run for a corresponding target group. To ensure all eval suites follow the conventions, a script is added to validate the eval suites and report errors when it sees any violation. To run the script, execute this command from the `scripts/` directory.
+Vally eval suites in this repo follow certain conventions. For example, all eval suites must have a `type`, `tier`, `cost` and `area` tag so they can be run for a corresponding target group. To ensure all eval suites follow the conventions, a script is added to validate the eval suites and report errors when it sees any violation. To run the script, execute this command from the `scripts/` directory.
 
 ```bash
 # cwd as <repo-root>/scripts/
@@ -56,10 +56,10 @@ Use vally-cli to run vally eval suites. In most cases, you would like to use a c
 
 ```bash
 # In tests/
-npm run test:vally -- --skill $SKILL
+npm run test:vally -- --plugin $PLUGIN_DIR --skill $SKILL
 ```
 
-`--eval-spec ../evals/<skill-name>/eval.yaml` tells vally which eval spec to run. The path is relative to the current working directory of the process running the command. `--output-dir ./results` tells vally to write its output to a `results/` directory relative to the current working directory of the process running the command. `--executor-plugin ../../tests/vally/vally-executor.ts` tells vally to load and execute the code in this module, which registers the custom executor used by azure-skill vally eval suites. Note that this path is relative to the parent directory of the eval spec to run. For example, if the eval spec to run is `<repo-root>/evals/azure-ai/eval.yaml`, resolving this relative path ends at `<repo-root>/tests/vally/vally-executor.ts`.
+See [vally test runner](/tests/run-vally-test.ts) on how it composes the vally commands under the hood.
 
 ## Run vally eval suites in CI
 
@@ -75,12 +75,20 @@ Test authors commonly need to fine-tune grader configurations to reduce result f
 
 ```bash
 # in tests/
-npx @microsoft/vally-cli grade --eval-spec ../evals/<skill-name>/eval.yaml --verbose < results/<test-run-name>/results.jsonl
+npx @microsoft/vally-cli grade --eval-spec ../evals/<plugin-dirname>/<skill-name>/eval.yaml --verbose < results/<test-run-name>/results.jsonl
 ```
 
 You can keep tuning the grader config in `eval.yaml` and re-grade the trajectory until the results meet your expectations.
 
-If your skill uses a custom grader, add `--grader-plugin ./tests/vally/vally-graders.ts` to load the custom graders.
+If your skill uses a custom grader, add `--grader-plugin` to load the custom graders.
+
+```bash
+# in tests/
+npx @microsoft/vally-cli grade --eval-spec ../evals/<plugin-dirname>/<skill-name>/eval.yaml --grader-plugin
+../../../tests/vally/vally-graders.ts --verbose < results/<test-run-name>/results.jsonl
+```
+
+Note that the grader plugin's path is relative to the parent directory of the eval spec to run. For example, if the eval spec to run is `<repo-root>/evals/azure-skills/azure-ai/eval.yaml`, resolving this relative path ends at `<repo-root>/tests/vally/vally-executor.ts`.
 
 ### Collect test results
 

.github/skills/vally-eval/references/ci-test.md

@@ -4,7 +4,7 @@ Follow these steps to add a skill's Vally suites to the CI test workflow so they
 
 ## Prerequisites
 
-- The skill's Vally suites are implemented under `evals/<skill-name>/eval.yaml` (or split across multiple YAML files).
+- The skill's Vally suites are implemented under `evals/<plugin-dirname>/<skill-name>/eval.yaml` (or split across multiple YAML files).
 - The Vally suites use the `integration-test-agent-runner` custom executor.
 - The test results can be made public.
 
@@ -14,9 +14,7 @@ The scheduled CI test workflow determines which skills to test by reading `tests
 
 ### Use shared job template
 
-Most skills use a shared job template to run eval suites. This template is defined as the `test` job in `.github/workflows/test-all-integration.yml`.
-
-If you use the shared job template, add the skill in the workflow’s `VALLY_SKILLS` list. Otherwise the job will run Jest-based integration tests instead of `npm run test:vally`. The CI workflow creates one job per skill from this template and runs all eval suites with `npm run test:vally`.
+Most skills use a shared job template to run eval suites. This template is defined as the `test` job in `.github/workflows/test-all-integration.yml`. The CI workflow creates one job per skill from this template and runs all eval suites with `npm run test:vally`.
 
 Reuse this template whenever possible. It provisions a test environment, installs common tools (for example, Azure CLI and Azure Developer CLI), connects to a test Azure subscription, and includes utility steps that collect and publish test results to a well-known storage location for downstream processing.
 

.github/workflows/eval.yml

@@ -4,7 +4,7 @@ on:
   pull_request:
     paths:
       - "evals/**"
-      - "plugin/skills/**"
+      - "plugins/*/skills/**"
       - ".vally.yaml"
   workflow_dispatch:
     inputs:
@@ -55,7 +55,7 @@ jobs:
           npm run vally validate-stimulus
 
       - name: Lint skill and eval specs
-        run: npx --yes @microsoft/vally-cli@^0.7.0 lint plugin/skills/ --eval-spec evals/ --strict --grader-plugin ./tests/vally/vally-graders.ts
+        run: npx --yes @microsoft/vally-cli@^0.7.0 lint plugins/ --eval-spec evals/ --strict --grader-plugin ./tests/vally/vally-graders.ts
 
       - name: Determine suite
         if: github.event_name != 'pull_request'

.github/workflows/pr-plugin-version-check.yml

@@ -4,9 +4,9 @@ on:
   pull_request:
     branches: [main]
     paths:
-      - "plugin/.plugin/plugin.json"
-      - "plugin/.claude-plugin/plugin.json"
-      - "plugin/.cursor-plugin/plugin.json"
+      - "plugins/*/.plugin/plugin.json"
+      - "plugins/*/.claude-plugin/plugin.json"
+      - "plugins/*/.cursor-plugin/plugin.json"
 
 permissions:
   contents: read