From 93787a0733e1d55c1e0fbc7379d20929b90c03ff Mon Sep 17 00:00:00 2001 From: Chunan Ye Date: Mon, 18 May 2026 10:16:22 -0700 Subject: [PATCH 1/2] Delete unit tests --- .github/agents/SkillCreator.agent.md | 8 +- .github/copilot-instructions.md | 1 - .../references/review-checklist.md | 1 - docs/spec/azure-hosted-copilot-sdk.md | 2 - tests/AGENTS.md | 29 +- tests/README.md | 71 +- tests/_template/README.md | 5 - tests/_template/unit.test.ts | 82 -- tests/airunway-aks-setup/unit.test.ts | 115 -- .../appinsights-instrumentation/unit.test.ts | 85 -- tests/azure-ai/unit.test.ts | 90 -- tests/azure-aigateway/unit.test.ts | 176 --- tests/azure-cloud-migrate/unit.test.ts | 143 -- tests/azure-compliance/unit.test.ts | 168 --- tests/azure-compute/unit.test.ts | 1285 ----------------- tests/azure-cost/unit.test.ts | 345 ----- tests/azure-deploy/unit.test.ts | 108 -- tests/azure-diagnostics/unit.test.ts | 165 --- .../unit.test.ts | 210 --- tests/azure-hosted-copilot-sdk/unit.test.ts | 149 -- .../unit.test.ts | 146 -- tests/azure-kubernetes/unit.test.ts | 192 --- tests/azure-messaging/unit.test.ts | 98 -- tests/azure-prepare/unit.test.ts | 143 -- tests/azure-quotas/unit.test.ts | 185 --- tests/azure-rbac/unit.test.ts | 134 -- tests/azure-reliability/unit.test.ts | 245 ---- tests/azure-resource-lookup/unit.test.ts | 96 -- tests/azure-resource-visualizer/unit.test.ts | 88 -- tests/azure-upgrade/unit.test.ts | 174 --- tests/azure-validate/unit.test.ts | 90 -- tests/entra-agent-id/unit.test.ts | 71 - tests/entra-app-registration/unit.test.ts | 63 - tests/mcp-tool-references/unit.test.ts | 142 -- .../foundry-agent/create/unit.test.ts | 89 -- .../foundry-agent/deploy/unit.test.ts | 209 --- .../foundry-agent/eval-datasets/unit.test.ts | 236 --- .../foundry-agent/invoke/unit.test.ts | 80 - .../foundry-agent/observe/unit.test.ts | 383 ----- .../foundry-agent/trace/unit.test.ts | 100 -- .../foundry-agent/troubleshoot/unit.test.ts | 41 - .../models/deploy/capacity/unit.test.ts | 92 -- .../deploy/customize-deployment/unit.test.ts | 88 -- .../deploy-model-optimal-region/unit.test.ts | 48 - .../models/deploy/deploy-model/unit.test.ts | 105 -- .../resource/create/unit.test.ts | 216 --- tests/microsoft-foundry/unit.test.ts | 343 ----- tests/package.json | 2 - tests/scripts/generate-quality-report.js | 1013 ------------- tests/scripts/run-tests.js | 7 - tests/utils/__tests__/unit.test.ts | 87 -- 51 files changed, 7 insertions(+), 8237 deletions(-) delete mode 100644 tests/_template/unit.test.ts delete mode 100644 tests/airunway-aks-setup/unit.test.ts delete mode 100644 tests/appinsights-instrumentation/unit.test.ts delete mode 100644 tests/azure-ai/unit.test.ts delete mode 100644 tests/azure-aigateway/unit.test.ts delete mode 100644 tests/azure-cloud-migrate/unit.test.ts delete mode 100644 tests/azure-compliance/unit.test.ts delete mode 100644 tests/azure-compute/unit.test.ts delete mode 100644 tests/azure-cost/unit.test.ts delete mode 100644 tests/azure-deploy/unit.test.ts delete mode 100644 tests/azure-diagnostics/unit.test.ts delete mode 100644 tests/azure-enterprise-infra-planner/unit.test.ts delete mode 100644 tests/azure-hosted-copilot-sdk/unit.test.ts delete mode 100644 tests/azure-kubernetes/azure-kubernetes-automatic-readiness/unit.test.ts delete mode 100644 tests/azure-kubernetes/unit.test.ts delete mode 100644 tests/azure-messaging/unit.test.ts delete mode 100644 tests/azure-prepare/unit.test.ts delete mode 100644 tests/azure-quotas/unit.test.ts delete mode 100644 tests/azure-rbac/unit.test.ts delete mode 100644 tests/azure-reliability/unit.test.ts delete mode 100644 tests/azure-resource-lookup/unit.test.ts delete mode 100644 tests/azure-resource-visualizer/unit.test.ts delete mode 100644 tests/azure-upgrade/unit.test.ts delete mode 100644 tests/azure-validate/unit.test.ts delete mode 100644 tests/entra-agent-id/unit.test.ts delete mode 100644 tests/entra-app-registration/unit.test.ts delete mode 100644 tests/mcp-tool-references/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/create/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/deploy/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/eval-datasets/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/invoke/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/observe/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/trace/unit.test.ts delete mode 100644 tests/microsoft-foundry/foundry-agent/troubleshoot/unit.test.ts delete mode 100644 tests/microsoft-foundry/models/deploy/capacity/unit.test.ts delete mode 100644 tests/microsoft-foundry/models/deploy/customize-deployment/unit.test.ts delete mode 100644 tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/unit.test.ts delete mode 100644 tests/microsoft-foundry/models/deploy/deploy-model/unit.test.ts delete mode 100644 tests/microsoft-foundry/resource/create/unit.test.ts delete mode 100644 tests/microsoft-foundry/unit.test.ts delete mode 100644 tests/scripts/generate-quality-report.js delete mode 100644 tests/utils/__tests__/unit.test.ts diff --git a/.github/agents/SkillCreator.agent.md b/.github/agents/SkillCreator.agent.md index 114ab9f3a..479040b75 100644 --- a/.github/agents/SkillCreator.agent.md +++ b/.github/agents/SkillCreator.agent.md @@ -51,16 +51,12 @@ REQUIREMENTS.md should _always_ include the following: When creating a new skill, tests must be created following the patterns documented in `/tests/AGENTS.md`. The test suite should include: -1. **Unit Tests** (`tests/{skill-name}/unit.test.js`): - - Validate SKILL.md metadata - - Test any validation or processing logic - -2. **Trigger Tests** (`tests/{skill-name}/triggers.test.js`): +1. **Trigger Tests** (`tests/{skill-name}/triggers.test.js`): - At least 5 prompts that SHOULD trigger the skill - At least 5 prompts that should NOT trigger the skill - Snapshot tests for keyword changes -3. **Integration Tests** (`tests/{skill-name}/integration.test.js`) - if applicable: +2. **Integration Tests** (`tests/{skill-name}/integration.test.js`) - if applicable: - Mock MCP tool interactions - Test error handling diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c0e15895d..f0554cea7 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -72,7 +72,6 @@ cd tests npm install npm test # Run all tests npm test -- --testPathPatterns= # Run tests for a single skill -npm run test:unit # Unit tests only npm run typecheck # TypeScript type checking npm run lint # ESLint ``` diff --git a/.github/skills/skill-reviewer/references/review-checklist.md b/.github/skills/skill-reviewer/references/review-checklist.md index f95c04172..b3ae97f32 100644 --- a/.github/skills/skill-reviewer/references/review-checklist.md +++ b/.github/skills/skill-reviewer/references/review-checklist.md @@ -89,7 +89,6 @@ See [routing-analysis.md](routing-analysis.md) for detailed checks. |-------|------| | Registered | Skill listed in `tests/skills.json` | | Trigger tests | `tests/{skill-name}/triggers.test.ts` exists with shouldTrigger/shouldNotTrigger arrays | -| Unit tests | `tests/{skill-name}/unit.test.ts` exists | | Snapshots | Updated if description changed | ## 10. Positive Acknowledgment diff --git a/docs/spec/azure-hosted-copilot-sdk.md b/docs/spec/azure-hosted-copilot-sdk.md index 5cfd6ed3d..de5c68944 100644 --- a/docs/spec/azure-hosted-copilot-sdk.md +++ b/docs/spec/azure-hosted-copilot-sdk.md @@ -54,7 +54,6 @@ The `azure-hosted-copilot-sdk` skill enables users to build, deploy, and configu | `tests/utils/agent-runner.ts` | Enhanced with `runConversation()` for multi-turn scenarios | ✅ | | `tests/utils/evaluate.ts` | Shared evaluation helpers (`matchesCommand`, `getAllAssistantMessages`, etc.) | ✅ | | `tests/utils/regression-detectors.ts` | Regression detectors (secrets, ACR spirals, port confusion, etc.) | ✅ | -| `tests/_template/unit.test.ts` | Unit test template for new skills | ✅ | ### Local Dev Tooling — 🔄 Replaced @@ -167,7 +166,6 @@ docs/spec/ tests/azure-hosted-copilot-sdk/ ├── integration.test.ts # 8 integration tests ├── triggers.test.ts # 5 test blocks (23 parameterized cases) -├── unit.test.ts # 17 unit tests └── __snapshots__/triggers.test.ts.snap # Trigger keyword snapshots tests/utils/ diff --git a/tests/AGENTS.md b/tests/AGENTS.md index b76a70263..1f4fbfbb1 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -45,15 +45,12 @@ Based on the skill's description and content, add to `triggers.test.ts`: - Different cloud providers ("AWS", "GCP") - Different Azure services not covered by this skill -### Step 5: Add skill-specific unit tests -In `unit.test.ts`, add tests that verify the skill's content contains expected sections, commands, or patterns documented in its SKILL.md. - -### Step 6: Configure integration tests (optional) +### Step 5: Configure integration tests (optional) In `integration.test.ts`, customize the prompts to test real agent behavior. Follow existing `integration.test.ts` files for how to implement such tests. -### Step 7: Run and verify +### Step 6: Run and verify ```bash cd tests @@ -74,7 +71,6 @@ npm run coverage:grid ## Overview This testing framework uses **Jest** to validate skill behavior across three test categories: -- **Unit Tests** - Isolated logic testing - **Trigger Tests** - Skill activation validation - **Integration Tests** - MCP tool interaction testing @@ -108,26 +104,6 @@ tests/{skill-name}/ --- -## Writing Unit Tests - -Unit tests validate skill metadata and any extractable logic. - -### Required Tests - -Every skill should have these basic unit tests: - -```typescript -describe('Skill Metadata', () => { - test('has valid SKILL.md with required fields', () => { - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); -}); -``` - ---- - ## Writing Trigger Tests Trigger tests verify that prompts correctly activate (or don't activate) your skill. @@ -392,7 +368,6 @@ When adding tests for a new skill: - [ ] Update `SKILL_NAME` in all test files - [ ] Add 5+ prompts that should trigger - [ ] Add 5+ prompts that should NOT trigger -- [ ] Add unit tests for any validation logic - [ ] Run tests locally and verify passing - [ ] Update coverage grid if significant changes diff --git a/tests/README.md b/tests/README.md index 9c60c3f4e..06cf158e2 100644 --- a/tests/README.md +++ b/tests/README.md @@ -43,7 +43,6 @@ Each skill in `/plugin/skills/{skill-name}/` can have a corresponding test suite There are 3 types of tests. -- unit test: tests that validate the file structure of a skill and the content. - trigger test: tests that validate if the description of a skill can trigger or not trigger a given prompt using a heuristic. - integration test: tests that validate if the skill can lead to successful completion of a task by running a given prompt against a Copilot SDK agent. @@ -71,17 +70,7 @@ Run tests manually anytime during development (see [Running Tests Locally](#runn ## What Tests Validate -### 1. Unit Tests (`unit.test.ts`) - -**Purpose:** Validate skill metadata and any embedded logic. - -**What it checks:** -- `SKILL.md` exists and has valid frontmatter (`name`, `description`) -- Description is meaningful (not empty, appropriate length) -- Content contains expected sections -- Any validation rules documented in the skill work correctly - -### 2. Trigger Tests (`triggers.test.ts`) +### 1. Trigger Tests (`triggers.test.ts`) **Purpose:** Verify the skill activates on correct prompts and ignores unrelated ones. @@ -96,7 +85,7 @@ Run tests manually anytime during development (see [Running Tests Locally](#runn npm run update:snapshots -- --testPathPatterns={skill-name} ``` -### 3. Integration Tests (`integration.test.ts`) +### 2. Integration Tests (`integration.test.ts`) **Purpose:** Test skill behavior with a real Copilot agent session. @@ -322,66 +311,12 @@ const shouldNotTriggerPrompts = [ ]; ``` -#### Step 4: Customize Unit Tests - -In `unit.test.ts`, add tests specific to your skill's content: - -```typescript -test('documents cache tiers', () => { - expect(skill.content).toContain('Basic'); - expect(skill.content).toContain('Standard'); - expect(skill.content).toContain('Premium'); -}); -``` - -#### Step 5: Run and Verify +#### Step 4: Run and Verify ```bash npm run test:skill -- {skill-name} ``` -#### Step 6: Update Coverage Grid - -```bash -npm run coverage:grid -``` - -This updates the Skills Coverage Grid in this README. - ---- - -## Skills Coverage Grid - - -| Skill | Tests | Unit | Triggers | Integration | Coverage | -|-------|-------|------|----------|-------------|----------| -| appinsights-instrumentation | ✅ | ✅ | ✅ | ✅ | - | -| azure-ai | ✅ | ✅ | ✅ | ✅ | - | -| azure-aigateway | ✅ | ✅ | ✅ | ✅ | - | -| azure-cloud-migrate | ✅ | ✅ | ✅ | ✅ | - | -| azure-compliance | ✅ | ✅ | ✅ | ✅ | - | -| azure-compute | ✅ | ✅ | ✅ | ✅ | - | -| azure-cost | ✅ | ✅ | ✅ | ✅ | - | -| azure-deploy | ✅ | ✅ | ✅ | ✅ | - | -| azure-diagnostics | ✅ | ✅ | ✅ | ✅ | - | -| azure-enterprise-infra-planner | ✅ | ✅ | ✅ | ✅ | - | -| azure-hosted-copilot-sdk | ✅ | ✅ | ✅ | ✅ | - | -| azure-kusto | ✅ | - | - | ✅ | - | -| azure-messaging | ✅ | ✅ | ✅ | ✅ | - | -| azure-prepare | ✅ | ✅ | ✅ | ✅ | - | -| azure-quotas | ✅ | ✅ | ✅ | ✅ | - | -| azure-rbac | ✅ | ✅ | ✅ | ✅ | - | -| azure-resource-lookup | ✅ | ✅ | ✅ | ✅ | - | -| azure-resource-visualizer | ✅ | ✅ | ✅ | ✅ | - | -| azure-storage | ✅ | - | - | ✅ | - | -| azure-upgrade | ✅ | ✅ | ✅ | ✅ | - | -| azure-validate | ✅ | ✅ | ✅ | ✅ | - | -| entra-app-registration | ✅ | ✅ | ✅ | ✅ | - | -| microsoft-foundry | ✅ | ✅ | ✅ | ✅ | - | - -**Legend:** ✅ Exists | ❌ Missing | Coverage shown as percentage - - --- ## Troubleshooting diff --git a/tests/_template/README.md b/tests/_template/README.md index 62dfc9a0c..485fe1fcd 100644 --- a/tests/_template/README.md +++ b/tests/_template/README.md @@ -42,11 +42,6 @@ your-skill-name/ ## Test Types -### Unit Tests (`unit.test.js`) -- Test skill metadata (SKILL.md parsing) -- Test validation logic -- Test utility functions - ### Trigger Tests (`triggers.test.js`) - Verify correct prompts activate the skill - Verify unrelated prompts don't activate diff --git a/tests/_template/unit.test.ts b/tests/_template/unit.test.ts deleted file mode 100644 index 5f4d9a050..000000000 --- a/tests/_template/unit.test.ts +++ /dev/null @@ -1,82 +0,0 @@ -/** - * Unit Tests for {SKILL_NAME} - * - * Test isolated skill logic and validation rules. - * Copy this file to /tests/{skill-name}/unit.test.ts - */ - -import { readFileSync } from "node:fs"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -// Replace with your skill name -const SKILL_NAME = "your-skill-name"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - // Descriptions should be 50-500 chars for readability - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(500); - }); - - test("description contains trigger phrases", () => { - // Descriptions should contain keywords that help with skill activation - const description = skill.metadata.description.toLowerCase(); - const hasTriggerPhrases = - description.includes("use this") || - description.includes("use when") || - description.includes("helps") || - description.includes("activate") || - description.includes("trigger"); - expect(hasTriggerPhrases).toBe(true); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - const supported = ["name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable"]; - // Extract top-level keys (lines starting with a word followed by colon) - const keys = frontmatter.split("\n") - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - - test("WHEN clause is inside description", () => { - // These must be embedded in the description string, not parsed as YAML keys - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - }); -}); diff --git a/tests/airunway-aks-setup/unit.test.ts b/tests/airunway-aks-setup/unit.test.ts deleted file mode 100644 index d82b80132..000000000 --- a/tests/airunway-aks-setup/unit.test.ts +++ /dev/null @@ -1,115 +0,0 @@ -import * as fs from "node:fs"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "airunway-aks-setup"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1000); - }); - - test("description mentions AI Runway and AKS", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/ai runway|airunway/); - expect(desc).toMatch(/aks|cluster/); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = fs.readFileSync(skill.filePath, "utf-8"); - // Extract content between the opening and closing --- delimiters - const frontmatterMatch = raw.match(/^-{3}\n([\s\S]*?)\n-{3}/); - const frontmatter = frontmatterMatch ? frontmatterMatch[1] : ""; - // Fail loudly if frontmatter extraction failed — prevents silent vacuous pass - expect(frontmatter.length).toBeGreaterThan(0); - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const supported = ["name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable"]; - // Use parsed metadata keys rather than re-parsing the raw YAML - const metadataKeys = Object.keys(skill.metadata); - for (const key of metadataKeys) { - expect(supported).toContain(key); - } - }); - - test("WHEN clause is inside description", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - }); - - describe("Required Section Structure", () => { - test("contains the standard skill sections", () => { - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## Prerequisites"); - expect(skill.content).toContain("## Rules"); - expect(skill.content).toContain("## Steps"); - expect(skill.content).toContain("## Error Handling"); - }); - }); - - describe("Six-Phase Workflow", () => { - test("covers all six setup phases", () => { - expect(skill.content).toMatch(/Cluster Verification/i); - expect(skill.content).toMatch(/Controller Installation/i); - expect(skill.content).toMatch(/GPU Assessment/i); - expect(skill.content).toMatch(/Provider Setup/i); - expect(skill.content).toMatch(/First Deployment/i); - expect(skill.content).toMatch(/Summary/i); - }); - - test("references step detail files", () => { - expect(skill.content).toMatch(/step-1-verify\.md/i); - expect(skill.content).toMatch(/step-2-controller\.md/i); - expect(skill.content).toMatch(/step-3-gpu\.md/i); - expect(skill.content).toMatch(/step-4-provider\.md/i); - expect(skill.content).toMatch(/step-5-deploy\.md/i); - expect(skill.content).toMatch(/step-6-summary\.md/i); - }); - }); - - describe("Quick Reference Section", () => { - test("lists kubectl as a CLI tool", () => { - expect(skill.content).toMatch(/kubectl/); - }); - - test("lists make as a CLI tool", () => { - expect(skill.content).toMatch(/make/i); - }); - - test("lists curl as a CLI tool", () => { - expect(skill.content).toMatch(/curl/i); - }); - }); - - describe("Troubleshooting Reference", () => { - test("links to troubleshooting reference", () => { - expect(skill.content).toMatch(/troubleshooting\.md/i); - }); - }); -}); diff --git a/tests/appinsights-instrumentation/unit.test.ts b/tests/appinsights-instrumentation/unit.test.ts deleted file mode 100644 index 294930719..000000000 --- a/tests/appinsights-instrumentation/unit.test.ts +++ /dev/null @@ -1,85 +0,0 @@ -/** - * Unit Tests for appinsights-instrumentation - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "appinsights-instrumentation"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description mentions instrumentation or telemetry", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/instrument|telemetry|app insights|observability/); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(500); - }); - - test("contains when to use section", () => { - expect(skill.content).toMatch(/## When to use/i); - }); - - test("documents prerequisites", () => { - expect(skill.content).toMatch(/## Prerequisites/i); - expect(skill.content).toContain("ASP.NET Core"); - expect(skill.content).toContain("Node.js"); - }); - - test("documents guidelines section", () => { - expect(skill.content).toMatch(/## Guidelines/i); - }); - - test("mentions auto-instrumentation", () => { - expect(skill.content).toContain("auto-instrument"); - expect(skill.content).toContain("auto.md"); - }); - - test("references manual instrumentation guides", () => { - expect(skill.content).toContain("aspnetcore.md"); - expect(skill.content).toContain("nodejs.md"); - expect(skill.content).toContain("python.md"); - }); - - test("mentions creating AppInsights resource", () => { - expect(skill.content).toContain("AppInsights resource"); - expect(skill.content).toMatch(/Bicep|Azure CLI/); - }); - - test("references example files", () => { - expect(skill.content).toContain("examples/appinsights.bicep"); - expect(skill.content).toContain("scripts/appinsights.ps1"); - }); - }); - - describe("Supported Technologies", () => { - test("lists ASP.NET Core as supported", () => { - expect(skill.content).toContain("ASP.NET Core"); - }); - - test("lists Node.js as supported", () => { - expect(skill.content).toContain("Node.js"); - }); - - test("lists Python as supported", () => { - expect(skill.content).toContain("Python"); - }); - }); -}); diff --git a/tests/azure-ai/unit.test.ts b/tests/azure-ai/unit.test.ts deleted file mode 100644 index 7b106d37f..000000000 --- a/tests/azure-ai/unit.test.ts +++ /dev/null @@ -1,90 +0,0 @@ -/** - * Unit Tests for azure-ai - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-ai"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description mentions Azure AI services", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/azure ai|ai search|speech|openai|document intelligence/); - }); - - test("description includes WHEN guidance", () => { - expect(skill.metadata.description).toContain("WHEN:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(500); - }); - - test("documents services table", () => { - expect(skill.content).toMatch(/## Services/i); - expect(skill.content).toContain("AI Search"); - expect(skill.content).toContain("Speech"); - expect(skill.content).toContain("OpenAI"); - expect(skill.content).toContain("Document Intelligence"); - }); - - test("documents MCP Server section", () => { - expect(skill.content).toMatch(/## MCP Server/i); - expect(skill.content).toContain("azure__search"); - expect(skill.content).toContain("azure__speech"); - }); - }); - - describe("AI Search Coverage", () => { - test("documents AI Search capabilities", () => { - expect(skill.content).toMatch(/## AI Search Capabilities/i); - expect(skill.content).toContain("Full-text search"); - expect(skill.content).toContain("Vector search"); - expect(skill.content).toContain("Hybrid search"); - }); - - test("lists AI Search MCP commands", () => { - expect(skill.content).toContain("search_index_list"); - expect(skill.content).toContain("search_index_get"); - expect(skill.content).toContain("search_query"); - }); - }); - - describe("Speech Coverage", () => { - test("documents Speech capabilities", () => { - expect(skill.content).toMatch(/## Speech Capabilities/i); - expect(skill.content).toContain("Speech-to-text"); - expect(skill.content).toContain("Text-to-speech"); - }); - - test("lists Speech MCP commands", () => { - expect(skill.content).toContain("speech_transcribe"); - expect(skill.content).toContain("speech_synthesize"); - }); - }); - - describe("Documentation Links", () => { - test("references Azure documentation", () => { - expect(skill.content).toContain("learn.microsoft.com"); - expect(skill.content).toContain("azure/search"); - expect(skill.content).toContain("speech-service"); - }); - }); -}); diff --git a/tests/azure-aigateway/unit.test.ts b/tests/azure-aigateway/unit.test.ts deleted file mode 100644 index 0a8bd6247..000000000 --- a/tests/azure-aigateway/unit.test.ts +++ /dev/null @@ -1,176 +0,0 @@ -/** - * Unit Tests for azure-aigateway - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { fileURLToPath } from "url"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); - -const SKILL_NAME = "azure-aigateway"; -const SKILLS_PATH = path.resolve(__dirname, "../../output/skills"); -const REFERENCES_PATH = path.join(SKILLS_PATH, "azure-aigateway/references"); - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(150); - }); - - test("description meets minimum length requirement", () => { - // Descriptions should be > 150 chars for Medium adherence - expect(skill.metadata.description.length).toBeGreaterThan(150); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## When to Use This Skill"); - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## Common Tasks"); - expect(skill.content).toContain("## Troubleshooting"); - expect(skill.content).toContain("## References"); - }); - - test("contains AI Gateway governance categories", () => { - expect(skill.content).toContain("Model Governance"); - expect(skill.content).toContain("Tool Governance"); - expect(skill.content).toContain("Agent Governance"); - }); - - test("contains key policy names", () => { - expect(skill.content).toContain("azure-openai-token-limit"); - expect(skill.content).toContain("azure-openai-semantic-cache-lookup"); - expect(skill.content).toContain("llm-content-safety"); - expect(skill.content).toContain("rate-limit-by-key"); - }); - - test("contains MCP tool governance", () => { - expect(skill.content).toContain("MCP"); - expect(skill.content).toContain("convert API to MCP"); - }); - - test("references azure-prepare for APIM deployment", () => { - expect(skill.content).toContain("azure-prepare"); - }); - - test("includes Azure CLI examples", () => { - expect(skill.content).toContain("az apim show"); - expect(skill.content).toContain("az apim backend"); - }); - - test("uses progressive disclosure to reference files", () => { - expect(skill.content).toContain("references/policies.md"); - expect(skill.content).toContain("references/patterns.md"); - expect(skill.content).toContain("references/troubleshooting.md"); - }); - }); - - describe("Reference Files", () => { - test("policies.md exists and contains policy examples", () => { - const filePath = path.join(REFERENCES_PATH, "policies.md"); - expect(fs.existsSync(filePath)).toBe(true); - const content = fs.readFileSync(filePath, "utf-8"); - expect(content).toContain("azure-openai-token-limit"); - expect(content).toContain("azure-openai-semantic-cache"); - expect(content).toContain("llm-content-safety"); - expect(content).toContain(""); - }); - - test("patterns.md exists and contains configuration patterns", () => { - const filePath = path.join(REFERENCES_PATH, "patterns.md"); - expect(fs.existsSync(filePath)).toBe(true); - const content = fs.readFileSync(filePath, "utf-8"); - expect(content).toContain("Add AI Model Backend"); - expect(content).toContain("Load Balance"); - expect(content).toContain("az apim backend create"); - }); - - test("troubleshooting.md exists and contains common issues", () => { - const filePath = path.join(REFERENCES_PATH, "troubleshooting.md"); - expect(fs.existsSync(filePath)).toBe(true); - const content = fs.readFileSync(filePath, "utf-8"); - expect(content).toContain("401"); - expect(content).toContain("429"); - expect(content).toContain("Cognitive Services User"); - }); - - test("SDK reference files exist", () => { - const sdkPath = path.join(REFERENCES_PATH, "sdk"); - expect(fs.existsSync(path.join(sdkPath, "azure-ai-contentsafety-py.md"))).toBe(true); - expect(fs.existsSync(path.join(sdkPath, "azure-ai-contentsafety-ts.md"))).toBe(true); - expect(fs.existsSync(path.join(sdkPath, "azure-mgmt-apimanagement-py.md"))).toBe(true); - expect(fs.existsSync(path.join(sdkPath, "azure-mgmt-apimanagement-dotnet.md"))).toBe(true); - }); - }); - - describe("Cross-Skill References", () => { - test("azure-prepare has APIM deployment guide", () => { - const apimPath = path.join(SKILLS_PATH, "azure-prepare/references/apim.md"); - expect(fs.existsSync(apimPath)).toBe(true); - const content = fs.readFileSync(apimPath, "utf-8"); - expect(content).toContain("APIM Deployment Guide"); - expect(content).toContain("StandardV2"); - }); - - test("azure-prepare research.md maps API Management to azure-aigateway", () => { - const researchPath = path.join(SKILLS_PATH, "azure-prepare/references/research.md"); - expect(fs.existsSync(researchPath)).toBe(true); - const content = fs.readFileSync(researchPath, "utf-8"); - expect(content).toContain("API Management"); - expect(content).toContain("azure-aigateway"); - }); - - test("azure-prepare SKILL.md routes APIM prompts correctly", () => { - const preparePath = path.join(SKILLS_PATH, "azure-prepare/SKILL.md"); - const content = fs.readFileSync(preparePath, "utf-8"); - expect(content).toContain("APIM"); - expect(content).toContain("apim.md"); - }); - - test("azure-deploy SKILL.md references APIM", () => { - const deployPath = path.join(SKILLS_PATH, "azure-deploy/SKILL.md"); - const content = fs.readFileSync(deployPath, "utf-8"); - expect(content).toContain("APIM"); - }); - - test("no cross-skill directory links in reference files", () => { - // Reference files should not link outside their skill directory - const patternsContent = fs.readFileSync( - path.join(REFERENCES_PATH, "patterns.md"), "utf-8" - ); - const troubleshootingContent = fs.readFileSync( - path.join(REFERENCES_PATH, "troubleshooting.md"), "utf-8" - ); - // Should not contain ../../azure-prepare or ../../azure-deploy links - expect(patternsContent).not.toMatch(/\]\(\.\.\/\.\.\/azure-prepare\//); - expect(patternsContent).not.toMatch(/\]\(\.\.\/\.\.\/azure-deploy\//); - expect(troubleshootingContent).not.toMatch(/\]\(\.\.\/\.\.\/azure-prepare\//); - expect(troubleshootingContent).not.toMatch(/\]\(\.\.\/\.\.\/azure-deploy\//); - }); - }); -}); diff --git a/tests/azure-cloud-migrate/unit.test.ts b/tests/azure-cloud-migrate/unit.test.ts deleted file mode 100644 index c8aaf70bd..000000000 --- a/tests/azure-cloud-migrate/unit.test.ts +++ /dev/null @@ -1,143 +0,0 @@ -/** - * Unit Tests for azure-cloud-migrate - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-cloud-migrate"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - - test("description word count is within limit", () => { - const words = skill.metadata.description.split(/\s+/).length; - expect(words).toBeLessThanOrEqual(60); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Rules"); - expect(skill.content).toContain("## Steps"); - expect(skill.content).toContain("## Migration Scenarios"); - expect(skill.content).toContain("## Output Directory"); - expect(skill.content).toContain("-azure/"); - }); - - test("references azure-prepare for post-migration", () => { - expect(skill.content).toContain("azure-prepare"); - }); - }); - - describe("Migration Workflow", () => { - test("mentions assessment phase", () => { - expect(skill.content.toLowerCase()).toContain("assessment"); - }); - - test("includes code migration guidance", () => { - const content = skill.content.toLowerCase(); - expect(content).toContain("code migration"); - }); - - test("references migration scenarios", () => { - const content = skill.content.toLowerCase(); - const hasLambda = content.includes("lambda"); - const hasScenarios = content.includes("migration scenarios"); - expect(hasLambda || hasScenarios).toBe(true); - }); - - test("references workflow details", () => { - expect(skill.content).toContain("workflow-details.md"); - }); - }); - - describe("GCP Cloud Run to Container Apps Scenario", () => { - test("includes Cloud Run migration scenario", () => { - expect(skill.content).toContain("Cloud Run"); - expect(skill.content).toContain("Container Apps"); - }); - - test("references cloudrun-to-container-apps guide", () => { - expect(skill.content).toContain("cloudrun-to-container-apps.md"); - }); - - test("includes GCP Cloud Run source in scenario table", () => { - expect(skill.content).toContain("GCP Cloud Run"); - }); - }); - - describe("Spring Boot to Container Apps Scenario", () => { - test("includes Spring Boot to Container Apps scenario", () => { - expect(skill.content).toContain("Spring Boot"); - expect(skill.content).toContain("Container Apps"); - }); - - test("references spring-apps-to-aca guide", () => { - expect(skill.content).toContain("spring-apps-to-aca.md"); - }); - - test("includes Azure Spring Apps source in scenario table", () => { - expect(skill.content).toContain("Azure Spring Apps"); - }); - }); - - describe("AWS Fargate to Container Apps Scenario", () => { - test("includes Fargate migration scenario", () => { - expect(skill.content).toContain("Fargate"); - expect(skill.content).toContain("Container Apps"); - }); - - test("references fargate-to-container-apps guide", () => { - expect(skill.content).toContain("fargate-to-container-apps.md"); - }); - - test("includes AWS Fargate source in scenario table", () => { - expect(skill.content).toContain("AWS Fargate"); - }); - }); - - describe("Kubernetes to Container Apps Scenario", () => { - test("includes Kubernetes migration scenario", () => { - expect(skill.content).toContain("Kubernetes"); - expect(skill.content).toContain("Container Apps"); - }); - - test("references k8s-to-container-apps guide", () => { - expect(skill.content).toContain("k8s-to-container-apps.md"); - }); - - test("includes GKE/EKS sources in scenario table", () => { - expect(skill.content).toMatch(/GKE|EKS|Kubernetes/); - }); - }); -}); \ No newline at end of file diff --git a/tests/azure-compliance/unit.test.ts b/tests/azure-compliance/unit.test.ts deleted file mode 100644 index 8e9b3fce1..000000000 --- a/tests/azure-compliance/unit.test.ts +++ /dev/null @@ -1,168 +0,0 @@ -/** - * Unit Tests for azure-compliance - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-compliance"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1025); - }); - - test("description contains trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - const hasTriggerPhrases = - description.includes("when") || - description.includes("helps") || - description.includes("activate") || - description.includes("trigger"); - expect(hasTriggerPhrases).toBe(true); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains comprehensive compliance assessment sections", () => { - expect(skill.content).toContain("Assessment Workflow"); - expect(skill.content.toLowerCase()).toContain("azqr"); - }); - - test("contains Key Vault expiration audit sections", () => { - expect(skill.content).toContain("Key Vault"); - expect(skill.content.toLowerCase()).toContain("expir"); - }); - - test("includes skill activation triggers section", () => { - expect(skill.content).toContain("Skill Activation Triggers"); - }); - - test("describes both primary capabilities", () => { - expect(skill.content).toContain("Comprehensive Resources Assessment"); - expect(skill.content).toContain("Key Vault Expiration Monitoring"); - }); - }); - - describe("Compliance Assessment Features", () => { - test("mentions azqr tool usage", () => { - expect(skill.content).toContain("mcp_azure_mcp_extension_azqr"); - }); - - test("includes scan scope options", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/subscription|resource group|management group/); - }); - - test("describes result analysis", () => { - expect(skill.content).toContain("Analyze Scan Results"); - expect(skill.content.toLowerCase()).toMatch(/recommendations|findings/); - }); - - test("includes remediation guidance", () => { - expect(skill.content.toLowerCase()).toMatch(/remediation|fix|resolution/); - }); - }); - - describe("Key Vault Expiration Features", () => { - test("mentions all three resource types", () => { - expect(skill.content).toContain("keys"); - expect(skill.content).toContain("secrets"); - expect(skill.content).toContain("certificates"); - }); - - test("mentions expiration date checking", () => { - expect(skill.content.toLowerCase()).toMatch(/expiresOn|expiration.*date/); - }); - - test("mentions resources without expiration", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/no expiration|without expiration|missing expiration/); - }); - - test("includes priority classification", () => { - expect(skill.content).toContain("Critical"); - expect(skill.content.toLowerCase()).toMatch(/critical|high|medium|low/); - }); - - test("references Key Vault specific MCP tools", () => { - expect(skill.content).toContain("keyvault_key"); - expect(skill.content).toContain("keyvault_secret"); - expect(skill.content).toContain("keyvault_certificate"); - }); - }); - - describe("MCP Tools Documentation", () => { - test("documents compliance scanning tools", () => { - expect(skill.content).toContain("mcp_azure_mcp_extension_azqr"); - }); - - test("documents Key Vault tools", () => { - expect(skill.content).toMatch(/keyvault_.*_list/); - expect(skill.content).toMatch(/keyvault_.*_get/); - }); - }); - - describe("Prerequisites", () => { - test("mentions authentication requirements", () => { - expect(skill.content.toLowerCase()).toMatch(/authentication|logged in|az login/); - }); - - test("mentions required permissions", () => { - expect(skill.content.toLowerCase()).toMatch(/reader|permissions|access/); - }); - }); - - describe("References", () => { - test("references original skills", () => { - expect(skill.content).toContain("azure-keyvault-expiration-audit"); - expect(skill.content).toContain("azure-quick-review"); - }); - - test("links to reference documentation", () => { - expect(skill.content).toContain("references/"); - }); - }); - - describe("Azure Resource Graph Integration", () => { - test("links to Azure Resource Graph reference", () => { - expect(skill.content).toContain("references/azure-resource-graph.md"); - }); - - test("includes Resource Graph in assessments", () => { - expect(skill.content).toContain("Resource Graph"); - }); - }); - - describe("Best Practices", () => { - test("includes best practices section or guidance", () => { - expect(skill.content).toContain("Best Practices"); - }); - - test("mentions regular auditing schedules", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/weekly|monthly|regular|schedule/); - }); - }); -}); diff --git a/tests/azure-compute/unit.test.ts b/tests/azure-compute/unit.test.ts deleted file mode 100644 index 130e1be57..000000000 --- a/tests/azure-compute/unit.test.ts +++ /dev/null @@ -1,1285 +0,0 @@ -/** - * Unit Tests for azure-compute - * - * Tests the azure-compute router skill, its workflows - * (vm-recommender and vm-troubleshooter), and reference files. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; -import * as fs from "fs/promises"; -import * as path from "path"; - -const SKILL_NAME = "azure-compute"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Router Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThanOrEqual(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN triggers", () => { - expect(skill.metadata.description).toMatch(/WHEN:/i); - }); - - test("description covers recommendation, troubleshooting, and EMM", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/recommend|vm size|pricing/); - expect(desc).toMatch(/troubleshoot|can't connect|rdp|ssh/); - expect(desc).toMatch(/essential machine management|emm|machine enrollment/); - }); - - test("description mentions VM and VMSS keywords", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/vm/); - expect(desc).toMatch(/vmss|scale set/); - }); - - test("description mentions pricing", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/pric/); - }); - }); - - describe("Router Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains routing section", () => { - expect(skill.content).toContain("## Routing"); - }); - - test("contains workflows section", () => { - expect(skill.content).toContain("## Workflows"); - }); - - test("routes to vm-recommender", () => { - expect(skill.content).toContain("vm-recommender.md"); - expect(skill.content).toContain("VM Recommender"); - }); - - test("routes to vm-troubleshooter", () => { - expect(skill.content).toContain("vm-troubleshooter.md"); - expect(skill.content).toContain("VM Troubleshooter"); - }); - - test("routes to capacity-reservation", () => { - expect(skill.content).toContain("capacity-reservation.md"); - expect(skill.content).toContain("Capacity Reservation"); - }); - - test("routes to essential-machine-management", () => { - expect(skill.content).toContain("essential-machine-management.md"); - expect(skill.content).toContain("Essential Machine Management"); - }); - - test("documents when to use this skill", () => { - expect(skill.content).toContain("## When to Use This Skill"); - }); - - test("includes decision tree for routing", () => { - expect(skill.content).toMatch(/Recommend.*VM Recommender/s); - expect(skill.content).toMatch(/connect.*VM Troubleshooter/is); - }); - }); - - describe("VM Recommender Workflow", () => { - let recommenderContent: string; - - beforeAll(async () => { - const agentFile = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-recommender/vm-recommender.md" - ); - recommenderContent = await fs.readFile(agentFile, "utf-8"); - }); - - test("file exists and has content", () => { - expect(recommenderContent).toBeDefined(); - expect(recommenderContent.length).toBeGreaterThan(500); - }); - - test("contains expected sections", () => { - expect(recommenderContent).toContain("## When to Use This Skill"); - expect(recommenderContent).toContain("## Workflow"); - expect(recommenderContent).toContain("## Error Handling"); - expect(recommenderContent).toContain("## References"); - }); - - test("Step 1: documents requirements gathering", () => { - expect(recommenderContent).toContain("### Step 1: Gather Requirements"); - expect(recommenderContent).toContain("Workload type"); - expect(recommenderContent).toContain("vCPU / RAM needs"); - expect(recommenderContent).toContain("GPU needed?"); - expect(recommenderContent).toContain("Storage needs"); - expect(recommenderContent).toContain("Budget priority"); - expect(recommenderContent).toContain("OS"); - expect(recommenderContent).toContain("Region"); - }); - - test("Step 1: documents scaling-related requirements", () => { - expect(recommenderContent).toContain("Instance count"); - expect(recommenderContent).toContain("Scaling needs"); - expect(recommenderContent).toContain("Availability needs"); - expect(recommenderContent).toContain("Load balancing"); - }); - - test("Step 2: documents VM vs VMSS decision", () => { - expect(recommenderContent).toContain("### Step 2: Determine VM vs VMSS"); - expect(recommenderContent).toContain("Needs autoscaling?"); - }); - - test("Step 2: has signal-to-recommendation table", () => { - expect(recommenderContent).toContain("Autoscale on CPU, memory, or schedule"); - expect(recommenderContent).toContain("**VMSS**"); - expect(recommenderContent).toContain("**VM**"); - }); - - test("Step 3: documents VM family selection", () => { - expect(recommenderContent).toContain("### Step 3: Select VM Family"); - expect(recommenderContent).toContain("vm-families.md"); - }); - - test("Step 3: includes web_fetch verification pattern", () => { - expect(recommenderContent).toContain("web_fetch"); - expect(recommenderContent).toContain( - "learn.microsoft.com/en-us/azure/virtual-machines/sizes" - ); - }); - - test("Step 4: documents pricing lookup", () => { - expect(recommenderContent).toContain("### Step 4: Look Up Pricing"); - expect(recommenderContent).toContain("Azure Retail Prices API"); - expect(recommenderContent).toContain("retail-prices-api.md"); - }); - - test("Step 5: documents quota validation gate", () => { - expect(recommenderContent).toContain("### Step 5: Validate Quota Availability"); - expect(recommenderContent).toContain("GATE"); - expect(recommenderContent).toContain("review"); - expect(recommenderContent).toContain("vm-quotas.md"); - }); - - test("Step 6: documents recommendation presentation format", () => { - expect(recommenderContent).toContain("### Step 6: Present Recommendations"); - expect(recommenderContent).toContain("2–3 options"); - expect(recommenderContent).toContain("Hosting Model"); - expect(recommenderContent).toContain("VM Size"); - expect(recommenderContent).toContain("Estimated $/hr"); - expect(recommenderContent).toContain("Trade-off"); - }); - - test("Step 7: documents next steps", () => { - expect(recommenderContent).toContain("### Step 7: Offer Next Steps"); - expect(recommenderContent).toContain("Azure Pricing Calculator"); - }); - - test("handles API empty results", () => { - expect(recommenderContent).toContain("API returns empty results"); - }); - - test("handles unknown workload type", () => { - expect(recommenderContent).toContain("User unsure of workload type"); - expect(recommenderContent).toMatch(/D-series/i); - }); - - test("handles missing region", () => { - expect(recommenderContent).toContain("Region not specified"); - expect(recommenderContent).toContain("eastus"); - }); - - test("handles unclear VM vs VMSS choice", () => { - expect(recommenderContent).toContain( - "Unclear if VM or VMSS needed" - ); - }); - - test("handles VMSS pricing questions", () => { - expect(recommenderContent).toContain( - "User asks VMSS pricing directly" - ); - }); - - test("references use relative paths to shared references dir", () => { - expect(recommenderContent).toContain("../../references/vm-families.md"); - expect(recommenderContent).toContain("../../references/retail-prices-api.md"); - expect(recommenderContent).toContain("../../references/vmss-guide.md"); - expect(recommenderContent).toContain("../../references/vm-quotas.md"); - }); - }); - - describe("VM Troubleshooter Workflow", () => { - let troubleshooterContent: string; - - beforeAll(async () => { - const agentFile = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/vm-troubleshooter.md" - ); - troubleshooterContent = await fs.readFile(agentFile, "utf-8"); - }); - - test("file exists and has content", () => { - expect(troubleshooterContent).toBeDefined(); - expect(troubleshooterContent.length).toBeGreaterThan(500); - }); - - test("contains expected sections", () => { - expect(troubleshooterContent).toContain("## Workflow"); - expect(troubleshooterContent).toContain("## Error Handling"); - expect(troubleshooterContent).toContain("## References"); - }); - - test("documents connectivity troubleshooting triggers", () => { - expect(troubleshooterContent).toContain("## Triggers"); - expect(troubleshooterContent).toMatch(/RDP/); - expect(troubleshooterContent).toMatch(/SSH/); - expect(troubleshooterContent).toMatch(/NSG/); - }); - - test("Phase 1: documents user intent determination", () => { - expect(troubleshooterContent).toContain("### Phase 1: Determine User Intent"); - }); - - test("Phase 2: routes to solution via reference file", () => { - expect(troubleshooterContent).toContain("### Phase 2: Route to Solution"); - expect(troubleshooterContent).toContain("cannot-connect-to-vm.md"); - }); - - test("Phase 3: fetches live documentation", () => { - expect(troubleshooterContent).toContain("### Phase 3: Fetch Documentation"); - expect(troubleshooterContent).toContain("fetch_webpage"); - }); - - test("Phase 4: documents diagnose and respond workflow", () => { - expect(troubleshooterContent).toContain("### Phase 4: Diagnose and Respond"); - }); - - test("Phase 5: documents escalation path", () => { - expect(troubleshooterContent).toContain("### Phase 5: Escalation"); - }); - - test("references use local workflow reference path", () => { - expect(troubleshooterContent).toContain("references/cannot-connect-to-vm.md"); - }); - - test("documents MCP tools (fetch_webpage) with parameters", () => { - expect(troubleshooterContent).toContain("## MCP Tools"); - expect(troubleshooterContent).toContain("fetch_webpage"); - expect(troubleshooterContent).toContain("urls"); - expect(troubleshooterContent).toContain("query"); - }); - - test("Phase 1: documents all 5 routing categories", () => { - expect(troubleshooterContent).toContain("Unable to RDP"); - expect(troubleshooterContent).toContain("Unable to SSH"); - expect(troubleshooterContent).toContain("Network / Firewall"); - expect(troubleshooterContent).toContain("Credential / Auth"); - expect(troubleshooterContent).toContain("VM Agent / Tools"); - }); - - test("Phase 1: includes clarifying question for ambiguous intent", () => { - expect(troubleshooterContent).toMatch(/RDP.*Windows.*SSH.*Linux/is); - }); - - test("Phase 3: shows fetch_webpage example with URL and query params", () => { - expect(troubleshooterContent).toContain("fetch_webpage({"); - expect(troubleshooterContent).toContain(""); - }); - - test("Phase 5: documents escalation commands", () => { - expect(troubleshooterContent).toContain("az vm get-instance-view"); - expect(troubleshooterContent).toContain("az vm restart"); - expect(troubleshooterContent).toContain("az vm redeploy"); - }); - - test("Phase 5: links to comprehensive troubleshooting guides", () => { - expect(troubleshooterContent).toContain("troubleshoot-rdp-connection"); - expect(troubleshooterContent).toContain("troubleshoot-ssh-connection"); - }); - - test("error handling covers fetch_webpage failure", () => { - expect(troubleshooterContent).toMatch(/fetch_webpage.*fails/i); - }); - - test("error handling covers CLI not found", () => { - expect(troubleshooterContent).toMatch(/CLI command fails/i); - expect(troubleshooterContent).toMatch(/VM name or resource group/i); - }); - - test("error handling covers Run Command timeout", () => { - expect(troubleshooterContent).toMatch(/Run Command times out/i); - expect(troubleshooterContent).toMatch(/VM agent/i); - }); - - test("error handling covers Serial Console unavailable", () => { - expect(troubleshooterContent).toMatch(/Serial Console not available/i); - expect(troubleshooterContent).toContain("boot-diagnostics enable"); - }); - - test("error handling covers password reset failure", () => { - expect(troubleshooterContent).toMatch(/Password reset fails/i); - expect(troubleshooterContent).toMatch(/VMAccess/i); - }); - }); - - describe("Reference Files", () => { - let vmFamiliesContent: string; - let retailPricesApiContent: string; - let vmssGuideContent: string; - let vmQuotasContent: string; - let cannotConnectContent: string; - let crOverviewContent: string; - let crAssociationContent: string; - - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - const crRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/capacity-reservation/references" - ); - - const subReferenceFiles = [ - "rdp-connectivity.md", - "ssh-connectivity.md", - "network-connectivity.md", - "firewall-blocking.md", - "vm-agent-not-responding.md", - "credential-auth-errors.md", - "rdp-service-config.md", - ]; - - beforeAll(async () => { - const refsDir = path.join( - SKILLS_PATH, - "azure-compute/references" - ); - - vmFamiliesContent = await fs.readFile( - path.join(refsDir, "vm-families.md"), - "utf-8" - ); - retailPricesApiContent = await fs.readFile( - path.join(refsDir, "retail-prices-api.md"), - "utf-8" - ); - vmssGuideContent = await fs.readFile( - path.join(refsDir, "vmss-guide.md"), - "utf-8" - ); - vmQuotasContent = await fs.readFile( - path.join(refsDir, "vm-quotas.md"), - "utf-8" - ); - cannotConnectContent = await fs.readFile( - path.join(troubleshooterRefsDir, "cannot-connect-to-vm.md"), - "utf-8" - ); - crOverviewContent = await fs.readFile( - path.join(crRefsDir, "capacity-reservation-overview.md"), - "utf-8" - ); - crAssociationContent = await fs.readFile( - path.join(crRefsDir, "association-disassociation.md"), - "utf-8" - ); - }); - - test("vm-families reference file exists and has content", () => { - expect(vmFamiliesContent).toBeDefined(); - expect(vmFamiliesContent.length).toBeGreaterThan(100); - }); - - test("retail-prices-api reference file exists and has content", () => { - expect(retailPricesApiContent).toBeDefined(); - expect(retailPricesApiContent.length).toBeGreaterThan(100); - }); - - test("vmss-guide reference file exists and has content", () => { - expect(vmssGuideContent).toBeDefined(); - expect(vmssGuideContent.length).toBeGreaterThan(100); - }); - - test("vm-quotas reference file exists and has content", () => { - expect(vmQuotasContent).toBeDefined(); - expect(vmQuotasContent.length).toBeGreaterThan(100); - }); - - test("cannot-connect-to-vm reference file exists and has content", () => { - expect(cannotConnectContent).toBeDefined(); - expect(cannotConnectContent.length).toBeGreaterThan(100); - }); - - test("capacity-reservation-overview reference exists and has content", () => { - expect(crOverviewContent).toBeDefined(); - expect(crOverviewContent.length).toBeGreaterThan(100); - }); - - test("association-disassociation reference exists and has content", () => { - expect(crAssociationContent).toBeDefined(); - expect(crAssociationContent.length).toBeGreaterThan(100); - }); - - test("cannot-connect-to-vm acts as index and links to all sub-references", () => { - for (const ref of subReferenceFiles) { - expect(cannotConnectContent).toContain(ref); - } - }); - - test("cannot-connect-to-vm includes OS detection guidance", () => { - expect(cannotConnectContent).toMatch(/Determine OS/i); - expect(cannotConnectContent).toContain("Windows"); - expect(cannotConnectContent).toContain("Linux"); - }); - - test.each(subReferenceFiles)( - "troubleshooter sub-reference %s exists and has content", - async (file) => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, file), - "utf-8" - ); - expect(content).toBeDefined(); - expect(content.length).toBeGreaterThan(100); - } - ); - - test.each(subReferenceFiles)( - "troubleshooter sub-reference %s contains Symptoms → Solutions table", - async (file) => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, file), - "utf-8" - ); - expect(content).toMatch(/Symptoms?\s*→\s*Solutions?/i); - } - ); - - test.each(subReferenceFiles)( - "troubleshooter sub-reference %s contains Quick Commands", - async (file) => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, file), - "utf-8" - ); - expect(content).toContain("Quick Commands"); - expect(content).toContain("```bash"); - } - ); - }); - - describe("Troubleshooter OS Differentiation", () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - - test("rdp-connectivity.md is Windows-specific", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "rdp-connectivity.md"), - "utf-8" - ); - expect(content).toMatch(/Windows/i); - expect(content).toContain("3389"); - }); - - test("ssh-connectivity.md is Linux-specific", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "ssh-connectivity.md"), - "utf-8" - ); - expect(content).toMatch(/Linux/i); - expect(content).toContain("22"); - }); - - test("rdp-service-config.md is Windows-specific", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "rdp-service-config.md"), - "utf-8" - ); - expect(content).toContain("TermService"); - expect(content).toContain("3389"); - }); - - test("network-connectivity.md covers both Windows and Linux", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "network-connectivity.md"), - "utf-8" - ); - expect(content).toContain("Windows"); - expect(content).toContain("Linux"); - }); - - test("firewall-blocking.md covers both Windows and Linux", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "firewall-blocking.md"), - "utf-8" - ); - expect(content).toContain("Windows"); - expect(content).toContain("Linux"); - expect(content).toMatch(/iptables|firewalld|UFW/); - }); - - test("credential-auth-errors.md covers both Windows and Linux", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "credential-auth-errors.md"), - "utf-8" - ); - expect(content).toMatch(/Windows.*RDP/i); - expect(content).toMatch(/Linux.*SSH/i); - }); - - test("vm-agent-not-responding.md labels OS for each row", async () => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, "vm-agent-not-responding.md"), - "utf-8" - ); - expect(content).toContain("Windows"); - expect(content).toContain("Linux"); - expect(content).toMatch(/Serial Console.*Windows/is); - expect(content).toMatch(/Serial Console.*Linux/is); - }); - }); - - describe("Troubleshooter Routing Table Completeness", () => { - let cannotConnectContent: string; - - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - - beforeAll(async () => { - cannotConnectContent = await fs.readFile( - path.join(troubleshooterRefsDir, "cannot-connect-to-vm.md"), - "utf-8" - ); - }); - - test("routing table covers all 7 categories", () => { - const categories = [ - "Unable to RDP", - "Unable to SSH", - "Network Issues", - "Firewall Blocking", - "VM Agent Not Responding", - "Credential / Auth Errors", - "RDP Service / Config", - ]; - for (const category of categories) { - expect(cannotConnectContent).toContain(category); - } - }); - - test("routing table links to all 7 reference files", () => { - const expectedLinks = [ - "rdp-connectivity.md", - "ssh-connectivity.md", - "network-connectivity.md", - "firewall-blocking.md", - "vm-agent-not-responding.md", - "credential-auth-errors.md", - "rdp-service-config.md", - ]; - for (const link of expectedLinks) { - expect(cannotConnectContent).toContain(link); - } - }); - - test("includes escalation section with restart and redeploy", () => { - expect(cannotConnectContent).toContain("## Escalation"); - expect(cannotConnectContent).toContain("az vm restart"); - expect(cannotConnectContent).toContain("az vm redeploy"); - }); - }); - - describe("Sub-Reference: rdp-connectivity.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "rdp-connectivity.md"), - "utf-8" - ); - }); - - test("covers connection timeout symptom", () => { - expect(content).toMatch(/Connection times out/i); - }); - - test("covers credentials error symptom", () => { - expect(content).toMatch(/credentials did not work/i); - }); - - test("covers internal error symptom", () => { - expect(content).toMatch(/internal error/i); - }); - - test("covers black screen symptom", () => { - expect(content).toMatch(/Black screen/i); - }); - - test("covers licensing error symptom", () => { - expect(content).toMatch(/License Servers/i); - }); - - test("covers authentication/CredSSP error symptom", () => { - expect(content).toMatch(/authentication error|CredSSP/i); - }); - - test("covers NIC disabled symptom", () => { - expect(content).toMatch(/NIC.*disabled/i); - }); - - test("quick commands include NSG check and IP flow verify", () => { - expect(content).toContain("az network nsg rule list"); - expect(content).toContain("test-ip-flow"); - }); - - test("quick commands include RDP reset and password reset", () => { - expect(content).toContain("az vm user reset-remote-desktop"); - expect(content).toContain("az vm user update"); - }); - - test("every solution row links to Microsoft Learn documentation", () => { - expect(content).toContain("learn.microsoft.com"); - // Count documentation links — should have at least one per symptom row - const docLinks = content.match(/https:\/\/learn\.microsoft\.com[^\s)]+/g) || []; - expect(docLinks.length).toBeGreaterThanOrEqual(10); - }); - }); - - describe("Sub-Reference: ssh-connectivity.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "ssh-connectivity.md"), - "utf-8" - ); - }); - - test("covers connection refused symptom", () => { - expect(content).toMatch(/Connection refused/i); - }); - - test("covers connection timed out symptom", () => { - expect(content).toMatch(/Connection timed out/i); - }); - - test("covers permission denied publickey symptom", () => { - expect(content).toMatch(/Permission denied \(publickey\)/i); - }); - - test("covers permission denied password symptom", () => { - expect(content).toMatch(/Permission denied \(password\)/i); - }); - - test("covers host key verification failure", () => { - expect(content).toMatch(/Host key verification failed/i); - }); - - test("covers SSH hangs symptom", () => { - expect(content).toMatch(/SSH hangs/i); - }); - - test("covers SELinux blocking SSH", () => { - expect(content).toMatch(/SELinux/i); - }); - - test("covers Entra ID (AAD) SSH login", () => { - expect(content).toMatch(/Entra ID|AAD/i); - }); - - test("quick commands include SSH reset and key reset", () => { - expect(content).toContain("az vm user reset-ssh"); - expect(content).toContain("ssh-key-value"); - }); - - test("quick commands include sshd status check", () => { - expect(content).toContain("systemctl status sshd"); - }); - - test("every solution row links to Microsoft Learn documentation", () => { - const docLinks = content.match(/https:\/\/learn\.microsoft\.com[^\s)]+/g) || []; - expect(docLinks.length).toBeGreaterThanOrEqual(8); - }); - }); - - describe("Sub-Reference: network-connectivity.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "network-connectivity.md"), - "utf-8" - ); - }); - - test("covers NSG missing allow rule", () => { - expect(content).toMatch(/NSG.*no allow rule/i); - }); - - test("covers dual NSG (NIC and subnet) blocking", () => { - expect(content).toMatch(/NIC and subnet/i); - }); - - test("covers custom route (UDR) issues", () => { - expect(content).toMatch(/UDR|Custom route/i); - }); - - test("covers no public IP", () => { - expect(content).toMatch(/no public IP/i); - }); - - test("covers NIC disabled (Windows and Linux)", () => { - expect(content).toMatch(/NIC.*disabled/i); - expect(content).toMatch(/NIC.*down/i); - }); - - test("covers static IP misconfiguration", () => { - expect(content).toMatch(/Static IP/i); - }); - - test("covers DNS resolution failure", () => { - expect(content).toMatch(/DNS/i); - }); - - test("quick commands include effective NSG rules and routes", () => { - expect(content).toContain("list-effective-nsg"); - expect(content).toContain("show-effective-route-table"); - }); - - test("quick commands include public IP check", () => { - expect(content).toContain("az vm list-ip-addresses"); - }); - - test("has OS-specific quick commands for Windows and Linux", () => { - expect(content).toContain("Quick Commands — Windows"); - expect(content).toContain("Quick Commands — Linux"); - }); - }); - - describe("Sub-Reference: firewall-blocking.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "firewall-blocking.md"), - "utf-8" - ); - }); - - test("covers Windows Firewall blocking RDP", () => { - expect(content).toMatch(/Windows Firewall blocking RDP/i); - }); - - test("covers BlockInboundAlways policy", () => { - expect(content).toContain("BlockInboundAlways"); - }); - - test("covers third-party AV/firewall", () => { - expect(content).toMatch(/Third-party/i); - }); - - test("covers iptables/nftables blocking SSH", () => { - expect(content).toMatch(/iptables|nftables/); - }); - - test("covers firewalld blocking SSH", () => { - expect(content).toContain("firewalld"); - }); - - test("covers UFW blocking SSH", () => { - expect(content).toContain("UFW"); - }); - - test("covers offline repair for unreachable Windows VM", () => { - expect(content).toMatch(/offline.*repair/i); - }); - - test("covers Serial Console fallback for unreachable Linux VM", () => { - expect(content).toMatch(/Serial Console|repair VM/i); - }); - - test("has OS-specific quick commands for Windows and Linux", () => { - expect(content).toContain("Quick Commands — Windows"); - expect(content).toContain("Quick Commands — Linux"); - }); - }); - - describe("Sub-Reference: credential-auth-errors.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "credential-auth-errors.md"), - "utf-8" - ); - }); - - test("covers Windows 'credentials did not work' error", () => { - expect(content).toMatch(/credentials did not work/i); - }); - - test("covers account expired error", () => { - expect(content).toMatch(/account.*expired/i); - }); - - test("covers domain trust relationship failure", () => { - expect(content).toMatch(/Trust relationship/i); - }); - - test("covers CredSSP encryption oracle error", () => { - expect(content).toContain("CredSSP"); - expect(content).toContain("AllowEncryptionOracle"); - }); - - test("covers Linux permission denied publickey", () => { - expect(content).toMatch(/Permission denied \(publickey\)/i); - }); - - test("covers Linux permission denied password", () => { - expect(content).toMatch(/Permission denied \(password\)/i); - }); - - test("covers account locked after failed attempts", () => { - expect(content).toMatch(/locked/i); - }); - - test("covers Entra ID (AAD) missing role assignment", () => { - expect(content).toMatch(/Entra ID|AAD/i); - expect(content).toMatch(/Virtual Machine.*Login/i); - }); - - test("has separate Quick Commands for Windows and Linux", () => { - expect(content).toContain("Quick Commands — Windows"); - expect(content).toContain("Quick Commands — Linux"); - }); - - test("quick commands include password reset for both OSes", () => { - expect(content).toContain("az vm user update"); - expect(content).toContain("az vm user reset-remote-desktop"); - expect(content).toContain("ssh-key-value"); - }); - }); - - describe("Sub-Reference: vm-agent-not-responding.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "vm-agent-not-responding.md"), - "utf-8" - ); - }); - - test("covers Run Command timeout for both OSes", () => { - expect(content).toMatch(/Run Command times out/i); - }); - - test("covers password reset failure when agent is down", () => { - expect(content).toMatch(/Password.*reset fails/i); - }); - - test("covers BSOD / boot failure (Windows)", () => { - expect(content).toMatch(/BSOD/i); - }); - - test("covers kernel panic / boot failure (Linux)", () => { - expect(content).toMatch(/kernel panic/i); - }); - - test("covers VMAccess limitation on domain controllers", () => { - expect(content).toMatch(/domain controller/i); - }); - - test("quick commands include Serial Console connect", () => { - expect(content).toContain("az serial-console connect"); - }); - - test("quick commands include boot diagnostics enable", () => { - expect(content).toContain("az vm boot-diagnostics enable"); - }); - - test("quick commands include repair VM create and restore", () => { - expect(content).toContain("az vm repair create"); - expect(content).toContain("az vm repair restore"); - }); - }); - - describe("Sub-Reference: rdp-service-config.md — Symptom Coverage", () => { - let content: string; - - beforeAll(async () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - content = await fs.readFile( - path.join(troubleshooterRefsDir, "rdp-service-config.md"), - "utf-8" - ); - }); - - test("covers TermService not running", () => { - expect(content).toContain("TermService not running"); - }); - - test("covers RDP port changed from 3389", () => { - expect(content).toMatch(/port changed/i); - expect(content).toContain("3389"); - }); - - test("covers RDP disabled (fDenyTSConnections)", () => { - expect(content).toContain("fDenyTSConnections"); - }); - - test("covers TLS/SSL certificate issues", () => { - expect(content).toMatch(/TLS|SSL/); - expect(content).toMatch(/certificate/i); - }); - - test("covers NLA/Security Layer mismatch", () => { - expect(content).toMatch(/NLA/i); - }); - - test("covers GPO overriding RDP settings", () => { - expect(content).toContain("GPO"); - expect(content).toContain("Terminal Services"); - }); - - test("covers RDS licensing expired", () => { - expect(content).toMatch(/licensing/i); - }); - - test("quick commands include RDP reset and TermService check", () => { - expect(content).toContain("az vm user reset-remote-desktop"); - expect(content).toContain("Get-Service TermService"); - }); - - test("quick commands include restart and redeploy as last resort", () => { - expect(content).toContain("az vm restart"); - expect(content).toContain("az vm redeploy"); - }); - }); - - describe("Sub-Reference Documentation URL Integrity", () => { - const troubleshooterRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/vm-troubleshooter/references" - ); - - const subReferenceFiles = [ - "rdp-connectivity.md", - "ssh-connectivity.md", - "network-connectivity.md", - "firewall-blocking.md", - "vm-agent-not-responding.md", - "credential-auth-errors.md", - "rdp-service-config.md", - ]; - - test.each(subReferenceFiles)( - "%s contains only learn.microsoft.com documentation links", - async (file) => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, file), - "utf-8" - ); - const docLinks = content.match(/https:\/\/[^\s)]+/g) || []; - expect(docLinks.length).toBeGreaterThan(0); - for (const link of docLinks) { - expect(link).toMatch(/^https:\/\/learn\.microsoft\.com\//); - } - } - ); - - test.each(subReferenceFiles)( - "%s has at least one documentation link per solution row", - async (file) => { - const content = await fs.readFile( - path.join(troubleshooterRefsDir, file), - "utf-8" - ); - // Count table rows (lines starting with |) excluding header/separator - const tableRows = content.split("\n").filter( - (line) => line.startsWith("|") && !line.includes("---") && !line.includes("Symptom") - ); - const docLinks = content.match(/https:\/\/learn\.microsoft\.com[^\s)]+/g) || []; - // At least half the rows should have doc links (some rows share the same URL) - expect(docLinks.length).toBeGreaterThanOrEqual(Math.floor(tableRows.length / 2)); - } - ); - }); - - describe("Capacity Reservation Workflow", () => { - let crContent: string; - - beforeAll(async () => { - const agentFile = path.join( - SKILLS_PATH, - "azure-compute/workflows/capacity-reservation/capacity-reservation.md" - ); - crContent = await fs.readFile(agentFile, "utf-8"); - }); - - test("file exists and has content", () => { - expect(crContent).toBeDefined(); - expect(crContent.length).toBeGreaterThan(500); - }); - - test("contains expected sections", () => { - expect(crContent).toContain("## When to Use This Workflow"); - expect(crContent).toContain("## Workflow"); - expect(crContent).toContain("## Error Handling"); - expect(crContent).toContain("## Key Concepts"); - }); - - test("documents proactive suggestion patterns", () => { - expect(crContent).toContain("proactively suggest"); - expect(crContent).toContain("Deployment failure is unacceptable"); - expect(crContent).toContain("Known scale-out events"); - expect(crContent).toContain("In-demand SKUs"); - }); - - test("Step 1: documents requirements gathering", () => { - expect(crContent).toContain("### Step 1: Gather Requirements"); - expect(crContent).toContain("Region"); - expect(crContent).toContain("VM size(s)"); - expect(crContent).toContain("Quantity"); - expect(crContent).toContain("Availability Zone(s)"); - expect(crContent).toContain("Resource group"); - }); - - test("Step 1: quantity must always be asked", () => { - expect(crContent).toContain("Always ask — do not infer"); - }); - - test("Step 1: zones should not be inferred", () => { - expect(crContent).toContain("Do not pick a zone on the user's behalf"); - }); - - test("Step 2: documents CRG and reservation creation", () => { - expect(crContent).toContain("### Step 2: Create Capacity Reservation Group and Reservation"); - expect(crContent).toContain("az capacity reservation group create"); - expect(crContent).toContain("az capacity reservation create"); - }); - - test("Step 3: documents verification", () => { - expect(crContent).toContain("### Step 3: Verify Reservation"); - expect(crContent).toContain("az capacity reservation show"); - }); - - test("Step 4: documents next steps", () => { - expect(crContent).toContain("### Step 4: Offer Next Steps"); - }); - - test("documents managing existing reservations", () => { - expect(crContent).toContain("## Managing Existing Reservations"); - expect(crContent).toContain("Associate a VM or VMSS"); - expect(crContent).toContain("Disassociate a VM or VMSS"); - expect(crContent).toContain("Find a matching CRG"); - }); - - test("error handling covers common failures", () => { - expect(crContent).toContain("SKU not available in region/zone"); - expect(crContent).toContain("Quota exceeded"); - expect(crContent).toContain("Insufficient platform capacity"); - expect(crContent).toContain("Duplicate SKU + zone in CRG"); - }); - - test("references local workflow reference files", () => { - expect(crContent).toContain("references/capacity-reservation-overview.md"); - expect(crContent).toContain("references/association-disassociation.md"); - }); - - test("key concepts table documents billing model", () => { - expect(crContent).toContain("Charges begin as soon as the reservation is created"); - }); - }); - - describe("Essential Machine Management Workflow", () => { - let emmContent: string; - - beforeAll(async () => { - const agentFile = path.join( - SKILLS_PATH, - "azure-compute/workflows/essential-machine-management/essential-machine-management.md" - ); - emmContent = await fs.readFile(agentFile, "utf-8"); - }); - - test("file exists and has content", () => { - expect(emmContent).toBeDefined(); - expect(emmContent.length).toBeGreaterThan(500); - }); - - test("contains expected sections", () => { - expect(emmContent).toContain("## Routing"); - expect(emmContent).toContain("## Browse Enrolled Subscriptions"); - expect(emmContent).toContain("## Offboard a Subscription"); - expect(emmContent).toContain("## Troubleshooting"); - expect(emmContent).toContain("## Error Handling"); - }); - - test("routing covers enable, overview, prerequisites, browse, offboard, and troubleshoot", () => { - expect(emmContent).toContain("Enable / onboard / enroll"); - expect(emmContent).toContain("What is EMM"); - expect(emmContent).toContain("Prerequisites"); - expect(emmContent).toContain("View enrolled subscriptions"); - expect(emmContent).toContain("Offboard"); - expect(emmContent).toContain("Troubleshoot"); - }); - - test("routing links to all reference files", () => { - expect(emmContent).toContain("references/emm-enable-flow.md"); - expect(emmContent).toContain("references/emm-enable-flow-portal-guidance.md"); - expect(emmContent).toContain("references/emm-overview.md"); - expect(emmContent).toContain("references/emm-prerequisites.md"); - }); - - test("browse section documents the EMM status API endpoint", () => { - expect(emmContent).toContain("Microsoft.ManagedOps/managedOps/default"); - expect(emmContent).toContain("api-version=2025-07-28-preview"); - }); - - test("browse section documents response interpretation", () => { - expect(emmContent).toContain("provisioningState: Succeeded"); - expect(emmContent).toContain("Subscription is enrolled"); - expect(emmContent).toContain("404"); - expect(emmContent).toContain("Subscription is not enrolled"); - }); - - test("error handling covers common failures", () => { - expect(emmContent).toContain("Permission denied during enable"); - expect(emmContent).toContain("UAMI role check fails"); - expect(emmContent).toContain("RP not registered"); - expect(emmContent).toContain("Cross-subscription workspace error"); - }); - - test("defaults to Copilot-guided flow and requires explicit portal mention", () => { - expect(emmContent).toContain("Only route to the portal guide when the user explicitly mentions"); - }); - - test("notes EMM is in public preview", () => { - expect(emmContent).toContain("public preview"); - }); - }); - - describe("EMM Reference Files", () => { - const emmRefsDir = path.join( - SKILLS_PATH, - "azure-compute/workflows/essential-machine-management/references" - ); - - const emmReferenceFiles = [ - "emm-enable-flow.md", - "emm-enable-flow-portal-guidance.md", - "emm-overview.md", - "emm-prerequisites.md", - ]; - - test.each(emmReferenceFiles)( - "EMM reference %s exists and has content", - async (file) => { - const content = await fs.readFile( - path.join(emmRefsDir, file), - "utf-8" - ); - expect(content).toBeDefined(); - expect(content.length).toBeGreaterThan(100); - } - ); - - test("emm-enable-flow.md documents Copilot-guided steps", async () => { - const content = await fs.readFile( - path.join(emmRefsDir, "emm-enable-flow.md"), - "utf-8" - ); - expect(content).toContain("Step 1"); - expect(content).toContain("Step 2"); - expect(content).toMatch(/role/i); - expect(content).toMatch(/UAMI|managed identity/i); - }); - - test("emm-enable-flow.md documents role validation with assignedTo pattern", async () => { - const content = await fs.readFile( - path.join(emmRefsDir, "emm-enable-flow.md"), - "utf-8" - ); - expect(content).toContain("assignedTo"); - expect(content).toContain("roleDefinitions"); - }); - - test("emm-prerequisites.md documents required roles", async () => { - const content = await fs.readFile( - path.join(emmRefsDir, "emm-prerequisites.md"), - "utf-8" - ); - expect(content).toMatch(/Essential Machine Management Administrator/i); - expect(content).toMatch(/Managed Identity Operator/i); - expect(content).toMatch(/Resource Policy Contributor/i); - }); - - test("emm-prerequisites.md documents required resource providers", async () => { - const content = await fs.readFile( - path.join(emmRefsDir, "emm-prerequisites.md"), - "utf-8" - ); - expect(content).toContain("Microsoft.ManagedOps"); - }); - - test("emm-overview.md documents features and tiers", async () => { - const content = await fs.readFile( - path.join(emmRefsDir, "emm-overview.md"), - "utf-8" - ); - expect(content).toMatch(/Essential/i); - expect(content).toMatch(/monitor|update|security/i); - }); - }); - -}); diff --git a/tests/azure-cost/unit.test.ts b/tests/azure-cost/unit.test.ts deleted file mode 100644 index f9a1c731c..000000000 --- a/tests/azure-cost/unit.test.ts +++ /dev/null @@ -1,345 +0,0 @@ -/** - * Unit Tests for azure-cost - * - * Test isolated skill logic and validation rules. - * Covers shared structure, metadata, and content for all three - * sub-areas: Cost Query, Cost Forecast, and Cost Optimization. - * - * Note: Workflow content lives in separate files under cost-query/, - * cost-forecast/, and cost-optimization/ folders. The SKILL.md acts - * as a router that links to these workflows. Tests load each source - * independently to verify content in the correct location. - */ - -import { readFileSync } from "node:fs"; -import { resolve } from "node:path"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-cost"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - let queryWorkflow: string; - let forecastWorkflow: string; - let optimizationWorkflow: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const skillDir = skill.path; - queryWorkflow = readFileSync(resolve(skillDir, "cost-query", "workflow.md"), "utf-8"); - forecastWorkflow = readFileSync(resolve(skillDir, "cost-forecast", "workflow.md"), "utf-8"); - optimizationWorkflow = readFileSync(resolve(skillDir, "cost-optimization", "workflow.md"), "utf-8"); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1000); - }); - - test("description contains trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - const hasTriggerPhrases = - description.includes("use this") || - description.includes("use when") || - description.includes("helps") || - description.includes("activate") || - description.includes("trigger") || - description.includes("when:"); - expect(hasTriggerPhrases).toBe(true); - }); - }); - - describe("Skill Content Structure", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(1000); - }); - - test("contains Quick Reference section", () => { - expect(skill.content).toMatch(/## Quick Reference/i); - }); - - test("contains When to Use This Skill section", () => { - expect(skill.content).toMatch(/## When to Use This Skill/i); - }); - - test("contains MCP Tools section", () => { - expect(skill.content).toMatch(/## MCP Tools/i); - }); - - test("contains Error Handling in workflow files", () => { - const hasErrorHandling = - queryWorkflow.match(/## Error Handling/i) || - forecastWorkflow.match(/## Error Handling/i); - expect(hasErrorHandling).toBeTruthy(); - }); - - test("contains Guardrails in workflow files", () => { - const hasGuardrails = - queryWorkflow.match(/Guardrails/i) || - forecastWorkflow.match(/Guardrails/i); - expect(hasGuardrails).toBeTruthy(); - }); - - test("contains Best Practices section", () => { - expect(skill.content).toMatch(/## Best Practices/i); - }); - - test("contains Safety Requirements section", () => { - expect(skill.content).toMatch(/## Safety Requirements/i); - }); - - test("contains Common Pitfalls section", () => { - expect(skill.content).toMatch(/## Common Pitfalls/i); - expect(skill.content).toContain("free tier"); - }); - }); - - describe("MCP Tool References", () => { - test("references azure__documentation tool", () => { - expect(skill.content).toContain("azure__documentation"); - }); - - test("references azure__extension_cli_generate tool", () => { - expect(skill.content).toContain("azure__extension_cli_generate"); - }); - - test("references azure__get_azure_bestpractices tool", () => { - expect(skill.content).toContain("azure__get_azure_bestpractices"); - }); - }); - - describe("Scope Reference", () => { - test("references scopes", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/subscription/); - expect(content).toMatch(/resource group/); - expect(content).toMatch(/billing account/); - }); - }); - - describe("Data Classification", () => { - test("includes data classification guidance", () => { - expect(skill.content).toContain("ACTUAL DATA"); - expect(skill.content).toContain("ESTIMATED"); - expect(skill.content).toContain("VALIDATED"); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - const supported = ["name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable"]; - const keys = frontmatter.split("\n") - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - - test("WHEN clause is inside description", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - }); - - // --- Cost Query (Part 1) --- - describe("Cost Query Workflow", () => { - test("description contains cost-query-related keywords", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toMatch(/cost|query|spend|breakdown|actual|amortized/); - }); - - test("contains Cost Query Workflow section", () => { - expect(skill.content).toMatch(/## Part 1: Cost Query Workflow/i); - }); - - test("references Cost Management Query API endpoint", () => { - expect(skill.content).toMatch(/Microsoft\.CostManagement\/query/); - }); - - test("mentions key guardrails", () => { - const content = queryWorkflow.toLowerCase(); - expect(content).toMatch(/granularity/); - expect(content).toMatch(/date range/); - expect(content).toMatch(/groupby/i); - }); - }); - - // --- Cost Forecast (Part 3) --- - describe("Cost Forecast Workflow", () => { - test("description contains forecast-related keywords", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toMatch(/forecast|predict|project|estimate|future/); - }); - - test("contains Cost Forecast Workflow section", () => { - expect(skill.content).toMatch(/## Part 3: Cost Forecast Workflow/i); - }); - - test("references forecast API endpoint", () => { - expect(skill.content).toMatch(/Microsoft\.CostManagement\/forecast|forecast\s+API/i); - }); - - test("mentions to-date must be in the future", () => { - const content = forecastWorkflow.toLowerCase(); - expect(content).toMatch(/future|must be in the future/); - }); - - test("mentions grouping not supported", () => { - const content = forecastWorkflow.toLowerCase(); - expect(content).toContain("grouping"); - expect(content).toMatch(/not supported/); - }); - - test("mentions includeActualCost field", () => { - expect(forecastWorkflow).toContain("includeActualCost"); - }); - - test("mentions minimum training data requirement", () => { - const content = forecastWorkflow.toLowerCase(); - expect(content).toMatch(/training data|28 days/); - }); - - test("references CostStatus or response types", () => { - expect(forecastWorkflow).toMatch(/CostStatus|("Actual".*"Forecast"|Actual.*Forecast)/); - }); - }); - - // --- Cost Optimization (Part 2) --- - describe("Cost Optimization Workflow", () => { - test("description contains optimization-related keywords", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toMatch(/orphaned|rightsize|unused/); - }); - - test("contains Cost Optimization Workflow section", () => { - expect(skill.content).toMatch(/## Part 2: Cost Optimization Workflow/i); - }); - - test("documents step-by-step instructions", () => { - expect(optimizationWorkflow).toMatch(/## Step \d+:/); - }); - - test("includes prerequisites validation", () => { - expect(optimizationWorkflow).toMatch(/## Step 0: Validate Prerequisites/i); - expect(optimizationWorkflow).toContain("Azure CLI"); - expect(optimizationWorkflow).toContain("azqr"); - }); - - test("includes cost query instructions", () => { - expect(optimizationWorkflow).toMatch(/## Step \d+: Query Actual Costs/i); - expect(optimizationWorkflow).toContain("Cost Management API"); - expect(optimizationWorkflow).toContain("ActualCost"); - }); - - test("includes report generation step", () => { - expect(optimizationWorkflow).toMatch(/## Step \d+: Generate Optimization Report/i); - expect(optimizationWorkflow).toContain("output/"); - expect(optimizationWorkflow).toContain("costoptimizereport"); - }); - }); - - describe("Azure Quick Review", () => { - test("mentions Azure Quick Review (azqr)", () => { - expect(skill.content).toContain("azqr"); - expect(skill.content).toContain("Azure Quick Review"); - expect(skill.content).toMatch(/orphaned resources/i); - }); - - test("documents azqr installation", () => { - expect(optimizationWorkflow).toContain("azqr"); - expect(optimizationWorkflow).toMatch(/azqr version/i); - }); - }); - - describe("Required Tools and Extensions", () => { - test("documents Azure CLI requirement", () => { - expect(optimizationWorkflow).toContain("az login"); - expect(optimizationWorkflow).toMatch(/Azure CLI/i); - }); - - test("lists required Azure CLI extensions", () => { - expect(skill.content).toContain("costmanagement"); - expect(optimizationWorkflow).toContain("resource-graph"); - }); - - test("documents required permissions", () => { - expect(skill.content).toMatch(/Cost Management Reader/i); - expect(skill.content).toMatch(/Monitoring Reader/i); - expect(skill.content).toMatch(/Reader/i); - }); - }); - - describe("Cost Analysis Features", () => { - test("mentions orphaned resources", () => { - expect(skill.content).toMatch(/orphaned resources/i); - expect(optimizationWorkflow).toMatch(/unattached disks|unused NICs/i); - }); - - test("mentions rightsizing", () => { - expect(skill.content).toMatch(/rightsize|rightsizing/i); - expect(skill.content).toMatch(/VM|virtual machines/i); - }); - - test("mentions utilization metrics", () => { - expect(skill.content).toContain("Azure Monitor"); - expect(optimizationWorkflow).toMatch(/utilization|metrics/i); - expect(optimizationWorkflow).toContain("Percentage CPU"); - }); - - test("includes pricing validation", () => { - expect(optimizationWorkflow).toMatch(/## Step \d+: Validate Pricing/i); - expect(optimizationWorkflow).toContain("azure.microsoft.com/pricing"); - }); - }); - - describe("Output and Reporting", () => { - test("defines output folder convention", () => { - expect(optimizationWorkflow).toContain("output/"); - expect(optimizationWorkflow).toMatch(/costoptimizereport.*\.md/); - }); - - test("references report template", () => { - expect(optimizationWorkflow).toContain("report-template.md"); - }); - - test("documents audit trail", () => { - expect(skill.content).toContain("audit trail"); - expect(optimizationWorkflow).toContain("cost-query-result"); - expect(optimizationWorkflow).toMatch(/\.json/); - }); - - test("references Redis-specific optimization", () => { - expect(skill.content).toContain("Redis"); - expect(optimizationWorkflow).toContain("azure-cache-for-redis.md"); - }); - }); - - describe("Azure Resource Graph Integration", () => { - test("links to Azure Resource Graph reference", () => { - expect(optimizationWorkflow).toContain("azure-resource-graph.md"); - }); - - test("mentions Resource Graph for resource discovery", () => { - expect(optimizationWorkflow).toContain("Azure Resource Graph"); - }); - }); -}); diff --git a/tests/azure-deploy/unit.test.ts b/tests/azure-deploy/unit.test.ts deleted file mode 100644 index 614e04e38..000000000 --- a/tests/azure-deploy/unit.test.ts +++ /dev/null @@ -1,108 +0,0 @@ -/** - * Unit Tests for azure-deploy - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-deploy"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Triggers"); - expect(skill.content).toContain("## Rules"); - expect(skill.content).toContain("## Steps"); - expect(skill.content).toContain("## MCP Tools"); - }); - - test("requires azure-validate prerequisite", () => { - expect(skill.content).toContain("azure-validate"); - expect(skill.content.toLowerCase()).toContain("prerequisite"); - }); - }); - - describe("Deployment Workflow", () => { - test("mentions plan file requirement", () => { - expect(skill.content).toContain(".azure/deployment-plan.md"); - }); - - test("references deployment recipes", () => { - expect(skill.content).toContain("recipes/README.md"); - }); - - test("includes verification step", () => { - expect(skill.content.toLowerCase()).toContain("verify"); - }); - - test("includes post-deployment step", () => { - expect(skill.content.toLowerCase()).toContain("post-deploy"); - }); - }); - - describe("Live Role Verification Step", () => { - test("includes live role verification step in workflow", () => { - expect(skill.content).toContain("Live Role Verification"); - }); - - test("references live-role-verification.md", () => { - expect(skill.content).toContain("live-role-verification.md"); - }); - - test("documents azure__role MCP tool", () => { - expect(skill.content).toContain("azure__role"); - }); - - test("live role verification comes before report results", () => { - const liveRoleIndex = skill.content.indexOf("Live Role Verification"); - const reportIndex = skill.content.indexOf("Report Results"); - expect(liveRoleIndex).toBeGreaterThan(-1); - expect(reportIndex).toBeGreaterThan(-1); - expect(liveRoleIndex).toBeLessThan(reportIndex); - }); - }); - - describe("Post-Deployment Support", () => { - test("references post-deployment documentation", () => { - expect(skill.content).toContain("Post-Deployment"); - expect(skill.content).toContain("post-deployment.md"); - }); - - test("mentions SQL and EF migrations support", () => { - const lowerContent = skill.content.toLowerCase(); - expect(lowerContent).toMatch(/\bsql\b|entity framework|\bef\b|\bmigrations\b/); - }); - }); -}); diff --git a/tests/azure-diagnostics/unit.test.ts b/tests/azure-diagnostics/unit.test.ts deleted file mode 100644 index caee4d5a3..000000000 --- a/tests/azure-diagnostics/unit.test.ts +++ /dev/null @@ -1,165 +0,0 @@ -/** - * Unit Tests for azure-diagnostics - * - * Test isolated skill logic and validation rules. - */ - -import { readFileSync, existsSync } from "node:fs"; -import * as path from "node:path"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-diagnostics"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is comprehensive and actionable", () => { - // Descriptions should be 150-1024 chars for optimal triggering - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - - test("has diagnostic-specific trigger keywords", () => { - const description = skill.metadata.description.toLowerCase(); - const hasDiagnosticKeywords = - description.includes("debug") || - description.includes("troubleshoot") || - description.includes("diagnose") || - description.includes("fix") || - description.includes("investigate"); - expect(hasDiagnosticKeywords).toBe(true); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(500); - }); - - test("contains diagnostic workflow sections", () => { - expect(skill.content).toContain("Quick Diagnosis Flow"); - expect(skill.content).toContain("Troubleshooting"); - }); - - test("includes AppLens MCP tool references", () => { - expect(skill.content).toContain("mcp_azure_mcp_applens"); - }); - - test("includes Azure Monitor tool references", () => { - expect(skill.content).toContain("mcp_azure_mcp_monitor"); - }); - - test("includes Resource Health tool references", () => { - expect(skill.content).toContain("mcp_azure_mcp_resourcehealth"); - }); - - test("provides diagnostic commands for Container Apps", () => { - expect(skill.content).toContain("containerapp logs"); - }); - - test("provides diagnostic commands for Function Apps", () => { - expect(skill.content).toContain("app-insights query"); - }); - - test("links to Function Apps troubleshooting reference", () => { - expect(skill.content).toContain("references/functions/README.md"); - }); - - test("references KQL query documentation", () => { - expect(skill.content).toContain("kql-queries.md"); - }); - - test("links to Azure Kubernetes troubleshooting reference", () => { - expect(skill.content).toContain("troubleshooting/aks/aks-troubleshooting.md"); - }); - - test("routes AKS incidents to the troubleshooting guide", () => { - expect(skill.content).toContain("Route active AKS incidents"); - }); - - test("links to messaging troubleshooting reference", () => { - expect(skill.content).toContain("troubleshooting/messaging/README.md"); - }); - - test("routes messaging incidents to the troubleshooting guide", () => { - expect(skill.content).toContain("Route Azure Messaging SDK troubleshooting"); - }); - - test("messaging troubleshooting files exist", () => { - const messagingDir = path.join(skill.path, "troubleshooting", "messaging"); - const messagingReadmePath = path.join(messagingDir, "README.md"); - const messagingReadme = readFileSync(messagingReadmePath, "utf-8"); - - expect(messagingReadme).toContain("Azure Messaging Troubleshooting"); - expect(messagingReadme).toContain("service-troubleshooting.md"); - expect(messagingReadme).toContain("azure-eventhubs-py.md"); - expect(messagingReadme).toContain("azure-servicebus-py.md"); - - // Verify referenced files actually exist on disk - const expectedFiles = [ - "service-troubleshooting.md", - "azure-eventhubs-dotnet.md", - "azure-eventhubs-java.md", - "azure-eventhubs-js.md", - "azure-eventhubs-py.md", - "azure-servicebus-dotnet.md", - "azure-servicebus-java.md", - "azure-servicebus-js.md", - "azure-servicebus-py.md", - ]; - for (const file of expectedFiles) { - expect(existsSync(path.join(messagingDir, file))).toBe(true); - } - }); - - test("supports sidecar references for AKS command flows and MCP guidance", () => { - const troubleshootingSkillPath = path.join(skill.path, "troubleshooting", "aks", "aks-troubleshooting.md"); - const troubleshootingContent = readFileSync(troubleshootingSkillPath, "utf-8"); - - expect(troubleshootingContent).toContain("references/command-flows.md"); - expect(troubleshootingContent).toContain("references/structured-input-modes.md"); - expect(troubleshootingContent).toContain("references/aks-mcp.md"); - }); - - test("documents AKS-MCP preference and lightweight discovery guidance", () => { - const aksMcpReferencePath = path.join(skill.path, "troubleshooting", "aks", "references", "aks-mcp.md"); - const aksMcpReference = readFileSync(aksMcpReferencePath, "utf-8"); - - expect(aksMcpReference).toContain("mcp_azure_mcp_aks"); - expect(aksMcpReference).toContain("enumerate the exact AKS-MCP tools"); - expect(aksMcpReference).toContain("readonly"); - expect(aksMcpReference).toContain("AZURE_CLIENT_ID"); - expect(aksMcpReference).toContain("AZURE_SUBSCRIPTION_ID"); - }); - }); - - describe("Skill Structure", () => { - test("follows authoritative guidance pattern", () => { - expect(skill.content).toContain("AUTHORITATIVE GUIDANCE"); - }); - - test("has systematic diagnosis approach", () => { - expect(skill.content).toContain("Identify symptoms"); - expect(skill.content).toContain("Check resource health"); - expect(skill.content).toContain("Review logs"); - }); - }); -}); diff --git a/tests/azure-enterprise-infra-planner/unit.test.ts b/tests/azure-enterprise-infra-planner/unit.test.ts deleted file mode 100644 index 4cf34af1e..000000000 --- a/tests/azure-enterprise-infra-planner/unit.test.ts +++ /dev/null @@ -1,210 +0,0 @@ -/** - * Unit Tests for azure-enterprise-infra-planner - * - * Test isolated skill logic, validation rules, and reference data integrity. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-enterprise-infra-planner"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - - test("description contains PREFER routing guidance", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("prefer"); - }); - }); - - describe("Skill Content", () => { - let workflowContent: string; - - beforeAll(() => { - const workflowPath = path.join(skill.path, "references", "workflow.md"); - workflowContent = fs.readFileSync(workflowPath, "utf-8"); - }); - - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("documents research phase", () => { - expect(skill.content).toContain("Research"); - expect(skill.content).toContain("microsoft_docs_search"); - expect(skill.content).toContain("microsoft_docs_fetch"); - }); - - test("documents plan generation phase", () => { - expect(skill.content).toContain("Plan Generation"); - expect(workflowContent).toContain("infrastructure-plan.json"); - }); - - test("documents IaC generation phase", () => { - expect(skill.content).toContain("IaC Generation"); - expect(skill.content).toContain("Bicep"); - expect(skill.content).toContain("Terraform"); - }); - - test("documents deployment phase", () => { - expect(skill.content).toContain("Deployment"); - expect(skill.content).toContain("az deployment group create"); - expect(skill.content).toContain("terraform apply"); - }); - - test("documents status lifecycle", () => { - expect(workflowContent).toContain("Status Lifecycle"); - expect(workflowContent).toContain("draft"); - expect(workflowContent).toContain("approved"); - expect(workflowContent).toContain("deployed"); - }); - - test("documents plan-first workflow gate", () => { - expect(workflowContent).toContain("wait for approval"); - }); - - test("lists all required MCP tools", () => { - const requiredTools = [ - "microsoft_docs_search", - "microsoft_docs_fetch", - ]; - for (const tool of requiredTools) { - expect(skill.content).toContain(tool); - } - }); - }); - - describe("Plan-First Workflow", () => { - let workflowContent: string; - - beforeAll(() => { - const workflowPath = path.join(skill.path, "references", "workflow.md"); - workflowContent = fs.readFileSync(workflowPath, "utf-8"); - }); - - test("requires user confirmation before deployment", () => { - expect(skill.content.toLowerCase()).toContain("approved"); - expect(skill.content.toLowerCase()).toContain("subscription"); - }); - - test("has blocking plan requirement", () => { - expect(skill.content).toContain("approved"); - }); - - test("references plan schema", () => { - expect(skill.content).toContain("plan-schema.md"); - }); - - test("references verification workflow", () => { - expect(workflowContent).toContain("verification.md"); - }); - }); - - describe("Reference Data Integrity", () => { - let referencesDir: string; - - const categoryFiles = [ - "README.md", - "ai-ml.md", - "compute-apps.md", - "compute-infra.md", - "data-analytics.md", - "data-relational.md", - "messaging.md", - "monitoring.md", - "networking-connectivity.md", - "networking-core.md", - "networking-traffic.md", - "security.md", - ]; - - beforeAll(() => { - referencesDir = path.join(skill.path, "references"); - }); - - test("references directory exists", () => { - expect(fs.existsSync(referencesDir)).toBe(true); - }); - - test("required reference files exist", () => { - const requiredFiles = [ - "plan-schema.md", - "verification.md", - "research.md", - "deployment.md", - "waf-checklist.md", - "pairing-checks.md", - "bicep-generation.md", - "terraform-generation.md", - ]; - for (const file of requiredFiles) { - expect(fs.existsSync(path.join(referencesDir, file))).toBe(true); - } - }); - - test("constraints/ directory exists with all category files", () => { - const constraintsDir = path.join(referencesDir, "constraints"); - expect(fs.existsSync(constraintsDir)).toBe(true); - for (const file of categoryFiles) { - expect(fs.existsSync(path.join(constraintsDir, file))).toBe(true); - } - }); - - test("resources/ directory exists with all category files", () => { - const resourcesDir = path.join(referencesDir, "resources"); - expect(fs.existsSync(resourcesDir)).toBe(true); - for (const file of categoryFiles) { - expect(fs.existsSync(path.join(resourcesDir, file))).toBe(true); - } - }); - - test("resources category files contain ARM type references", () => { - const content = fs.readFileSync( - path.join(referencesDir, "resources", "compute-infra.md"), "utf-8" - ); - expect(content).toContain("Microsoft."); - }); - - test("constraints category files have pairing rules", () => { - const content = fs.readFileSync( - path.join(referencesDir, "constraints", "networking-core.md"), "utf-8" - ); - expect(content.split("\n").length).toBeGreaterThan(10); - }); - - test("plan-schema.md documents the infrastructure plan format", () => { - const content = fs.readFileSync( - path.join(referencesDir, "plan-schema.md"), "utf-8" - ); - expect(content).toContain("meta"); - expect(content).toContain("resources"); - expect(content).toContain("reasoning"); - }); - - }); -}); diff --git a/tests/azure-hosted-copilot-sdk/unit.test.ts b/tests/azure-hosted-copilot-sdk/unit.test.ts deleted file mode 100644 index 1d72d9870..000000000 --- a/tests/azure-hosted-copilot-sdk/unit.test.ts +++ /dev/null @@ -1,149 +0,0 @@ -/** - * Unit Tests for azure-hosted-copilot-sdk - * - * Test skill metadata, content, and frontmatter formatting. - */ - -import { readFileSync } from "node:fs"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-hosted-copilot-sdk"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1025); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("references SDK template", () => { - expect(skill.content).toContain("copilot-sdk-service"); - }); - - test("references deploy workflow", () => { - expect(skill.content).toContain("azure-prepare"); - expect(skill.content).toContain("azure-deploy"); - }); - - test("references SDK documentation", () => { - expect(skill.content).toContain("SDK ref"); - }); - - test("includes deploy-existing path", () => { - expect(skill.content).toContain("deploy existing ref"); - expect(skill.content).toContain("deploy-existing"); - }); - }); - - describe("BYOM Content", () => { - test("includes BYOM routing step", () => { - expect(skill.content).toContain("Step 3"); - expect(skill.content).toContain("BYOM"); - }); - - test("links to azure-model-config reference", () => { - expect(skill.content).toContain("azure-model-config.md"); - }); - - test("mentions DefaultAzureCredential for BYOM", () => { - expect(skill.content).toContain("DefaultAzureCredential"); - }); - - test("description includes BYOM trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("BYOM"); - expect(description).toContain("bring your own model"); - }); - - test("azure-model-config.md reference file exists", () => { - const refPath = skill.filePath.replace("SKILL.md", "references/azure-model-config.md"); - const content = readFileSync(refPath, "utf-8"); - expect(content).toContain("DefaultAzureCredential"); - expect(content).toContain("bearerToken"); - expect(content).toContain("AZURE_OPENAI_ENDPOINT"); - }); - - test("BYOM routing row in step 1 table", () => { - expect(skill.content).toContain("Use Azure/own model"); - expect(skill.content).toContain("Step 3"); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - const supported = ["name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable"]; - const keys = frontmatter.split("\n") - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - - test("USE FOR and DO NOT USE FOR are inside description value, not separate keys", () => { - // ⚠️ Format-only check — passes silently when clause is absent. - // Mandatory existence checks are below (regression guards). - const description = skill.metadata.description; - if (description.includes("USE FOR")) { - expect(description).toContain("USE FOR:"); - } - if (description.includes("DO NOT USE FOR")) { - expect(description).toContain("DO NOT USE FOR:"); - } - }); - - test("description contains DO NOT USE FOR clause to disambiguate from azure-prepare", () => { - // Regression guard: removing the DO NOT USE FOR clause causes - // azure-prepare to win routing on generic deploy prompts like - // "Deploy this app to Azure" when codebase contains @github/copilot-sdk. - // The negative clause is critical for skills that share trigger overlap - // with broader skills (azure-prepare owns "deploy to Azure"). - const description = skill.metadata.description; - expect(description).toContain("DO NOT USE FOR:"); - expect(description).toMatch(/general web apps without copilot SDK/i); - }); - - test("description contains PREFER OVER clause for codebase-based routing", () => { - // Regression guard: the agent must know to prefer this skill over - // azure-prepare when codebase markers (not just prompt keywords) - // indicate a Copilot SDK project. - const description = skill.metadata.description; - expect(description).toMatch(/PREFER OVER azure-prepare/i); - expect(description).toContain("@github/copilot-sdk"); - }); - }); -}); diff --git a/tests/azure-kubernetes/azure-kubernetes-automatic-readiness/unit.test.ts b/tests/azure-kubernetes/azure-kubernetes-automatic-readiness/unit.test.ts deleted file mode 100644 index c564e0345..000000000 --- a/tests/azure-kubernetes/azure-kubernetes-automatic-readiness/unit.test.ts +++ /dev/null @@ -1,146 +0,0 @@ -/** - * Unit Tests for azure-kubernetes-automatic-readiness - * - * Tests skill content and structure without requiring external services. - */ - -import { readFileSync } from "node:fs"; -import { loadSkill, LoadedSkill } from "../../utils/skill-loader"; - -const SKILL_NAME = "azure-kubernetes-automatic-readiness"; -const SKILL_PATH = "azure-kubernetes/azure-kubernetes-automatic-readiness"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_PATH); - }); - - describe("Skill Metadata", () => { - test("has required frontmatter fields", () => { - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(50); - }); - - test("description contains WHEN triggers", () => { - expect(skill.metadata.description).toContain("WHEN:"); - }); - - test("description mentions AKS Automatic", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/aks automatic/); - }); - - test("description is within 1024 character limit", () => { - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - }); - - describe("Required Section Structure", () => { - test("contains the standard skill sections", () => { - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## When to Use This Skill"); - expect(skill.content).toContain("## MCP Tools"); - expect(skill.content).toContain("## Workflow"); - expect(skill.content).toContain("## Error Handling"); - }); - }); - - describe("Routing Rules", () => { - test("routes cluster creation to azure-kubernetes", () => { - expect(skill.content).toMatch(/Route to `azure-kubernetes` instead/i); - }); - - test("routes live troubleshooting to azure-diagnostics", () => { - expect(skill.content).toMatch(/Route to `azure-diagnostics` instead/i); - }); - }); - - describe("Assessment Workflow", () => { - test("covers cluster-connected assessment via MCP", () => { - expect(skill.content).toMatch(/mcp_azure_mcp_aks/i); - expect(skill.content).toMatch(/discover/i); - }); - - test("covers offline manifest validation", () => { - expect(skill.content).toMatch(/offline/i); - expect(skill.content).toMatch(/constraint-spec/i); - }); - - test("defines MCP-to-offline fallback chain", () => { - expect(skill.content).toMatch(/fallback/i); - expect(skill.content).toMatch(/offline/i); - }); - - test("includes MCP setup guidance when tool is unavailable", () => { - expect(skill.content).toMatch(/aka\.ms\/azure-mcp-setup/i); - }); - }); - - describe("Severity Classification", () => { - test("documents incompatible severity", () => { - expect(skill.content).toMatch(/incompatible/i); - }); - - test("documents requiresChanges severity", () => { - expect(skill.content).toMatch(/requiresChanges/i); - }); - - test("documents autoFixed severity", () => { - expect(skill.content).toMatch(/autoFixed/i); - }); - }); - - describe("Fix Guidance", () => { - test("covers deterministic fixes", () => { - expect(skill.content).toMatch(/deterministic/i); - }); - - test("covers LLM-reasoned fixes", () => { - expect(skill.content).toMatch(/LLM/i); - }); - - test("requires user approval before applying fixes", () => { - expect(skill.content).toMatch(/accept|approval|explicit/i); - }); - }); - - describe("Guardrails", () => { - test("enforces read-only cluster assessment", () => { - expect(skill.content).toMatch(/read-only/i); - }); - - test("prohibits exposing secrets", () => { - expect(skill.content).toMatch(/secret/i); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatterMatch = raw.match(/^---\r?\n([\s\S]*?)\r?\n---/); - const frontmatter = frontmatterMatch ? frontmatterMatch[1] : ""; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatterMatch = raw.match(/^---\r?\n([\s\S]*?)\r?\n---/); - const frontmatter = frontmatterMatch ? frontmatterMatch[1] : ""; - const supported = ["name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable"]; - const keys = frontmatter.split(/\r?\n/) - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - - test("WHEN clause is inside description", () => { - expect(skill.metadata.description).toContain("WHEN:"); - }); - }); -}); diff --git a/tests/azure-kubernetes/unit.test.ts b/tests/azure-kubernetes/unit.test.ts deleted file mode 100644 index 8112278bd..000000000 --- a/tests/azure-kubernetes/unit.test.ts +++ /dev/null @@ -1,192 +0,0 @@ -/** - * Unit Tests for azure-kubernetes - * - * Tests skill content and structure without requiring external services. - * Focuses on domain invariants rather than exact formatting. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-kubernetes"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has required frontmatter fields", () => { - expect(skill.metadata.name).toBe("azure-kubernetes"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(50); - }); - - test("description contains WHEN triggers", () => { - expect(skill.metadata.description).toMatch(/WHEN:/i); - }); - - test("description mentions key AKS concepts", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/aks|kubernetes/); - // Description should mention core topics covered by the skill - expect(desc).toMatch(/cluster|networking|security|deploy/); - }); - }); - - describe("Required Section Structure", () => { - test("contains the standard skill sections", () => { - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## When to Use This Skill"); - expect(skill.content).toContain("## MCP Tools"); - expect(skill.content).toContain("## Workflow"); - expect(skill.content).toContain("## Error Handling"); - }); - }); - - describe("Day-0 vs Day-1 Guidance", () => { - test("distinguishes between Day-0 and Day-1 decisions", () => { - expect(skill.content).toMatch(/Day-0/i); - expect(skill.content).toMatch(/Day-1/i); - }); - - test("identifies networking as Day-0 decision", () => { - expect(skill.content).toMatch(/networking.*day-0|day-0.*networking/i); - }); - - test("identifies API server access as Day-0 consideration", () => { - expect(skill.content).toMatch(/api server/i); - }); - }); - - describe("Cluster SKU Guidance", () => { - test("covers AKS Automatic SKU", () => { - expect(skill.content).toMatch(/AKS Automatic/i); - }); - - test("covers AKS Standard SKU", () => { - expect(skill.content).toMatch(/AKS Standard/i); - }); - - test("recommends Automatic as default", () => { - expect(skill.content).toMatch(/automatic.*default|default.*automatic/i); - }); - }); - - describe("Networking Guidance", () => { - test("covers Azure CNI options", () => { - expect(skill.content).toMatch(/Azure CNI/i); - }); - - test("covers overlay networking", () => { - expect(skill.content).toMatch(/overlay/i); - }); - - test("covers egress patterns", () => { - expect(skill.content).toMatch(/egress/i); - }); - - test("covers ingress options", () => { - expect(skill.content).toMatch(/ingress/i); - }); - }); - - describe("Security Best Practices", () => { - test("recommends Entra ID / Azure AD", () => { - expect(skill.content).toMatch(/entra|azure ad/i); - }); - - test("recommends Workload Identity", () => { - expect(skill.content).toMatch(/workload identity/i); - }); - - test("recommends Key Vault integration", () => { - expect(skill.content).toMatch(/key vault/i); - }); - - test("warns against static credentials", () => { - expect(skill.content).toMatch(/avoid.*static|static.*credential/i); - }); - - test("mentions Azure Policy", () => { - expect(skill.content).toMatch(/azure policy/i); - }); - }); - - describe("Observability Guidance", () => { - test("mentions monitoring options", () => { - expect(skill.content).toMatch(/monitor|observability/i); - }); - - test("mentions Prometheus", () => { - expect(skill.content).toMatch(/prometheus/i); - }); - - test("mentions Grafana", () => { - expect(skill.content).toMatch(/grafana/i); - }); - }); - - describe("Reliability Patterns", () => { - test("recommends availability zones", () => { - expect(skill.content).toMatch(/availability zone|--zones/i); - }); - - test("mentions PodDisruptionBudgets", () => { - expect(skill.content).toMatch(/poddisruptionbudget|pdb/i); - }); - - test("covers upgrade strategy", () => { - expect(skill.content).toMatch(/upgrade/i); - }); - - test("mentions maintenance windows", () => { - expect(skill.content).toMatch(/maintenance window/i); - }); - }); - - describe("Performance Recommendations", () => { - test("recommends ephemeral OS disks", () => { - expect(skill.content).toMatch(/ephemeral.*disk|--node-osdisk-type ephemeral/i); - }); - - test("warns against B-series VMs", () => { - expect(skill.content).toMatch(/avoid.*b-series|b-series.*avoid/i); - }); - - test("mentions autoscaling", () => { - expect(skill.content).toMatch(/autoscal|cluster.?autoscaler/i); - }); - }); - - describe("MCP Tools Section", () => { - test("lists MCP tools", () => { - expect(skill.content).toMatch(/mcp_azure_mcp_aks|mcp_aks_mcp/i); - }); - - test("has MCP Tools section", () => { - expect(skill.content).toMatch(/## MCP Tools/i); - }); - }); - - describe("Error Handling Section", () => { - test("has Error Handling section", () => { - expect(skill.content).toMatch(/## Error Handling/i); - }); - - test("includes remediation guidance", () => { - expect(skill.content).toMatch(/remediation|quota|policy/i); - }); - }); - - describe("Guardrails", () => { - test("warns about secrets handling", () => { - expect(skill.content).toMatch(/secret|token|key/i); - }); - - test("does not promise zero downtime", () => { - expect(skill.content).toMatch(/do not promise zero downtime/i); - }); - }); -}); diff --git a/tests/azure-messaging/unit.test.ts b/tests/azure-messaging/unit.test.ts deleted file mode 100644 index 255ffc6e2..000000000 --- a/tests/azure-messaging/unit.test.ts +++ /dev/null @@ -1,98 +0,0 @@ -/** - * Unit Tests for azure-messaging - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-messaging"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is comprehensive and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - - test("has messaging-specific trigger keywords", () => { - const description = skill.metadata.description.toLowerCase(); - const hasMessagingKeywords = - description.includes("event hub") || - description.includes("service bus") || - description.includes("amqp") || - description.includes("messaging"); - expect(hasMessagingKeywords).toBe(true); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(500); - }); - - test("contains diagnosis workflow section", () => { - expect(skill.content).toContain("Diagnosis Workflow"); - }); - - test("contains troubleshooting guides section", () => { - expect(skill.content).toContain("Troubleshooting Guides"); - }); - - test("directs to azure-diagnostics skill for troubleshooting content", () => { - expect(skill.content).toContain( - "troubleshooting guides are located in the azure-diagnostics skill under `troubleshooting/messaging/`" - ); - }); - }); - - describe("MCP Tools Documentation", () => { - test("references messaging MCP tools", () => { - expect(skill.content).toContain("mcp_azure_mcp_eventhubs"); - expect(skill.content).toContain("mcp_azure_mcp_servicebus"); - }); - - test("references diagnostic MCP tools", () => { - expect(skill.content).toContain("mcp_azure_mcp_monitor"); - expect(skill.content).toContain("mcp_azure_mcp_resourcehealth"); - expect(skill.content).toContain("mcp_azure_mcp_documentation"); - }); - }); - - describe("Reference Files", () => { - test("does not duplicate troubleshooting files locally", () => { - // Troubleshooting content moved to azure-diagnostics; no local references/ links should remain - expect(skill.content).not.toContain("references/sdk/"); - expect(skill.content).not.toContain("references/service-troubleshooting.md"); - expect(skill.content).not.toContain("references/auth-best-practices.md"); - }); - }); - - describe("Diagnosis Workflow", () => { - test("includes systematic diagnosis steps", () => { - expect(skill.content).toContain("Identify the SDK and version"); - expect(skill.content).toContain("Check resource health"); - expect(skill.content).toContain("Review the error message"); - expect(skill.content).toContain("Recommend fix"); - }); - }); -}); diff --git a/tests/azure-prepare/unit.test.ts b/tests/azure-prepare/unit.test.ts deleted file mode 100644 index 686bd1610..000000000 --- a/tests/azure-prepare/unit.test.ts +++ /dev/null @@ -1,143 +0,0 @@ -/** - * Unit Tests for azure-prepare - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-prepare"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - // Descriptions should be 150-2048 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(2048); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Triggers"); - expect(skill.content).toContain("## Rules"); - expect(skill.content).toContain("## Phase 1: Planning"); - expect(skill.content).toContain("## Phase 2: Execution"); - expect(skill.content).toContain("## Outputs"); - }); - - test("references azure-validate for next steps", () => { - expect(skill.content).toContain("azure-validate"); - }); - }); - - describe("Plan-First Workflow", () => { - test("mentions plan file requirement", () => { - expect(skill.content).toContain(".azure/deployment-plan.md"); - }); - - test("requires user confirmation for subscription and location", () => { - expect(skill.content.toLowerCase()).toContain("subscription"); - expect(skill.content.toLowerCase()).toContain("location"); - }); - - test("has blocking plan requirement", () => { - expect(skill.content).toContain("PLAN-FIRST"); - expect(skill.content).toContain("BLOCKING"); - }); - }); - - describe("Functional Verification Step", () => { - test("includes functional verification step in workflow", () => { - expect(skill.content).toContain("Functional Verification"); - }); - - test("references functional-verification.md", () => { - expect(skill.content).toContain("functional-verification.md"); - }); - - test("functional verification comes before plan update", () => { - const funcVerifIndex = skill.content.indexOf("Functional Verification"); - const updatePlanIndex = skill.content.indexOf("Update Plan"); - expect(funcVerifIndex).toBeGreaterThan(-1); - expect(updatePlanIndex).toBeGreaterThan(-1); - expect(funcVerifIndex).toBeLessThan(updatePlanIndex); - }); - }); - - describe("Subscription Policy Checks", () => { - test("references policy tool in requirements", () => { - const refsDir = path.join( - SKILLS_PATH, - "azure-prepare/references/requirements.md" - ); - const content = fs.readFileSync(refsDir, "utf-8"); - expect(content).toContain("mcp_azure_mcp_policy"); - }); - - test("mentions subscription policies in requirements", () => { - const refsDir = path.join( - SKILLS_PATH, - "azure-prepare/references/requirements.md" - ); - const content = fs.readFileSync(refsDir, "utf-8"); - expect(content).toContain("Subscription Policies"); - expect(content).toContain("policy_assignment_list"); - }); - }); - - describe("Aspire Support", () => { - test("aspire.md reference file exists", () => { - const aspirePath = path.join( - SKILLS_PATH, - "azure-prepare/references/recipes/azd/aspire.md" - ); - expect(fs.existsSync(aspirePath)).toBe(true); - }); - - test("aspire.md contains AddDockerfile guidance", () => { - const aspirePath = path.join( - SKILLS_PATH, - "azure-prepare/references/recipes/azd/aspire.md" - ); - const aspireContent = fs.readFileSync(aspirePath, "utf-8"); - expect(aspireContent).toContain("AddDockerfile"); - expect(aspireContent).toContain("container builds"); - }); - - test("azure-yaml.md references aspire.md", () => { - const azureYamlPath = path.join( - SKILLS_PATH, - "azure-prepare/references/recipes/azd/azure-yaml.md" - ); - const azureYamlContent = fs.readFileSync(azureYamlPath, "utf-8"); - expect(azureYamlContent).toContain("aspire.md"); - expect(azureYamlContent).toContain("docker"); - }); - }); -}); diff --git a/tests/azure-quotas/unit.test.ts b/tests/azure-quotas/unit.test.ts deleted file mode 100644 index 4185b586d..000000000 --- a/tests/azure-quotas/unit.test.ts +++ /dev/null @@ -1,185 +0,0 @@ -/** - * Unit Tests for azure-quotas - * - * Test isolated skill logic and validation rules. - */ - -import { readFileSync } from "node:fs"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-quotas"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - const hasTriggerPhrases = - description.includes("use for") || - description.includes("use when") || - description.includes("when:") || - description.includes("helps") || - description.includes("activate") || - description.includes("trigger"); - expect(hasTriggerPhrases).toBe(true); - }); - - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("mentions az quota CLI commands", () => { - expect(skill.content).toContain("az quota"); - }); - - test("mentions the quota extension requirement", () => { - expect(skill.content).toContain("az extension add --name quota"); - }); - - test("mentions az quota list command", () => { - expect(skill.content).toContain("az quota list"); - }); - - test("mentions az quota show command", () => { - expect(skill.content).toContain("az quota show"); - }); - - test("mentions az quota usage show command", () => { - expect(skill.content).toContain("az quota usage show"); - }); - - test("mentions az quota update command for requesting increases", () => { - expect(skill.content).toContain("az quota update"); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - const supported = [ - "name", "description", "compatibility", "license", "metadata", - "argument-hint", "disable-model-invocation", "user-invokable" - ]; - const keys = frontmatter.split("\n") - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - - test("USE FOR and DO NOT USE FOR are inside description value, not separate keys", () => { - // ⚠️ These conditionals are format-only checks – they verify correct - // punctuation when the clause exists, but pass silently when it is absent. - // If this skill gains routing competition with a broader skill (e.g. - // azure-prepare) in the future, convert these to mandatory existence - // checks like azure-hosted-copilot-sdk/unit.test.ts. - const description = skill.metadata.description; - if (description.includes("USE FOR")) { - expect(description).toContain("USE FOR:"); - } - if (description.includes("DO NOT USE FOR")) { - expect(description).toContain("DO NOT USE FOR:"); - } - }); - }); - - describe("Core Workflows", () => { - test("describes checking quota for a specific resource", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/check quota|verify quota|show quota/); - }); - - test("describes comparing quotas across regions", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/compare.*region|across.*region|multi.*region/); - }); - - test("describes requesting quota increases", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/request.*increase|quota.*increase|increase.*quota/); - }); - - test("describes listing all quotas for planning", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/list.*quota|quota.*list/); - }); - }); - - describe("Resource Name Mapping", () => { - test("warns about no 1:1 mapping between ARM types and quota names", () => { - const content = skill.content.toLowerCase(); - expect(content).toMatch(/no 1:1 mapping|resource name mapping/); - }); - - test("describes discovery workflow for quota resource names", () => { - expect(skill.content.toLowerCase()).toContain("localizedvalue"); - }); - }); - - describe("Scope Format", () => { - test("documents the required scope pattern", () => { - expect(skill.content).toContain("/subscriptions/"); - expect(skill.content).toContain("/providers/"); - expect(skill.content).toContain("/locations/"); - }); - }); - - describe("Troubleshooting", () => { - test("mentions ExtensionNotFound error", () => { - expect(skill.content).toContain("ExtensionNotFound"); - }); - - test("mentions BadRequest for unsupported providers", () => { - expect(skill.content).toContain("BadRequest"); - }); - - test("mentions MissingRegistration error", () => { - expect(skill.content).toContain("MissingRegistration"); - }); - - test("mentions QuotaExceeded error", () => { - expect(skill.content).toContain("QuotaExceeded"); - }); - - test("lists known unsupported providers", () => { - expect(skill.content).toContain("Microsoft.DocumentDB"); - }); - - test("lists confirmed working providers", () => { - const content = skill.content; - expect(content).toContain("Microsoft.Compute"); - expect(content).toContain("Microsoft.Network"); - expect(content).toContain("Microsoft.App"); - expect(content).toContain("Microsoft.Storage"); - }); - }); -}); diff --git a/tests/azure-rbac/unit.test.ts b/tests/azure-rbac/unit.test.ts deleted file mode 100644 index 410669cc8..000000000 --- a/tests/azure-rbac/unit.test.ts +++ /dev/null @@ -1,134 +0,0 @@ -/** - * Unit Tests for azure-rbac - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-rbac"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - const hasTriggerPhrases = - description.includes("use for") || - description.includes("use when") || - description.includes("helps") || - description.includes("activate") || - description.includes("trigger"); - expect(hasTriggerPhrases).toBe(true); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("mentions azure documentation tool", () => { - expect(skill.content.toLowerCase()).toContain("azure__documentation"); - }); - - test("mentions CLI generation", () => { - expect(skill.content.toLowerCase()).toContain("cli"); - }); - - test("mentions Bicep", () => { - expect(skill.content.toLowerCase()).toContain("bicep"); - }); - - test("mentions finding minimal role definition", () => { - expect(skill.content.toLowerCase()).toMatch(/minimal role|least privilege|role definition/); - }); - - test("mentions custom role creation", () => { - expect(skill.content.toLowerCase()).toContain("custom role"); - }); - - test("includes azure__extension_cli_generate tool", () => { - expect(skill.content.toLowerCase()).toContain("azure__extension_cli_generate"); - }); - - test("mentions azure__bicepschema tool", () => { - expect(skill.content.toLowerCase()).toContain("azure__bicepschema"); - }); - - test("includes azure__get_azure_bestpractices tool", () => { - expect(skill.content.toLowerCase()).toContain("azure__get_azure_bestpractices"); - }); - }); - - describe("RBAC Role Assignment Workflow", () => { - test("describes workflow for finding roles", () => { - const content = skill.content.toLowerCase(); - const hasWorkflow = content.includes("find") || content.includes("search") || content.includes("documentation"); - expect(hasWorkflow).toBe(true); - }); - - test("mentions role assignment process", () => { - expect(skill.content.toLowerCase()).toMatch(/assign.*role|role.*assign/); - }); - - test("includes guidance for no built-in role scenario", () => { - const content = skill.content.toLowerCase(); - const hasCustomRoleGuidance = content.includes("no built-in role") || content.includes("custom role"); - expect(hasCustomRoleGuidance).toBe(true); - }); - }); - - describe("Output Expectations", () => { - test("mentions generating CLI commands", () => { - expect(skill.content.toLowerCase()).toMatch(/cli command|generate.*command/); - }); - - test("mentions providing Bicep code snippet", () => { - expect(skill.content.toLowerCase()).toMatch(/bicep.*code|bicep.*snippet/); - }); - - test("references role assignment in Bicep", () => { - const content = skill.content.toLowerCase(); - const hasBicepAssignment = content.includes("role assignment") && content.includes("bicep"); - expect(hasBicepAssignment).toBe(true); - }); - }); - - describe("Prerequisites Guidance", () => { - test("mentions prerequisites for granting roles", () => { - const content = skill.content.toLowerCase(); - const hasPrerequisites = content.includes("prerequisite") || content.includes("permission"); - expect(hasPrerequisites).toBe(true); - }); - - test("mentions User Access Administrator role", () => { - expect(skill.content.toLowerCase()).toContain("user access administrator"); - }); - - test("mentions authorization write permission", () => { - const content = skill.content.toLowerCase(); - const hasAuthPermission = content.includes("microsoft.authorization/roleassignments/write") || - content.includes("roleassignments/write"); - expect(hasAuthPermission).toBe(true); - }); - }); -}); diff --git a/tests/azure-reliability/unit.test.ts b/tests/azure-reliability/unit.test.ts deleted file mode 100644 index 0bfa99b0c..000000000 --- a/tests/azure-reliability/unit.test.ts +++ /dev/null @@ -1,245 +0,0 @@ -/** - * Unit Tests for azure-reliability - * - * Test isolated skill logic and validation rules. - */ - -import { existsSync, readFileSync } from "node:fs"; -import * as path from "node:path"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-reliability"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - - test("description scopes the skill to Azure Functions", () => { - const description = skill.metadata.description.toLowerCase(); - const content = skill.content.toLowerCase(); - expect(description).toContain("functions"); - expect(content).toContain("this skill currently covers **azure functions** only"); - expect(description).not.toMatch( - /reliability posture of azure functions, container apps, and app service/ - ); - }); - - test("description mentions core reliability features", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("zone redundancy"); - expect(description).toContain("multi-region"); - }); - }); - - describe("Skill Content Structure", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(1000); - }); - - test("contains expected top-level sections", () => { - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## When to Use This Skill"); - expect(skill.content).toContain("## Prerequisites"); - expect(skill.content).toContain("## Assessment Workflow"); - expect(skill.content).toContain("## Configuration Workflow"); - expect(skill.content).toContain("## Skill Boundaries"); - }); - - test("documents the three assessment phases", () => { - expect(skill.content).toContain("Phase 1: Discover Resources"); - expect(skill.content).toContain("Phase 2: Assess Reliability"); - expect(skill.content).toContain("Phase 3: Generate Reliability Checklist"); - }); - - test("documents the configuration workflow steps", () => { - expect(skill.content).toContain("Step 1: Present Fix Plan"); - expect(skill.content).toContain("Path A: Fix Now (CLI)"); - expect(skill.content).toContain("Path B: Patch IaC"); - expect(skill.content).toContain("Re-Assess"); - expect(skill.content).toContain("Multi-region follow-up"); - }); - }); - - describe("Assessment Output Format", () => { - test("feature-pivoted table contains all four feature rows", () => { - // The four reliability features must all appear in the assessment table - expect(skill.content).toContain("Zone redundancy — compute"); - expect(skill.content).toContain("Zone-redundant storage"); - expect(skill.content).toContain("Health probes"); - expect(skill.content).toContain("Multi-region failover"); - }); - - test("uses traffic-light status indicators (no scoring)", () => { - expect(skill.content).toContain("🟢 ON"); - expect(skill.content).toContain("🟡 PARTIAL"); - expect(skill.content).toContain("🔴 OFF"); - }); - - test("explicitly forbids numeric scoring", () => { - // Two anti-scoring statements live in SKILL.md - expect(skill.content).toContain("Do **not** assign numeric scores"); - expect(skill.content).toContain("include numeric scores, grades, or point totals"); - }); - - test("does not contain a /10 score anywhere", () => { - // Refactored away from "Score: X/10" output - expect(skill.content).not.toMatch(/\bScore:\s*\d+\/10\b/); - }); - }); - - describe("Staged Remediation (Quick Wins → Storage → Multi-region)", () => { - test("Path A executes quick wins before storage migration", () => { - const pathA = - skill.content - .split("### Path A: Fix Now (CLI)")[1] - ?.split("### Path B")[0] ?? ""; - const quickWinsIdx = pathA.indexOf("quick wins first"); - const storageStopIdx = pathA.indexOf("Ask about storage upgrade"); - expect(quickWinsIdx).toBeGreaterThan(-1); - expect(storageStopIdx).toBeGreaterThan(quickWinsIdx); - }); - - test("Path B uses two-deploy flow with storage SKU isolated", () => { - expect(skill.content).toContain("Deploy 1"); - expect(skill.content).toContain("Deploy 2"); - expect(skill.content).toContain("storage SKU patch"); - }); - - test("multi-region requires explicit user consent", () => { - const step3 = - skill.content.split("### Step 3 (both paths): Multi-region")[1] ?? ""; - expect(step3).toContain("ASK and WAIT"); - expect(step3).toContain("yes / no / later"); - expect(step3).toContain("Do not skip the wait"); - }); - }); - - describe("Skill Drives Deploys Itself", () => { - test("documents that the skill executes deployments", () => { - expect(skill.content).toContain("the skill runs the deploy itself"); - }); - - test("Skill Boundaries marks 'Deploy IaC' as Yes", () => { - const boundaries = - skill.content.split("## Skill Boundaries")[1] ?? ""; - expect(boundaries).toMatch( - /Deploy IaC for reliability changes\s*\|\s*✅ Yes/ - ); - }); - }); - - describe("FC1 Health Probe Consent Gate", () => { - test("flags FC1 / Consumption health checks as code-only with consent required", () => { - expect(skill.content).toContain("FC1 / Consumption"); - expect(skill.content).toContain("Code-only"); - expect(skill.content).toContain("ask the user for explicit consent"); - }); - }); - - describe("References", () => { - let referencesDir: string; - - beforeAll(() => { - // skill.path points to the built skill directory (output/skills/{skill-name}) - referencesDir = path.join(skill.path, "references"); - }); - - const requiredReferences = [ - "zone-redundancy-checks.md", - "storage-redundancy-checks.md", - "multi-region-checks.md", - "health-probe-checks.md", - "configure-zone-redundancy.md", - "configure-storage.md", - "configure-health-probes.md", - "configure-multi-region.md", - "iac-patching-bicep.md", - "iac-patching-terraform.md", - ]; - - test.each(requiredReferences)( - "shared reference exists: %s", - (filename) => { - const fullPath = path.join(referencesDir, filename); - expect(existsSync(fullPath)).toBe(true); - } - ); - - test("Functions service reference exists", () => { - const fullPath = path.join( - referencesDir, - "services", - "functions", - "reliability.md" - ); - expect(existsSync(fullPath)).toBe(true); - }); - - test("Functions service reference covers expected sections", () => { - const fullPath = path.join( - referencesDir, - "services", - "functions", - "reliability.md" - ); - const content = readFileSync(fullPath, "utf-8"); - expect(content).toContain("Zone Redundancy"); - expect(content).toContain("Health Endpoint"); - expect(content).toContain("Multi-Region"); - expect(content).toContain("Reporting"); - }); - }); - - describe("Frontmatter Formatting", () => { - test("frontmatter has no tabs", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - expect(frontmatter).not.toMatch(/\t/); - }); - - test("frontmatter keys are only supported attributes", () => { - const raw = readFileSync(skill.filePath, "utf-8"); - const frontmatter = raw.split("---")[1]; - const supported = [ - "name", - "description", - "compatibility", - "license", - "metadata", - "argument-hint", - "disable-model-invocation", - "user-invokable", - ]; - const keys = frontmatter - .split("\n") - .filter((l: string) => /^[a-z][\w-]*\s*:/.test(l)) - .map((l: string) => l.split(":")[0].trim()); - for (const key of keys) { - expect(supported).toContain(key); - } - }); - }); -}); diff --git a/tests/azure-resource-lookup/unit.test.ts b/tests/azure-resource-lookup/unit.test.ts deleted file mode 100644 index 536ad9f20..000000000 --- a/tests/azure-resource-lookup/unit.test.ts +++ /dev/null @@ -1,96 +0,0 @@ -/** - * Unit Tests for azure-resource-lookup - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-resource-lookup"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("USE FOR:"); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toContain("DO NOT USE FOR:"); - }); - - test("description mentions cross-cutting resource queries", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toMatch(/across.*subscription/); - expect(description).toMatch(/orphaned|unattached/); - expect(description).toMatch(/tag/); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(500); - }); - - test("contains required sections", () => { - expect(skill.content).toContain("## When to Use This Skill"); - expect(skill.content).toContain("## Quick Reference"); - expect(skill.content).toContain("## MCP Tools"); - expect(skill.content).toContain("## Workflow"); - expect(skill.content).toContain("## Error Handling"); - }); - - test("positions as resource lookup skill with MCP routing", () => { - expect(skill.content).toMatch(/dedicated.*MCP tool/i); - expect(skill.content).toContain("extension_cli_generate"); - }); - - test("references Azure Resource Graph", () => { - expect(skill.content).toContain("Azure Resource Graph"); - expect(skill.content).toContain("az graph query"); - }); - - test("documents extension_cli_generate as primary tool", () => { - expect(skill.content).toContain("extension_cli_generate"); - }); - - test("links to ARG reference documentation", () => { - expect(skill.content).toContain("references/azure-resource-graph.md"); - }); - }); - - describe("Error Handling", () => { - test("documents common errors", () => { - expect(skill.content).toContain("resource-graph extension not found"); - expect(skill.content).toContain("AuthorizationFailed"); - expect(skill.content).toContain("BadRequest"); - }); - }); - - describe("Constraints", () => { - test("documents query best practices", () => { - expect(skill.content).toContain("=~"); - expect(skill.content).toContain("case-insensitive"); - }); - }); -}); diff --git a/tests/azure-resource-visualizer/unit.test.ts b/tests/azure-resource-visualizer/unit.test.ts deleted file mode 100644 index a1e2401e7..000000000 --- a/tests/azure-resource-visualizer/unit.test.ts +++ /dev/null @@ -1,88 +0,0 @@ -/** - * Unit Tests for azure-resource-visualizer - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-resource-visualizer"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is concise and actionable", () => { - // Descriptions should be 50-1024 chars for readability - expect(skill.metadata.description.length).toBeGreaterThan(50); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description; - expect(description).toContain("WHEN:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Core Responsibilities"); - expect(skill.content).toContain("## Workflow Process"); - expect(skill.content).toContain("Resource Group Selection"); - expect(skill.content).toContain("Resource Discovery & Analysis"); - expect(skill.content).toContain("Diagram Construction"); - }); - - test("includes Mermaid diagram examples", () => { - expect(skill.content).toContain("Mermaid"); - expect(skill.content).toContain("graph TB"); - expect(skill.content).toContain("graph LR"); - }); - - test("defines quality standards", () => { - expect(skill.content).toContain("Quality Standards"); - expect(skill.content).toContain("Accuracy"); - expect(skill.content).toContain("Completeness"); - }); - }); - - describe("Diagram Generation Guidelines", () => { - test("provides diagram structure guidelines", () => { - expect(skill.content).toContain("Key Diagram Requirements"); - expect(skill.content).toContain("subgraph"); - expect(skill.content).toContain("Resource Group"); - }); - - test("includes relationship mapping guidance", () => { - expect(skill.content).toContain("Relationship Mapping"); - expect(skill.content).toContain("Network connections"); - expect(skill.content).toContain("Data flow"); - }); - }); - - describe("Azure Resource Graph Integration", () => { - test("links to Azure Resource Graph reference", () => { - expect(skill.content).toContain("references/azure-resource-graph.md"); - }); - - test("mentions Resource Graph for resource discovery", () => { - expect(skill.content).toContain("Azure Resource Graph"); - }); - }); -}); diff --git a/tests/azure-upgrade/unit.test.ts b/tests/azure-upgrade/unit.test.ts deleted file mode 100644 index e4c5cd5d3..000000000 --- a/tests/azure-upgrade/unit.test.ts +++ /dev/null @@ -1,174 +0,0 @@ -/** - * Unit Tests for azure-upgrade - * - * Test isolated skill logic and validation rules. - */ - -import { spawnSync } from "node:child_process"; -import { existsSync } from "node:fs"; -import * as path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-upgrade"; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description meets Medium-High compliance length", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThanOrEqual(1024); - }); - - test("description contains WHEN trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("when:"); - }); - - test("description word count is within limit", () => { - const words = skill.metadata.description.split(/\s+/).length; - expect(words).toBeLessThanOrEqual(60); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Triggers"); - expect(skill.content).toContain("## Rules"); - expect(skill.content).toContain("## Steps"); - expect(skill.content).toContain("## Upgrade Scenarios"); - expect(skill.content).toContain("## MCP Tools"); - }); - - test("references azure-validate for post-upgrade validation", () => { - expect(skill.content).toContain("azure-validate"); - }); - - test("references azure-deploy for CI/CD hand-off", () => { - expect(skill.content).toContain("azure-deploy"); - }); - - test("distinguishes from azure-cloud-migrate", () => { - expect(skill.content).toContain("azure-cloud-migrate"); - }); - }); - - describe("Upgrade Workflow", () => { - test("mentions assessment phase", () => { - expect(skill.content.toLowerCase()).toContain("assess"); - }); - - test("includes identify phase", () => { - expect(skill.content.toLowerCase()).toContain("identify"); - }); - - test("includes pre-migrate phase", () => { - expect(skill.content.toLowerCase()).toContain("pre-migrate"); - }); - - test("includes validate phase", () => { - expect(skill.content.toLowerCase()).toContain("validate"); - }); - - test("tracks progress in upgrade-status.md", () => { - expect(skill.content).toContain("upgrade-status.md"); - }); - - test("references upgrade scenarios", () => { - const content = skill.content.toLowerCase(); - const hasConsumptionToFlex = content.includes("consumption") && content.includes("flex"); - const hasScenarios = content.includes("upgrade scenarios"); - expect(hasConsumptionToFlex || hasScenarios).toBe(true); - }); - - test("references consumption-to-flex scenario", () => { - expect(skill.content).toContain("consumption-to-flex.md"); - }); - - test("references workflow details", () => { - expect(skill.content).toContain("workflow-details.md"); - }); - - test("references global rules", () => { - expect(skill.content).toContain("global-rules.md"); - }); - }); - - describe("Safety and Confirmation", () => { - test("requires user confirmation for destructive actions", () => { - const content = skill.content.toLowerCase(); - expect(content).toContain("confirm"); - }); - - test("requires idempotent and resumable scripts", () => { - const content = skill.content.toLowerCase(); - expect(content).toContain("idempotent"); - }); - - test("prohibits deleting original app without confirmation", () => { - const content = skill.content.toLowerCase(); - const hasDeleteProtection = content.includes("never delete") || content.includes("explicit user confirmation"); - expect(hasDeleteProtection).toBe(true); - }); - }); - - describe("Helper Scripts (pytest)", () => { - const pytestDir = path.resolve(__dirname, "scripts"); - const pytestFile = path.join(pytestDir, "test_upgrade_bom.py"); - - test("upgrade_bom.py pytest suite passes", () => { - if (!existsSync(pytestFile)) { - throw new Error(`Expected pytest file at ${pytestFile}`); - } - - // Resolve a Python interpreter; skip gracefully on environments without one. - const pythonCmd = process.env.PYTHON ?? "python3"; - const probe = spawnSync(pythonCmd, ["--version"], { encoding: "utf8" }); - if (probe.status !== 0) { - console.warn(`[azure-upgrade] Skipping pytest: '${pythonCmd}' not available.`); - return; - } - - // Ensure pytest is importable; skip if unavailable so contributors without - // pytest installed aren't blocked. - const pytestProbe = spawnSync(pythonCmd, ["-c", "import pytest"], { encoding: "utf8" }); - if (pytestProbe.status !== 0) { - console.warn( - `[azure-upgrade] Skipping pytest: pytest not installed for '${pythonCmd}'. ` + - `Install with: ${pythonCmd} -m pip install pytest` - ); - return; - } - - const result = spawnSync( - pythonCmd, - ["-m", "pytest", "test_upgrade_bom.py", "-v"], - { cwd: pytestDir, encoding: "utf8" } - ); - - expect(result.status).toBe(0); - }, 60_000); - }); -}); diff --git a/tests/azure-validate/unit.test.ts b/tests/azure-validate/unit.test.ts deleted file mode 100644 index ecf960169..000000000 --- a/tests/azure-validate/unit.test.ts +++ /dev/null @@ -1,90 +0,0 @@ -/** - * Unit Tests for azure-validate - * - * Tests for deployment readiness validation skill. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-validate"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description mentions validation or deployment readiness", () => { - const desc = skill.metadata.description.toLowerCase(); - expect(desc).toMatch(/validate|validation|ready|deployment|preflight/); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(200); - }); - - test("contains triggers section", () => { - expect(skill.content).toMatch(/trigger/i); - }); - - test("documents the workflow steps", () => { - expect(skill.content).toMatch(/step/i); - }); - - test("references azure-prepare prerequisite", () => { - expect(skill.content).toMatch(/azure-prepare/i); - }); - - test("references azure-deploy as next step", () => { - expect(skill.content).toMatch(/azure-deploy/i); - }); - - test("mentions plan file", () => { - expect(skill.content).toMatch(/plan\.md/i); - }); - }); - - describe("Static Role Verification Step", () => { - test("includes static role verification step in workflow", () => { - expect(skill.content).toContain("Static Role Verification"); - }); - - test("references role-verification.md", () => { - expect(skill.content).toContain("role-verification.md"); - }); - - test("mentions RBAC role assignments", () => { - expect(skill.content).toMatch(/rbac|role\s+assignment/i); - }); - - test("role verification comes before record proof", () => { - const roleVerifIndex = skill.content.indexOf("Static Role Verification"); - const recordProofIndex = skill.content.indexOf("Record Proof"); - expect(roleVerifIndex).toBeGreaterThan(-1); - expect(recordProofIndex).toBeGreaterThan(-1); - expect(roleVerifIndex).toBeLessThan(recordProofIndex); - }); - }); - - describe("Workflow Integration", () => { - test("documents recipe references", () => { - expect(skill.content).toMatch(/recipe/i); - }); - - test("mentions validation status requirement", () => { - expect(skill.content).toMatch(/validated/i); - }); - }); -}); diff --git a/tests/entra-agent-id/unit.test.ts b/tests/entra-agent-id/unit.test.ts deleted file mode 100644 index 29565f862..000000000 --- a/tests/entra-agent-id/unit.test.ts +++ /dev/null @@ -1,71 +0,0 @@ -/** - * Unit Tests for entra-agent-id - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "entra-agent-id"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is within recommended length", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("use for:"); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("do not use for:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Conceptual Model"); - expect(skill.content).toContain("## Core Workflow"); - expect(skill.content).toContain("## Runtime Authentication"); - expect(skill.content).toContain("## API Reference"); - expect(skill.content).toContain("## Troubleshooting"); - }); - - test("references the typed Graph endpoints (not raw @odata.type)", () => { - expect(skill.content).toContain("microsoft.graph.agentIdentityBlueprint"); - expect(skill.content).toContain("microsoft.graph.agentIdentityBlueprintPrincipal"); - expect(skill.content).toContain("microsoft.graph.agentIdentity"); - }); - - test("references the fmi_path exchange and the SDK sidecar", () => { - expect(skill.content).toContain("fmi_path"); - expect(skill.content).toContain("Microsoft Entra SDK for AgentID"); - }); - - test("references runtime-token-exchange reference file", () => { - expect(skill.content).toContain("references/runtime-token-exchange.md"); - }); - }); -}); diff --git a/tests/entra-app-registration/unit.test.ts b/tests/entra-app-registration/unit.test.ts deleted file mode 100644 index 5ea3ae7de..000000000 --- a/tests/entra-app-registration/unit.test.ts +++ /dev/null @@ -1,63 +0,0 @@ -/** - * Unit Tests for entra-app-registration - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "entra-app-registration"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is within recommended length", () => { - // Descriptions should be 150-1024 chars - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR trigger phrases", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("use for:"); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description.toLowerCase(); - expect(description).toContain("do not use for:"); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Overview"); - expect(skill.content).toContain("## Core Workflow"); - expect(skill.content).toContain("## Common Patterns"); - }); - - test("references MSAL documentation", () => { - expect(skill.content).toContain("MSAL"); - }); - - test("references OAuth flows", () => { - expect(skill.content).toContain("OAuth"); - }); - }); -}); diff --git a/tests/mcp-tool-references/unit.test.ts b/tests/mcp-tool-references/unit.test.ts deleted file mode 100644 index d8d1343dd..000000000 --- a/tests/mcp-tool-references/unit.test.ts +++ /dev/null @@ -1,142 +0,0 @@ -import { readdirSync, readFileSync, statSync } from "node:fs"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); -const snapshotPath = path.resolve(__dirname, "../fixtures/azure-mcp-tool-names.snapshot.json"); -const skillsRoot = path.resolve(__dirname, "../../output/skills"); - -// Two naming conventions for Azure MCP tool references in skill markdown: -// mcp_azure_mcp_ — GitHub Copilot MCP tool reference format -// azure__ — Alternative GitHub Copilot MCP tool reference format -const MCP_AZURE_MCP_RE = /mcp_azure_mcp_([a-z0-9_-]+)(?![a-z0-9_-])/gi; -const AZURE_DOUBLE_UNDERSCORE_RE = /azure__([a-z0-9_-]+)(?![a-z0-9_-])/gi; - -// No legacy Azure MCP tool aliases are currently allowed in skill markdown tests. -// Keep this map empty so invalid tool names fail validation instead of being silently normalized. -const TOOL_NAME_ALIASES: Record = {}; - -interface ToolReference { - toolName: string; - filePath: string; - line: number; - context: string; - pattern: string; -} - -interface AzureMcpSnapshot { - source?: string; - azureMcpVersion?: string; - toolNames: string[]; -} - -function readAndValidateSnapshot(filePath: string): AzureMcpSnapshot { - const snapshot = JSON.parse(readFileSync(filePath, "utf8")) as AzureMcpSnapshot; - if (!Array.isArray(snapshot.toolNames) || snapshot.toolNames.length === 0) { - throw new Error( - `Invalid Azure MCP tool snapshot at ${filePath}: expected a non-empty "toolNames" array`, - ); - } - - return snapshot; -} - -function collectMarkdownFiles(rootDir: string): string[] { - const markdownFiles: string[] = []; - - for (const entry of readdirSync(rootDir, { withFileTypes: true })) { - const fullPath = path.join(rootDir, entry.name); - - if (entry.isDirectory()) { - markdownFiles.push(...collectMarkdownFiles(fullPath)); - continue; - } - - if (entry.isFile() && entry.name.toLowerCase().endsWith(".md")) { - markdownFiles.push(fullPath); - } - } - - return markdownFiles; -} - -function extractToolReferences(filePath: string): ToolReference[] { - const lines = readFileSync(filePath, "utf8").split("\n"); - const refs: ToolReference[] = []; - - for (let i = 0; i < lines.length; i++) { - const lineText = lines[i]; - for (const [pattern, re] of [ - ["mcp_azure_mcp_", MCP_AZURE_MCP_RE], - ["azure__", AZURE_DOUBLE_UNDERSCORE_RE], - ] as [string, RegExp][]) { - re.lastIndex = 0; - let match: RegExpExecArray | null; - while ((match = re.exec(lineText)) !== null) { - refs.push({ - toolName: match[1].toLowerCase(), - filePath, - line: i + 1, - context: lineText.trim(), - pattern, - }); - } - } - } - - return refs; -} - -describe("Azure MCP tool references in skill markdown", () => { - let validToolNames: Set; - let snapshotToolNames: string[]; - let allReferences: ToolReference[]; - - beforeAll(() => { - const snapshot = readAndValidateSnapshot(snapshotPath); - snapshotToolNames = snapshot.toolNames; - validToolNames = new Set(snapshotToolNames); - allReferences = collectMarkdownFiles(skillsRoot).flatMap(extractToolReferences); - }); - - test("snapshot file exists and has expected shape", () => { - const raw = readAndValidateSnapshot(snapshotPath); - - expect(typeof raw.source).toBe("string"); - expect(typeof raw.azureMcpVersion).toBe("string"); - expect(Array.isArray(raw.toolNames)).toBe(true); - expect(raw.toolNames.length).toBeGreaterThan(0); - expect(statSync(snapshotPath).size).toBeGreaterThan(0); - }); - - test("all referenced mcp_azure_mcp_* and azure__* tool names exist in the snapshot", () => { - const unknownReferences: string[] = []; - - for (const ref of allReferences) { - const resolvedToolName = TOOL_NAME_ALIASES[ref.toolName] ?? ref.toolName; - - if (validToolNames.has(resolvedToolName)) { - continue; - } - - // Allow namespaced tool references like mcp_azure_mcp_storage_blob_list - // where "storage" is a valid tool prefix. - const hasValidNamespacePrefix = - /^[a-z0-9_]+$/.test(resolvedToolName) && - snapshotToolNames.some((snapshotToolName) => - resolvedToolName.startsWith(`${snapshotToolName}_`), - ); - if (hasValidNamespacePrefix) { - continue; - } - - const relativePath = path.relative(skillsRoot, ref.filePath); - unknownReferences.push( - `${ref.pattern}${ref.toolName} (${relativePath}:${ref.line})`, - ); - } - - expect(unknownReferences).toEqual([]); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/create/unit.test.ts b/tests/microsoft-foundry/foundry-agent/create/unit.test.ts deleted file mode 100644 index 9bb1a9db5..000000000 --- a/tests/microsoft-foundry/foundry-agent/create/unit.test.ts +++ /dev/null @@ -1,89 +0,0 @@ -/** - * Unit Tests for create - * - * Test isolated skill logic and validation rules. - * Tests load the parent microsoft-foundry skill and verify - * the create-hosted.md reference document content directly. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; - -describe("create - Unit Tests", () => { - let skill: LoadedSkill; - let createContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - createContent = fs.readFileSync( - path.join(skill.path, "foundry-agent", "create", "create-hosted.md"), - "utf-8" - ); - }); - - describe("Parent Skill References", () => { - test("parent skill references foundry-agent", () => { - expect(skill.content).toContain("foundry-agent"); - }); - }); - - describe("Create Reference Content", () => { - test("has substantive content", () => { - expect(createContent).toBeDefined(); - expect(createContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(createContent).toContain("## Quick Reference"); - expect(createContent).toContain("## When to Use This Skill"); - expect(createContent).toContain("## Workflow"); - }); - - test("documents sample download workflow", () => { - expect(createContent).toContain("microsoft-foundry/foundry-samples"); - expect(createContent).toContain("Step 4: Download Sample Files"); - }); - - test("supports multiple frameworks", () => { - expect(createContent).toContain("Agent Framework"); - expect(createContent).toContain("LangGraph"); - expect(createContent).toContain("Custom"); - }); - - test("supports multiple languages", () => { - expect(createContent).toContain("Python"); - expect(createContent).toContain("C#"); - }); - - test("documents both hosted-agent protocols", () => { - expect(createContent).toContain("responses"); - expect(createContent).toContain("invocations"); - expect(createContent).toContain("Protocol consistency"); - }); - - test("distinguishes browse root from the selected sample path", () => { - expect(createContent).toContain("sample_browse_path"); - expect(createContent).toContain("selected_sample_path"); - }); - - test("contains error handling section", () => { - expect(createContent).toContain("## Error Handling"); - }); - - test("documents greenfield vs brownfield", () => { - expect(createContent).toContain("Greenfield"); - expect(createContent).toContain("Brownfield"); - }); - - test("documents brownfield hosting adapter workflow", () => { - expect(createContent).toContain("Hosting Adapter"); - expect(createContent).toContain("azure-ai-agentserver"); - expect(createContent).toContain("agent.yaml"); - expect(createContent).toContain("Dockerfile"); - }); - - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/deploy/unit.test.ts b/tests/microsoft-foundry/foundry-agent/deploy/unit.test.ts deleted file mode 100644 index cdea395d5..000000000 --- a/tests/microsoft-foundry/foundry-agent/deploy/unit.test.ts +++ /dev/null @@ -1,209 +0,0 @@ -/** - * Unit Tests for deploy - * - * Test isolated skill logic and validation rules. - * Tests verify the deploy.md content including the - * post-deployment auto-create evaluators & dataset flow. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; - -describe("deploy - Unit Tests", () => { - let skill: LoadedSkill; - let deployContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - deployContent = fs.readFileSync( - path.join(skill.path, "foundry-agent", "deploy", "deploy.md"), - "utf-8" - ); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(2048); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Deploy Content Structure", () => { - test("has substantive content", () => { - expect(deployContent).toBeDefined(); - expect(deployContent.length).toBeGreaterThan(100); - }); - - test("contains Quick Reference section", () => { - expect(deployContent).toContain("## Quick Reference"); - expect(deployContent).toContain("| MCP server | `azure` |"); - }); - - test("contains When to Use section", () => { - expect(deployContent).toContain("## When to Use This Skill"); - }); - - test("documents MCP tools", () => { - expect(deployContent).toContain("## MCP Tools"); - expect(deployContent).toContain("agent_definition_schema_get"); - expect(deployContent).toContain("agent_update"); - expect(deployContent).toContain("agent_get"); - }); - - test("contains hosted agent workflow", () => { - expect(deployContent).toContain("## Workflow: Hosted Agent Deployment"); - }); - - test("documents the hosted deployment verification flow", () => { - expect(deployContent).toMatch(/Capture the per-agent identity from the agent creation response/i); - expect(deployContent).toMatch(/project-level agent identity from the project resource/i); - expect(deployContent).toMatch(/Continue to Step 8/i); - expect(deployContent).toMatch(/required hosted-agent session handling/i); - }); - - test("contains prompt agent workflow", () => { - expect(deployContent).toContain("## Workflow: Prompt Agent Deployment"); - }); - - test("contains error handling section", () => { - expect(deployContent).toContain("## Error Handling"); - }); - - test("lists invocations as a supported container protocol", () => { - expect(deployContent).toContain("`invocations`"); - expect(deployContent).toMatch(/Invocation payload protocol/i); - }); - }); - - describe("After Deployment — Auto-Create Evaluators", () => { - test("has auto-create evaluators section", () => { - expect(deployContent).toContain("Auto-Create Evaluators & Dataset"); - }); - - test("marks auto-create as automatic (not optional)", () => { - expect(deployContent).toMatch(/automatic|immediately/i); - }); - - test("instructs reading agent instructions via agent_get", () => { - expect(deployContent).toContain("agent_get"); - }); - - test("specifies default evaluator categories", () => { - expect(deployContent).toContain("evaluator_catalog_get"); - expect(deployContent).toMatch(/custom.*built-in|built-in.*custom/i); - expect(deployContent).toMatch(/name, category, and version/i); - expect(deployContent).toMatch(/<=5/i); - expect(deployContent).toContain("Quality"); - expect(deployContent).toContain("Safety"); - expect(deployContent).toContain("relevance"); - expect(deployContent).toContain("intent_resolution"); - expect(deployContent).toContain("task_adherence"); - expect(deployContent).toContain("indirect_attack"); - expect(deployContent).toContain("tool_call_accuracy"); - }); - - test("uses the observe skill's two-phase evaluator strategy", () => { - expect(deployContent).toContain("Two-Phase Evaluator Strategy"); - expect(deployContent).toMatch(/Phase 1 is built-in only/i); - expect(deployContent).toContain("expected_behavior"); - expect(deployContent).toMatch(/behavioral scoring/i); - }); - - test("instructs identifying judge deployment from actual project deployments", () => { - expect(deployContent).toContain("model_deployment_get"); - expect(deployContent).toMatch(/actual model deployments/i); - expect(deployContent).toMatch(/supports chat completions/i); - expect(deployContent).toMatch(/do\s+\*\*not\*\*\s+assume\s+`gpt-4o`\s+exists/i); - }); - - test("instructs persisting artifacts to .foundry/evaluators/ and .foundry/datasets/", () => { - expect(deployContent).toContain(".foundry/evaluators/"); - expect(deployContent).toContain(".foundry/datasets/"); - expect(deployContent).toContain("datasetUri"); - expect(deployContent).toMatch(/filename must start with the selected environment's Foundry agent name/i); - }); - - test("scopes deploy scanning and cache usage to the selected agent root", () => { - expect(deployContent).toMatch(/selected agent root/i); - expect(deployContent).toMatch(/Do \*\*not\*\* scan sibling agent folders/i); - }); - - test("uses the seed dataset guide as the canonical registration flow", () => { - expect(deployContent).toContain("Generate Seed Evaluation Dataset"); - expect(deployContent).toMatch(/single source of truth for seed dataset registration/i); - expect(deployContent).toContain("project_connection_list"); - expect(deployContent).toContain("AzureStorageAccount"); - expect(deployContent).toContain("evaluation_dataset_create"); - expect(deployContent).toContain("connectionName"); - expect(deployContent).toContain("-eval-seed"); - expect(deployContent).toContain("datasetUri"); - expect(deployContent).not.toContain("--account-key "); - expect(deployContent).not.toContain("--auth-mode login"); - }); - - test("describes seed generation rules without a separate validation pass", () => { - expect(deployContent).toMatch(/keep rows valid by construction/i); - expect(deployContent).not.toContain( - "Validation gates (JSON parsing, required fields, category coverage, minimum row count)" - ); - }); - - test("asks to RUN evaluation (not just set up)", () => { - expect(deployContent).toMatch( - /run an evaluation to identify optimization opportunities/i - ); - }); - - test("directs to observe skill Step 2 for evaluation", () => { - expect(deployContent).toContain("observe skill"); - expect(deployContent).toMatch(/Step 2.*Evaluate/i); - }); - - test("documents required invocation RBAC for hosted agents", () => { - expect(deployContent).toContain("Azure AI User"); - expect(deployContent).not.toContain("Cognitive Services OpenAI User"); - expect(deployContent).toMatch(/per-agent identity.*agent creation response/i); - expect(deployContent).toMatch(/project-level agent identity.*project resource/i); - expect(deployContent).not.toContain("Required identities:"); - expect(deployContent).toMatch(/Cognitive Services account, not the project/i); - }); - }); - - describe("Document Deployment Context", () => { - test("persists deployment context to the selected metadata file", () => { - expect(deployContent).toContain("projectEndpoint"); - expect(deployContent).toContain("agentName"); - expect(deployContent).toContain("azureContainerRegistry"); - expect(deployContent).toContain("evaluationSuites[]"); - expect(deployContent).toContain("datasetUri"); - expect(deployContent).toContain("tags"); - expect(deployContent).toContain("tier: smoke"); - expect(deployContent).toContain("selected metadata file"); - expect(deployContent).toContain("agent-metadata.prod.yaml"); - expect(deployContent).toContain("single-environment file"); - expect(deployContent).toContain("older `testSuites[]`"); - expect(deployContent).toContain("legacy `testCases[]`"); - expect(deployContent).toContain("rewrite that environment to `evaluationSuites[]`"); - }); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/eval-datasets/unit.test.ts b/tests/microsoft-foundry/foundry-agent/eval-datasets/unit.test.ts deleted file mode 100644 index fed1bdf90..000000000 --- a/tests/microsoft-foundry/foundry-agent/eval-datasets/unit.test.ts +++ /dev/null @@ -1,236 +0,0 @@ -/** - * Unit Tests for eval-datasets - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { fileURLToPath } from "url"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); -const DATASETS_MD = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/eval-datasets/eval-datasets.md" -); -const REFERENCES_PATH = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/eval-datasets/references" -); - -describe("eval-datasets - Unit Tests", () => { - let skill: LoadedSkill; - let datasetsContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - datasetsContent = fs.readFileSync(DATASETS_MD, "utf-8"); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - }); - - describe("Eval-Datasets Content Structure", () => { - test("has substantive content", () => { - expect(datasetsContent).toBeDefined(); - expect(datasetsContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(datasetsContent).toContain("## Quick Reference"); - expect(datasetsContent).toContain("| MCP server | `azure` |"); - expect(datasetsContent).toContain("## Before Starting"); - expect(datasetsContent).toContain("## The Foundry Flywheel"); - expect(datasetsContent).toContain("## Behavioral Rules"); - }); - - test("documents .foundry cache and metadata", () => { - expect(datasetsContent).toContain(".foundry/agent-metadata*.yaml"); - expect(datasetsContent).toContain("selected metadata file"); - expect(datasetsContent).toContain(".foundry/datasets/"); - expect(datasetsContent).toContain(".foundry/results/"); - }); - - test("documents agentName-based versioning and cache reuse", () => { - expect(datasetsContent).toContain("-eval-seed"); - expect(datasetsContent).toContain("-traces"); - expect(datasetsContent).toContain("-curated"); - expect(datasetsContent).toContain("-prod"); - expect(datasetsContent).toContain(".foundry/datasets/-traces-v.jsonl"); - expect(datasetsContent).toContain(".foundry/datasets/-curated-v.jsonl"); - expect(datasetsContent).toContain(".foundry/datasets/-prod-v.jsonl"); - expect(datasetsContent).toContain("Foundry dataset version"); - expect(datasetsContent).toContain("v"); - expect(datasetsContent).toMatch(/filenames? must start with the selected Foundry agent name/i); - expect(datasetsContent).toMatch(/do\s+\*\*not\*\*\s+append the environment key a second time/i); - expect(datasetsContent).toMatch(/cache|refresh/i); - expect(datasetsContent).toContain("evaluationSuites[]"); - }); - - test("scopes local dataset guidance to the selected agent root", () => { - const traceToDatasetContent = fs.readFileSync( - path.join(REFERENCES_PATH, "trace-to-dataset.md"), - "utf-8" - ); - - expect(datasetsContent).toMatch(/selected agent root/i); - expect(traceToDatasetContent).toMatch(/selected agent root's `.foundry\/datasets\/`, `.foundry\/results\/`, and metadata files/i); - expect(traceToDatasetContent).toMatch(/Do \*\*not\*\* merge sibling agent folders/i); - }); - - test("documents dataset metadata conventions and remote dataset tracking", () => { - expect(datasetsContent).toContain("agent"); - expect(datasetsContent).toContain("stage"); - expect(datasetsContent).toContain("version"); - expect(datasetsContent).toContain("datasetUri"); - expect(datasetsContent).toContain("AzureStorageAccount"); - expect(datasetsContent).toMatch(/evaluation_dataset_create.*does not expose a first-class `tags` parameter/i); - }); - - test("uses AzureStorageAccount connections and always includes connectionName when registering datasets", () => { - const traceToDatasetContent = fs.readFileSync( - path.join(REFERENCES_PATH, "trace-to-dataset.md"), - "utf-8" - ); - const seedGuideContent = fs.readFileSync( - path.join(REFERENCES_PATH, "generate-seed-dataset.md"), - "utf-8" - ); - - expect(traceToDatasetContent).toContain('category: "AzureStorageAccount"'); - expect(traceToDatasetContent).toContain("connectionName"); - expect(traceToDatasetContent).toContain("evaluation_dataset_create"); - expect(traceToDatasetContent).toMatch(/include it in this workflow so the dataset is bound/i); - expect(traceToDatasetContent).toContain("--container-name eval-datasets"); - expect(traceToDatasetContent).toContain("blob.core.windows.net/eval-datasets/"); - expect(traceToDatasetContent).not.toContain("--container-name datasets"); - expect(seedGuideContent).toMatch(/appending a new entry to the `datasets\[\]` list/i); - expect(seedGuideContent).toContain('"datasets": ['); - expect(seedGuideContent).toContain('"name": "-eval-seed"'); - expect(seedGuideContent).toContain("evaluationSuites:"); - expect(seedGuideContent).toContain("older `testSuites[]`"); - expect(seedGuideContent).toContain("tags:"); - expect(seedGuideContent).toContain("legacy `testCases[]`"); - expect(seedGuideContent).toContain("rewrite that environment to `evaluationSuites[]`"); - expect(seedGuideContent).toContain("--account-key "); - expect(seedGuideContent).toContain("--auth-mode login"); - }); - - test("keeps dataset names versionless and stores versions separately in metadata examples", () => { - const metadataContractContent = fs.readFileSync( - path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/references/agent-metadata-contract.md" - ), - "utf-8" - ); - const traceToDatasetContent = fs.readFileSync( - path.join(REFERENCES_PATH, "trace-to-dataset.md"), - "utf-8" - ); - const versioningContent = fs.readFileSync( - path.join(REFERENCES_PATH, "dataset-versioning.md"), - "utf-8" - ); - const lineageContent = fs.readFileSync( - path.join(REFERENCES_PATH, "eval-lineage.md"), - "utf-8" - ); - - expect(metadataContractContent).toContain("dataset: support-agent-dev-traces"); - expect(metadataContractContent).toContain("datasetVersion: v3"); - expect(metadataContractContent).toContain("dataset: support-agent-prod-curated"); - expect(metadataContractContent).toContain("datasetVersion: v2"); - expect(metadataContractContent).toContain("agent-metadata.prod.yaml"); - expect(metadataContractContent).toContain("selected metadata file"); - expect(metadataContractContent).toContain("evaluationSuites:"); - expect(metadataContractContent).toContain("tags:"); - expect(metadataContractContent).toContain("Legacy Compatibility (`testCases[]` / `testSuites[]` -> `evaluationSuites[]`)"); - expect(metadataContractContent).toContain("map `P0` -> `smoke`, `P1` -> `regression`, `P2` -> `coverage`"); - expect(metadataContractContent).not.toContain("priority: P0"); - expect(metadataContractContent).toMatch(/do not append the environment key again/i); - expect(traceToDatasetContent).toContain('datasetVersion: "v"'); - expect(traceToDatasetContent).toContain('"name": "support-bot-prod-traces"'); - expect(traceToDatasetContent).toContain('"version": "v3"'); - expect(versioningContent).toContain('"name": "support-bot-prod-traces"'); - expect(versioningContent).toContain('"version": "v1"'); - expect(versioningContent).toContain('"version": "v2"'); - expect(versioningContent).toContain('"version": "v3"'); - expect(versioningContent).toContain('"agent": "support-bot-prod"'); - expect(versioningContent).toContain('"stage": "traces"'); - expect(versioningContent).toContain('"datasetUri": ""'); - expect(versioningContent).toContain('"datasetUri": ""'); - expect(versioningContent).toContain('"datasetUri": ""'); - expect(versioningContent).not.toContain('"name": "support-bot-prod-traces-v1"'); - expect(versioningContent).not.toContain('"name": "support-bot-prod-traces-v2"'); - expect(versioningContent).not.toContain('"version": "1"'); - expect(versioningContent).not.toContain('"version": "2"'); - expect(lineageContent).toContain('"name": "support-bot-prod-traces"'); - expect(lineageContent).toContain('"version": "v3"'); - }); - - test("documents rewriting older metadata during dataset updates", () => { - expect(datasetsContent).toContain("older `testSuites[]`"); - expect(datasetsContent).toContain("legacy `testCases[]`"); - expect(datasetsContent).toContain("rewrite it as `evaluationSuites[]`"); - }); - - test("documents evalId versus evaluationId guidance", () => { - const comparisonContent = fs.readFileSync( - path.join(REFERENCES_PATH, "dataset-comparison.md"), - "utf-8" - ); - const trendingContent = fs.readFileSync( - path.join(REFERENCES_PATH, "eval-trending.md"), - "utf-8" - ); - - expect(datasetsContent).toContain("evaluationId"); - expect(datasetsContent).toContain("evalId"); - expect(comparisonContent).toMatch(/switch to `evalId`/i); - expect(trendingContent).toMatch(/evaluation_get expects `evalId`, not `evaluationId`/i); - }); - - test("documents eval group immutability for evaluator and threshold changes", () => { - const comparisonContent = fs.readFileSync( - path.join(REFERENCES_PATH, "dataset-comparison.md"), - "utf-8" - ); - const trendingContent = fs.readFileSync( - path.join(REFERENCES_PATH, "eval-trending.md"), - "utf-8" - ); - - expect(comparisonContent).toMatch(/create a new evaluation group/i); - expect(comparisonContent).toMatch(/thresholds/i); - expect(trendingContent).toMatch(/evaluator set and thresholds stayed fixed/i); - }); - }); - - describe("Reference Files Exist", () => { - const expectedFiles = [ - "trace-to-dataset.md", - "dataset-versioning.md", - "dataset-organization.md", - "dataset-curation.md", - "eval-trending.md", - "eval-regression.md", - "dataset-comparison.md", - "eval-lineage.md", - ]; - - test.each(expectedFiles)("has reference file: %s", (file) => { - const filePath = path.join(REFERENCES_PATH, file); - expect(fs.existsSync(filePath)).toBe(true); - }); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/invoke/unit.test.ts b/tests/microsoft-foundry/foundry-agent/invoke/unit.test.ts deleted file mode 100644 index 430735430..000000000 --- a/tests/microsoft-foundry/foundry-agent/invoke/unit.test.ts +++ /dev/null @@ -1,80 +0,0 @@ -/** - * Unit Tests for invoke - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; - -describe("invoke - Unit Tests", () => { - let skill: LoadedSkill; - let invokeContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - invokeContent = fs.readFileSync( - path.join(skill.path, "foundry-agent", "invoke", "invoke.md"), - "utf-8" - ); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(2048); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Invoke Reference Content", () => { - test("has substantive content", () => { - expect(invokeContent).toBeDefined(); - expect(invokeContent.length).toBeGreaterThan(100); - }); - - test("documents hosted session behavior", () => { - expect(invokeContent).toContain("sessionId"); - expect(invokeContent).toContain("session_create"); - expect(invokeContent).toContain("^[A-Za-z0-9_-]{8,128}$"); - }); - - test("does not reference removed Python fallback scripts", () => { - expect(invokeContent).not.toContain("scripts/invoke_agent_response.py"); - expect(invokeContent).not.toContain("scripts/invoke_agent_invocation.py"); - expect(invokeContent).not.toContain("scripts/requirements.txt"); - }); - - test("documents hosted agent readiness checks", () => { - expect(invokeContent).toMatch(/hosted agents/i); - expect(invokeContent).toMatch(/agent_get/i); - expect(invokeContent).toMatch(/active/i); - expect(invokeContent).not.toContain("Hosted Agent (ACA)"); - expect(invokeContent).not.toContain("Hosted Agent (vNext)"); - }); - - test("documents required hosted-agent invocation RBAC", () => { - expect(invokeContent).toMatch(/RBAC|permission/i); - expect(invokeContent).toMatch(/troubleshoot/i); - }); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/observe/unit.test.ts b/tests/microsoft-foundry/foundry-agent/observe/unit.test.ts deleted file mode 100644 index 9dfd1b826..000000000 --- a/tests/microsoft-foundry/foundry-agent/observe/unit.test.ts +++ /dev/null @@ -1,383 +0,0 @@ -/** - * Unit Tests for observe - * - * Test isolated skill logic and validation rules. - * Tests verify the observe.md content and reference files - * for the eval-driven optimization loop. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { fileURLToPath } from "url"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); -const OBSERVE_MD = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/observe/observe.md" -); -const REFERENCES_PATH = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/observe/references" -); - -describe("observe - Unit Tests", () => { - let skill: LoadedSkill; - let observeContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - observeContent = fs.readFileSync(OBSERVE_MD, "utf-8"); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Observe Content Structure", () => { - test("has substantive content", () => { - expect(observeContent).toBeDefined(); - expect(observeContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(observeContent).toContain("## Entry Points"); - expect(observeContent).toContain("## Before Starting"); - expect(observeContent).toContain("## Loop Overview"); - expect(observeContent).toContain("## Behavioral Rules"); - }); - - test("contains Quick Reference table", () => { - expect(observeContent).toContain("## Quick Reference"); - expect(observeContent).toContain("azure"); - }); - }); - - describe("Loop Overview", () => { - test("contains numbered loop steps", () => { - expect(observeContent).toContain("Auto-setup evaluators"); - expect(observeContent).toContain("Evaluate"); - expect(observeContent).toContain("cluster failures"); - expect(observeContent).toContain("Optimize prompt"); - expect(observeContent).toContain("Deploy new version"); - expect(observeContent).toContain("Re-evaluate"); - expect(observeContent).toContain("Compare versions"); - expect(observeContent).toContain("CI/CD"); - }); - }); - - describe("Entry Points — Post-Deploy Flow", () => { - test("has entry point for post-deploy auto-setup", () => { - expect(observeContent).toMatch(/Agent just deployed|Set up evaluation/i); - }); - - test("routes evaluate intent through auto-setup when cache is missing or stale", () => { - expect(observeContent).toMatch(/cache is missing|stale|refresh|check.*evaluators/i); - }); - - test("warns to check for existing evaluators before evaluation", () => { - expect(observeContent).toContain(".foundry/agent-metadata*.yaml"); - expect(observeContent).toContain("selected metadata file"); - expect(observeContent).toContain(".foundry/evaluators/"); - expect(observeContent).toContain(".foundry/datasets/"); - expect(observeContent).toMatch(/auto-setup|Auto-Setup/i); - }); - - test("scopes local observe context to the selected agent root", () => { - const setupContent = fs.readFileSync( - path.join(REFERENCES_PATH, "deploy-and-setup.md"), - "utf-8" - ); - - expect(observeContent).toMatch(/selected agent root/i); - expect(observeContent).toMatch(/Do \*\*not\*\* merge .*sibling agent folders/i); - expect(setupContent).toMatch(/selected agent root only/i); - expect(setupContent).toMatch(/Do \*\*not\*\* merge sibling agent folders/i); - }); - }); - - describe("Reference Files Exist", () => { - const expectedFiles = [ - "deploy-and-setup.md", - "evaluate-step.md", - "analyze-results.md", - "optimize-deploy.md", - "compare-iterate.md", - "cicd-monitoring.md", - ]; - - test.each(expectedFiles)("has reference file: %s", (file) => { - const filePath = path.join(REFERENCES_PATH, file); - expect(fs.existsSync(filePath)).toBe(true); - }); - }); - - describe("Deploy-and-Setup Reference", () => { - let setupContent: string; - - beforeAll(() => { - setupContent = fs.readFileSync( - path.join(REFERENCES_PATH, "deploy-and-setup.md"), - "utf-8" - ); - }); - - test("has auto-create evaluators as primary content", () => { - expect(setupContent).toContain("Auto-Create Evaluators & Dataset"); - }); - - test("marks auto-create as automatic", () => { - expect(setupContent).toMatch(/automatic|fully automatic/i); - }); - - test("includes evaluator selection with quality and safety categories", () => { - expect(setupContent).toContain("evaluator_catalog_get"); - expect(setupContent).toMatch(/custom.*built-in|built-in.*custom/i); - expect(setupContent).toMatch(/name, category, and version/i); - expect(setupContent).toMatch(/<=5/i); - expect(setupContent).toContain("Quality"); - expect(setupContent).toContain("Safety"); - expect(setupContent).toContain("relevance"); - expect(setupContent).toContain("intent_resolution"); - expect(setupContent).toContain("task_adherence"); - expect(setupContent).toContain("indirect_attack"); - expect(setupContent).toContain("tool_call_accuracy"); - }); - - test("references the built-in-first two-phase evaluator strategy", () => { - expect(setupContent).toContain("Two-Phase Evaluator Strategy"); - expect(setupContent).toMatch(/Phase 1 is built-in only/i); - expect(setupContent).toMatch(/do not create a new custom evaluator during the initial setup pass/i); - expect(setupContent).toContain("expected_behavior"); - expect(setupContent).toContain("behavioral rubric"); - }); - - test("includes judge deployment step based on actual project deployments", () => { - expect(setupContent).toContain("model_deployment_get"); - expect(setupContent).toMatch(/actual model deployments/i); - expect(setupContent).toMatch(/supports chat completions/i); - expect(setupContent).toMatch(/do\s+\*\*not\*\*\s+assume\s+`gpt-4o`\s+exists/i); - }); - - test("generates seed datasets directly instead of invoking the judge deployment", () => { - expect(setupContent).toMatch(/Generate the seed rows directly/i); - expect(setupContent).toMatch(/Do \*\*not\*\* call the identified chat-capable deployment/i); - }); - - test("includes artifact persistence structure", () => { - expect(setupContent).toContain("agent-metadata.yaml"); - expect(setupContent).toContain("agent-metadata.prod.yaml"); - expect(setupContent).toContain("selected metadata file"); - expect(setupContent).toContain(".foundry/evaluators/"); - expect(setupContent).toContain(".foundry/datasets/"); - expect(setupContent).toContain("datasetUri"); - expect(setupContent).toContain(".yaml"); - expect(setupContent).toContain(".jsonl"); - expect(setupContent).toMatch(/filename must start with the selected environment's Foundry agent name/i); - }); - - test("uses evaluationSuites metadata with tags as the primary schema", () => { - expect(observeContent).toContain("evaluationSuites[]"); - expect(setupContent).toMatch(/evaluation suites/i); - expect(setupContent).toContain("tags"); - expect(setupContent).toContain("tier: smoke"); - }); - - test("documents testSuites/testCases compatibility and migration on write", () => { - expect(observeContent).toContain("older `testSuites[]`"); - expect(observeContent).toContain("legacy `testCases[]`"); - expect(observeContent).toContain("rewrite that environment to `evaluationSuites[]`"); - expect(setupContent).toContain("older `testSuites[]`"); - expect(setupContent).toContain("legacy `testCases[]`"); - expect(setupContent).toContain("replace that list with `evaluationSuites[]`"); - expect(setupContent).toContain("map `priority` to `tags.tier`"); - }); - - test("uses the seed dataset guide as the canonical registration flow", () => { - expect(setupContent).toContain("Generate Seed Evaluation Dataset"); - expect(setupContent).toMatch(/single source of truth for registration/i); - expect(setupContent).toContain("project_connection_list"); - expect(setupContent).toContain("AzureStorageAccount"); - expect(setupContent).toContain("evaluation_dataset_create"); - expect(setupContent).toContain("connectionName"); - expect(setupContent).toContain("-eval-seed"); - expect(setupContent).toContain("datasetUri"); - expect(setupContent).not.toContain("--account-key "); - expect(setupContent).not.toContain("--auth-mode login"); - }); - - test("prompts user to run evaluation after auto-setup", () => { - expect(setupContent).toMatch( - /run an evaluation to identify optimization opportunities/i - ); - }); - - test("redirects to deploy skill for deployment (not inline)", () => { - expect(setupContent).toContain("deploy skill"); - expect(setupContent).toContain("deploy.md"); - }); - }); - - describe("Behavioral Rules", () => { - test("requires auto-poll in background", () => { - expect(observeContent).toMatch(/auto-poll|background/i); - }); - - test("requires confirmation before changes", () => { - expect(observeContent).toMatch(/confirm|sign-off/i); - }); - - test("requires prompting for next steps", () => { - expect(observeContent).toMatch(/prompt.*next|next.*steps/i); - }); - - test("requires scripts in files not inline", () => { - expect(observeContent).toContain("scripts/"); - }); - - test("requires persisting eval artifacts", () => { - expect(observeContent).toContain(".foundry/evaluators/"); - expect(observeContent).toContain(".foundry/datasets/"); - expect(observeContent).toContain(".foundry/results/"); - expect(observeContent).toContain("tier=smoke"); - }); - - test("documents evalId versus evaluationId guardrail", () => { - const evaluateContent = fs.readFileSync( - path.join(REFERENCES_PATH, "evaluate-step.md"), - "utf-8" - ); - const compareContent = fs.readFileSync( - path.join(REFERENCES_PATH, "compare-iterate.md"), - "utf-8" - ); - - expect(evaluateContent).toContain("evaluationId"); - expect(evaluateContent).toContain("evalId"); - expect(evaluateContent).toContain("expected_behavior"); - expect(evaluateContent).toMatch(/evaluation_get.*does\s+\*\*not\*\*\s+accept\s+`evaluationId`/i); - expect(compareContent).toMatch(/creation uses `evaluationId`.*`evaluation_get`.*`evalId`/i); - }); - - test("requires judge deployment lookup instead of assuming gpt-4o", () => { - const evaluateContent = fs.readFileSync( - path.join(REFERENCES_PATH, "evaluate-step.md"), - "utf-8" - ); - - expect(evaluateContent).toContain("model_deployment_get"); - expect(evaluateContent).toMatch(/supports chat completions/i); - expect(evaluateContent).toMatch(/do\s+\*\*not\*\*\s+assume\s+`gpt-4o`\s+exists/i); - }); - - test("requires checking existing evaluators before creating new ones", () => { - expect(observeContent).toContain("evaluator_catalog_get"); - expect(observeContent).toMatch(/existing evaluators before creating new ones/i); - expect(observeContent).toMatch(/initial setup, re-evaluation, and optimization loops/i); - }); - - test("documents the two-phase evaluator strategy and expected_behavior usage", () => { - expect(observeContent).toContain("## Two-Phase Evaluator Strategy"); - expect(observeContent).toMatch(/Phase 1 - Initial setup/i); - expect(observeContent).toMatch(/Phase 2 - After analysis/i); - expect(observeContent).toContain("expected_behavior"); - expect(observeContent).toContain("behavioral_adherence"); - expect(observeContent).toMatch(/per-query behavioral rubric/i); - }); - - test("requires custom evaluator prompts to use result/reason output contract", () => { - const setupContent = fs.readFileSync( - path.join(REFERENCES_PATH, "deploy-and-setup.md"), - "utf-8" - ); - - expect(observeContent).toMatch(/custom evaluator output contract/i); - expect(observeContent).toContain("`result` plus `reason`"); - expect(observeContent).toMatch(/score.*reasoning/i); - expect(observeContent).toMatch(/duplicate `OUTPUT FORMAT` blocks/i); - expect(setupContent).toMatch(/runtime-enforced JSON fields are `result` and `reason`/i); - expect(setupContent).toMatch(/omit conflicting output JSON schemas/i); - }); - - test("documents LLM judge knowledge-cutoff mitigation for real-time data", () => { - const analyzeContent = fs.readFileSync( - path.join(REFERENCES_PATH, "analyze-results.md"), - "utf-8" - ); - - expect(observeContent).toMatch(/LLM judge knowledge cutoff/i); - expect(observeContent).toMatch(/web search, Bing Grounding, live APIs/i); - expect(observeContent).toMatch(/fabricated|beyond knowledge cutoff/i); - expect(analyzeContent).toMatch(/LLM judge knowledge cutoff/i); - expect(analyzeContent).toMatch(/cannot verify|beyond knowledge cutoff|no evidence/i); - expect(analyzeContent).toContain("Behavioral Rule 13"); - }); - - test("documents evaluator deletion parameter requirements", () => { - expect(observeContent).toContain("evaluator_catalog_delete"); - expect(observeContent).toMatch(/`name` \(not `evaluatorName`\) and `version`/i); - expect(observeContent).toMatch(/delete each version individually/i); - expect(observeContent).toMatch(/Discover version numbers with `evaluator_catalog_get`/i); - }); - - test("documents Data Viewer deeplinks for dataset and result files", () => { - expect(observeContent).toMatch(/Data Viewer deeplinks/i); - expect(observeContent).toContain("vscode://ms-windows-ai-studio.windows-ai-studio/open_data_viewer"); - expect(observeContent).toContain(".foundry/datasets/"); - expect(observeContent).toContain(".foundry/results/"); - }); - - test("documents eval group immutability for evaluators and thresholds", () => { - const evaluateContent = fs.readFileSync( - path.join(REFERENCES_PATH, "evaluate-step.md"), - "utf-8" - ); - const compareContent = fs.readFileSync( - path.join(REFERENCES_PATH, "compare-iterate.md"), - "utf-8" - ); - - expect(evaluateContent).toMatch(/new evaluation group/i); - expect(evaluateContent).toMatch(/thresholds/i); - expect(compareContent).toMatch(/reuse the same `evaluationId` only when `evaluatorNames` and thresholds are unchanged/i); - }); - - test("documents downloading detailed results via Azure AI Projects Python SDK", () => { - const analyzeContent = fs.readFileSync( - path.join(REFERENCES_PATH, "analyze-results.md"), - "utf-8" - ); - - expect(analyzeContent).toContain("AIProjectClient"); - expect(analyzeContent).toContain("get_openai_client()"); - expect(analyzeContent).toMatch(/evals\.runs\.output_items\.list/); - expect(analyzeContent).toContain("datasource_item.query"); - expect(analyzeContent).toContain("sample.output_text"); - expect(analyzeContent).toContain("custom_score"); - expect(analyzeContent).toContain("extract_evaluator_result"); - expect(analyzeContent).not.toContain("/openai/evals/{eval_id}/runs/{run_id}/output_items"); - }); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/trace/unit.test.ts b/tests/microsoft-foundry/foundry-agent/trace/unit.test.ts deleted file mode 100644 index ae7a695d7..000000000 --- a/tests/microsoft-foundry/foundry-agent/trace/unit.test.ts +++ /dev/null @@ -1,100 +0,0 @@ -/** - * Unit Tests for trace - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { fileURLToPath } from "url"; -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); -const TRACE_MD = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/trace/trace.md" -); -const REFERENCES_PATH = path.resolve( - __dirname, - "../../../../output/skills/microsoft-foundry/foundry-agent/trace/references" -); - -describe("trace - Unit Tests", () => { - let skill: LoadedSkill; - let traceContent: string; - let kqlTemplatesContent: string; - let searchTracesContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - traceContent = fs.readFileSync(TRACE_MD, "utf-8"); - kqlTemplatesContent = fs.readFileSync( - path.join(REFERENCES_PATH, "kql-templates.md"), - "utf-8" - ); - searchTracesContent = fs.readFileSync( - path.join(REFERENCES_PATH, "search-traces.md"), - "utf-8" - ); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Hosted Agent Identity Guidance", () => { - test("keeps hosted-agent filtering on requests-scoped name fields", () => { - expect(traceContent).toMatch(/Resolve hosted-agent identity from `requests` first/i); - expect(kqlTemplatesContent).toMatch(/Use `requests` as the hosted-agent entry point/i); - expect(searchTracesContent).toMatch(/Use `requests` as the preferred entry point/i); - }); - - test("documents parsing agent name and version from gen_ai.agent.id when available", () => { - expect(traceContent).toContain("gen_ai.agent.id"); - expect(kqlTemplatesContent).toContain("agentNameFromId"); - expect(kqlTemplatesContent).toContain("agentVersion"); - expect(kqlTemplatesContent).toMatch(/:/i); - expect(searchTracesContent).toContain("agentVersion"); - expect(searchTracesContent).toContain('split(agentId, ":")'); - }); - }); - - describe("Span Discovery Guidance", () => { - test("documents operation_Id-based span expansion for hosted agents", () => { - expect(traceContent).toMatch(/Use `operation_Id` to fan out hosted-agent traces/i); - expect(kqlTemplatesContent).toMatch(/Operation_Id Join \(requests → dependencies\)/i); - expect(kqlTemplatesContent).toContain("project operation_Id, conversationId, agentVersion"); - expect(kqlTemplatesContent).toContain("join kind=inner agentRequests on operation_Id"); - expect(searchTracesContent).toContain("join kind=inner agentRequests on operation_Id"); - expect(searchTracesContent).toContain("coalesce("); - expect(searchTracesContent).toContain("operation_Id, agentVersion"); - }); - - test("surfaces agent version in hosted-agent conversation summaries", () => { - expect(searchTracesContent).toContain("| Conversation ID | Agent Version |"); - expect(searchTracesContent).toContain("| conv_abc123 | 3 |"); - }); - }); -}); diff --git a/tests/microsoft-foundry/foundry-agent/troubleshoot/unit.test.ts b/tests/microsoft-foundry/foundry-agent/troubleshoot/unit.test.ts deleted file mode 100644 index 923c16d3a..000000000 --- a/tests/microsoft-foundry/foundry-agent/troubleshoot/unit.test.ts +++ /dev/null @@ -1,41 +0,0 @@ -/** - * Unit Tests for troubleshoot - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; - -describe("troubleshoot - Unit Tests", () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe("microsoft-foundry"); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(2048); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); -}); diff --git a/tests/microsoft-foundry/models/deploy/capacity/unit.test.ts b/tests/microsoft-foundry/models/deploy/capacity/unit.test.ts deleted file mode 100644 index 177225ea6..000000000 --- a/tests/microsoft-foundry/models/deploy/capacity/unit.test.ts +++ /dev/null @@ -1,92 +0,0 @@ -/** - * Unit Tests for capacity discovery - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import matter from "gray-matter"; -import { loadSkill, LoadedSkill } from "../../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const NESTED_FILE = "models/deploy-model/capacity/SKILL.md"; - -interface NestedSkillMetadata { - name: string; - description: string; - [key: string]: unknown; -} - -describe("capacity - Unit Tests", () => { - let skill: LoadedSkill; - let nestedMetadata: NestedSkillMetadata; - let nestedContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const nestedFilePath = path.join(skill.path, NESTED_FILE); - const fileContent = fs.readFileSync(nestedFilePath, "utf-8"); - const { data: metadata, content } = matter(fileContent); - - nestedMetadata = { - name: (metadata.name as string) || "", - description: (metadata.description as string) || "", - ...metadata - }; - nestedContent = content.trim(); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(nestedMetadata).toBeDefined(); - expect(nestedMetadata.name).toBe("capacity"); - expect(nestedMetadata.description).toBeDefined(); - expect(nestedMetadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(nestedMetadata.description.length).toBeGreaterThan(150); - expect(nestedMetadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - expect(nestedMetadata.description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - expect(nestedMetadata.description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(nestedContent).toBeDefined(); - expect(nestedContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(nestedContent).toContain("## Quick Reference"); - expect(nestedContent).toContain("## When to Use This Skill"); - expect(nestedContent).toContain("## Workflow"); - }); - - test("documents discovery scripts", () => { - expect(nestedContent).toContain("discover_and_rank"); - expect(nestedContent).toContain("query_capacity"); - }); - - test("contains error handling section", () => { - expect(nestedContent).toContain("## Error Handling"); - }); - - test("references hand-off to preset and customize", () => { - expect(nestedContent).toContain("preset"); - expect(nestedContent).toContain("customize"); - }); - - test("is read-only — does not deploy", () => { - expect(nestedContent).toContain("does NOT deploy"); - }); - }); -}); diff --git a/tests/microsoft-foundry/models/deploy/customize-deployment/unit.test.ts b/tests/microsoft-foundry/models/deploy/customize-deployment/unit.test.ts deleted file mode 100644 index f62d65103..000000000 --- a/tests/microsoft-foundry/models/deploy/customize-deployment/unit.test.ts +++ /dev/null @@ -1,88 +0,0 @@ -/** - * Unit Tests for customize (customize-deployment) - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import matter from "gray-matter"; -import { loadSkill, LoadedSkill } from "../../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const NESTED_FILE = "models/deploy-model/customize/SKILL.md"; - -interface NestedSkillMetadata { - name: string; - description: string; - [key: string]: unknown; -} - -describe("customize (customize-deployment) - Unit Tests", () => { - let skill: LoadedSkill; - let nestedMetadata: NestedSkillMetadata; - let nestedContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const nestedFilePath = path.join(skill.path, NESTED_FILE); - const fileContent = fs.readFileSync(nestedFilePath, "utf-8"); - const { data: metadata, content } = matter(fileContent); - - nestedMetadata = { - name: (metadata.name as string) || "", - description: (metadata.description as string) || "", - ...metadata - }; - nestedContent = content.trim(); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(nestedMetadata).toBeDefined(); - expect(nestedMetadata.name).toBe("customize"); - expect(nestedMetadata.description).toBeDefined(); - expect(nestedMetadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(nestedMetadata.description.length).toBeGreaterThan(150); - expect(nestedMetadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - expect(nestedMetadata.description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - expect(nestedMetadata.description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(nestedContent).toBeDefined(); - expect(nestedContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(nestedContent).toContain("## Quick Reference"); - expect(nestedContent).toContain("## Prerequisites"); - }); - - test("documents customization options", () => { - expect(nestedContent).toContain("SKU"); - expect(nestedContent).toContain("capacity"); - expect(nestedContent).toContain("RAI"); - }); - - test("documents PTU deployment support", () => { - expect(nestedContent).toContain("PTU"); - expect(nestedContent).toContain("ProvisionedManaged"); - }); - - test("contains comparison with preset mode", () => { - expect(nestedContent).toContain("## When to Use"); - }); - }); -}); diff --git a/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/unit.test.ts b/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/unit.test.ts deleted file mode 100644 index ccaf6947a..000000000 --- a/tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/unit.test.ts +++ /dev/null @@ -1,48 +0,0 @@ -/** - * Unit Tests for preset (deploy-model-optimal-region) - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import { loadSkill, LoadedSkill } from "../../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const NESTED_FILE = "models/deploy-model/preset/references/preset-workflow.md"; - -describe("preset (deploy-model-optimal-region) - Unit Tests", () => { - let skill: LoadedSkill; - let referenceContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const nestedFilePath = path.join(skill.path, NESTED_FILE); - referenceContent = fs.readFileSync(nestedFilePath, "utf-8"); - }); - - describe("Reference Content", () => { - test("has substantive content", () => { - expect(referenceContent).toBeDefined(); - expect(referenceContent.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(referenceContent).toContain("## Phase 1"); - expect(referenceContent).toContain("## Phase 2"); - }); - - test("contains deployment phases", () => { - expect(referenceContent).toContain("## Phase 1"); - expect(referenceContent).toContain("## Phase 2"); - }); - - test("contains Azure CLI commands", () => { - expect(referenceContent).toContain("az cognitiveservices"); - }); - - test("documents GlobalStandard SKU usage", () => { - expect(referenceContent).toContain("GlobalStandard"); - }); - }); -}); diff --git a/tests/microsoft-foundry/models/deploy/deploy-model/unit.test.ts b/tests/microsoft-foundry/models/deploy/deploy-model/unit.test.ts deleted file mode 100644 index 245308721..000000000 --- a/tests/microsoft-foundry/models/deploy/deploy-model/unit.test.ts +++ /dev/null @@ -1,105 +0,0 @@ -/** - * Unit Tests for deploy-model (router) - * - * Test isolated skill logic and validation rules. - */ - -import * as fs from "fs"; -import * as path from "path"; -import matter from "gray-matter"; -import { loadSkill, LoadedSkill } from "../../../../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; -const NESTED_FILE = "models/deploy-model/SKILL.md"; - -interface NestedSkillMetadata { - name: string; - description: string; - [key: string]: unknown; -} - -describe("deploy-model (router) - Unit Tests", () => { - let skill: LoadedSkill; - let nestedMetadata: NestedSkillMetadata; - let nestedContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const nestedFilePath = path.join(skill.path, NESTED_FILE); - const fileContent = fs.readFileSync(nestedFilePath, "utf-8"); - const { data: metadata, content } = matter(fileContent); - - nestedMetadata = { - name: (metadata.name as string) || "", - description: (metadata.description as string) || "", - ...metadata - }; - nestedContent = content.trim(); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(nestedMetadata).toBeDefined(); - expect(nestedMetadata.name).toBe("deploy-model"); - expect(nestedMetadata.description).toBeDefined(); - expect(nestedMetadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - expect(nestedMetadata.description.length).toBeGreaterThan(150); - expect(nestedMetadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - expect(nestedMetadata.description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - expect(nestedMetadata.description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(nestedContent).toBeDefined(); - expect(nestedContent.length).toBeGreaterThan(100); - }); - - test("contains routing sections", () => { - expect(nestedContent).toContain("## Quick Reference"); - expect(nestedContent).toContain("## Intent Detection"); - expect(nestedContent).toContain("### Routing Rules"); - }); - - test("contains sub-skill references", () => { - expect(nestedContent).toContain("preset/SKILL.md"); - expect(nestedContent).toContain("customize/SKILL.md"); - expect(nestedContent).toContain("capacity/SKILL.md"); - }); - - test("documents all three deployment modes", () => { - expect(nestedContent).toContain("Preset"); - expect(nestedContent).toContain("Customize"); - expect(nestedContent).toContain("Capacity"); - }); - - test("contains project selection guidance", () => { - expect(nestedContent).toContain("## Project Selection"); - expect(nestedContent).toContain("PROJECT_RESOURCE_ID"); - }); - - test("contains multi-mode chaining documentation", () => { - expect(nestedContent).toContain("### Multi-Mode Chaining"); - }); - }); - - describe("Prerequisites", () => { - test("lists Azure CLI requirement", () => { - expect(nestedContent).toContain("Azure CLI"); - }); - - test("lists subscription requirement", () => { - expect(nestedContent).toContain("Azure subscription"); - }); - }); -}); diff --git a/tests/microsoft-foundry/resource/create/unit.test.ts b/tests/microsoft-foundry/resource/create/unit.test.ts deleted file mode 100644 index 9063f2d7b..000000000 --- a/tests/microsoft-foundry/resource/create/unit.test.ts +++ /dev/null @@ -1,216 +0,0 @@ -/** - * Unit Tests for microsoft-foundry:resource/create - * - * Test isolated skill logic and validation for the resource/create sub-skill. - * Following progressive disclosure best practices from the skills development guide. - */ - -import { loadSkill, LoadedSkill } from "../../../utils/skill-loader"; -import * as fs from "fs/promises"; -import * as path from "path"; - -const SKILL_NAME = "microsoft-foundry"; - -describe("microsoft-foundry:resource/create - Unit Tests", () => { - let skill: LoadedSkill; - let resourceCreateContent: string; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - const resourceCreatePath = path.join( - SKILLS_PATH, - "microsoft-foundry/resource/create/create-foundry-resource.md" - ); - resourceCreateContent = await fs.readFile(resourceCreatePath, "utf-8"); - }); - - describe("Parent Skill Integration", () => { - test("parent skill references resource/create sub-skill", () => { - expect(skill.content).toContain("resource/create"); - expect(skill.content).toContain("create-foundry-resource.md"); - }); - - test("parent skill description includes resource creation triggers", () => { - const description = skill.metadata.description; - expect(description).toContain("USE FOR:"); - expect(description).toMatch(/create Foundry resource|create AI Services|multi-service resource/i); - }); - - test("resource/create is in sub-skills table", () => { - expect(skill.content).toContain("## Sub-Skills"); - expect(skill.content).toMatch(/\*\*resource\/create\*\*/i); - }); - }); - - describe("Skill Metadata", () => { - test("has valid frontmatter with required fields", () => { - expect(resourceCreateContent).toMatch(/^---\r?\n/); - expect(resourceCreateContent).toContain("name: microsoft-foundry:resource/create"); - expect(resourceCreateContent).toContain("description:"); - }); - - test("description includes USE FOR and DO NOT USE FOR", () => { - expect(resourceCreateContent).toContain("USE FOR:"); - expect(resourceCreateContent).toContain("DO NOT USE FOR:"); - }); - - test("description mentions key triggers", () => { - expect(resourceCreateContent).toMatch(/create Foundry resource|create AI Services|multi-service resource|AIServices kind/i); - }); - }); - - describe("Skill Content - References Pattern", () => { - test("main file is condensed with links to references", () => { - expect(resourceCreateContent).toBeDefined(); - const lineCount = resourceCreateContent.split("\n").length; - // Main file should be under 200 lines for token optimization - expect(lineCount).toBeLessThan(200); - // Should link to reference files - expect(resourceCreateContent).toContain("./references/workflows.md"); - expect(resourceCreateContent).toContain("./references/patterns.md"); - expect(resourceCreateContent).toContain("./references/troubleshooting.md"); - }); - - test("contains Quick Reference table", () => { - expect(resourceCreateContent).toContain("## Quick Reference"); - expect(resourceCreateContent).toContain("Classification"); - expect(resourceCreateContent).toContain("WORKFLOW SKILL"); - expect(resourceCreateContent).toContain("Control Plane"); - }); - - test("specifies correct resource type", () => { - expect(resourceCreateContent).toContain("Microsoft.CognitiveServices/accounts"); - expect(resourceCreateContent).toContain("AIServices"); - }); - - test("contains When to Use section", () => { - expect(resourceCreateContent).toContain("## When to Use"); - expect(resourceCreateContent).toContain("Create Foundry resource"); - }); - - test("contains Prerequisites section", () => { - expect(resourceCreateContent).toContain("## Prerequisites"); - expect(resourceCreateContent).toContain("Azure subscription"); - expect(resourceCreateContent).toContain("Azure CLI"); - expect(resourceCreateContent).toContain("RBAC roles"); - }); - - test("references RBAC skill for permissions", () => { - expect(resourceCreateContent).toContain("microsoft-foundry:rbac"); - }); - }); - - describe("Core Workflows", () => { - test("contains all 3 required workflows", () => { - expect(resourceCreateContent).toContain("## Core Workflows"); - expect(resourceCreateContent).toContain("### 1. Create Resource Group"); - expect(resourceCreateContent).toContain("### 2. Create Foundry Resource"); - expect(resourceCreateContent).toContain("### 3. Register Resource Provider"); - }); - - test("each workflow has command patterns", () => { - expect(resourceCreateContent).toContain("Create a resource group"); - expect(resourceCreateContent).toContain("Create a new Azure AI Services resource"); - expect(resourceCreateContent).toContain("Register Cognitive Services provider"); - }); - - test("workflows use Azure CLI commands", () => { - expect(resourceCreateContent).toContain("az cognitiveservices account create"); - expect(resourceCreateContent).toContain("az group create"); - expect(resourceCreateContent).toContain("az provider register"); - }); - - test("workflows have condensed steps with link to detailed content", () => { - // Main file has condensed steps, detailed content in references - expect(resourceCreateContent).toContain("#### Steps"); - expect(resourceCreateContent).toContain("See [Detailed Workflow Steps](./references/workflows.md)"); - }); - }); - - describe("Important Notes Section", () => { - test("explains resource kind requirement", () => { - expect(resourceCreateContent).toContain("## Important Notes"); - expect(resourceCreateContent).toContain("AIServices"); - }); - - test("explains SKU selection", () => { - expect(resourceCreateContent).toContain("## Important Notes"); - expect(resourceCreateContent).toContain("S0"); - }); - - test("mentions key requirements", () => { - expect(resourceCreateContent).toContain("## Important Notes"); - expect(resourceCreateContent).toMatch(/location|region/i); - }); - }); - - describe("Quick Commands Section", () => { - test("includes commonly used commands in workflows", () => { - expect(resourceCreateContent).toContain("az cognitiveservices account create"); - expect(resourceCreateContent).toContain("az group create"); - }); - - test("commands include proper parameters", () => { - expect(resourceCreateContent).toMatch(/--kind AIServices/); - expect(resourceCreateContent).toMatch(/--resource-group/); - expect(resourceCreateContent).toMatch(/--name/); - }); - - test("includes verification commands", () => { - expect(resourceCreateContent).toContain("az cognitiveservices account show"); - }); - - test("links to patterns reference with additional commands", () => { - expect(resourceCreateContent).toContain("./references/patterns.md"); - }); - }); - - describe("Troubleshooting Section", () => { - test("links to troubleshooting reference", () => { - expect(resourceCreateContent).toContain("./references/troubleshooting.md"); - }); - - test("mentions RBAC skill for permission issues", () => { - expect(resourceCreateContent).toMatch(/microsoft-foundry:rbac/); - }); - }); - - describe("External Resources", () => { - test("links to Microsoft documentation", () => { - expect(resourceCreateContent).toContain("## Additional Resources"); - expect(resourceCreateContent).toContain("learn.microsoft.com"); - }); - - test("includes relevant Azure docs", () => { - expect(resourceCreateContent).toMatch(/multi-service resource|Azure AI Services/i); - }); - }); - - describe("Best Practices Compliance", () => { - test("prioritizes Azure CLI for control plane operations", () => { - expect(resourceCreateContent).toContain("Primary Method"); - expect(resourceCreateContent).toContain("Azure CLI"); - expect(resourceCreateContent).toContain("Control Plane"); - }); - - test("follows skill = how, tools = what pattern", () => { - expect(resourceCreateContent).toContain("orchestrates"); - expect(resourceCreateContent).toContain("WORKFLOW SKILL"); - }); - - test("provides routing clarity", () => { - expect(resourceCreateContent).toContain("When to Use"); - expect(resourceCreateContent).toContain("Do NOT use for"); - }); - - test("follows references pattern for token optimization", () => { - // Should have condensed content with links to references - expect(resourceCreateContent).toContain("./references/workflows.md"); - expect(resourceCreateContent).toContain("./references/patterns.md"); - expect(resourceCreateContent).toContain("./references/troubleshooting.md"); - // Main file should be under 200 lines for token limit compliance - const lineCount = resourceCreateContent.split("\n").length; - expect(lineCount).toBeLessThan(200); - }); - }); -}); diff --git a/tests/microsoft-foundry/unit.test.ts b/tests/microsoft-foundry/unit.test.ts deleted file mode 100644 index f3b0ae89e..000000000 --- a/tests/microsoft-foundry/unit.test.ts +++ /dev/null @@ -1,343 +0,0 @@ -/** - * Unit Tests for microsoft-foundry - * - * Test isolated skill logic and validation rules. - */ - -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "microsoft-foundry"; - -describe(`${SKILL_NAME} - Unit Tests`, () => { - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - }); - - describe("Skill Metadata", () => { - test("has valid SKILL.md with required fields", () => { - expect(skill.metadata).toBeDefined(); - expect(skill.metadata.name).toBe(SKILL_NAME); - expect(skill.metadata.description).toBeDefined(); - expect(skill.metadata.description.length).toBeGreaterThan(10); - }); - - test("description is appropriately sized", () => { - // Descriptions should be 150-1024 chars for Medium-High compliance - expect(skill.metadata.description.length).toBeGreaterThan(150); - expect(skill.metadata.description.length).toBeLessThan(1024); - }); - - test("description contains USE FOR triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/USE FOR:/i); - }); - - test("description contains DO NOT USE FOR anti-triggers", () => { - const description = skill.metadata.description; - expect(description).toMatch(/DO NOT USE FOR:/i); - }); - }); - - describe("Skill Content", () => { - test("has substantive content", () => { - expect(skill.content).toBeDefined(); - expect(skill.content.length).toBeGreaterThan(100); - }); - - test("contains expected sections", () => { - expect(skill.content).toContain("## Agent Development Lifecycle"); - expect(skill.content).toContain("## Sub-Skills"); - expect(skill.content).toContain("## Agent: Project Context Resolution"); - expect(skill.content).toContain("## Agent: Agent Types"); - }); - - test("contains agent routing references", () => { - expect(skill.content).toContain("deploy"); - expect(skill.content).toContain("invoke"); - expect(skill.content).toContain("observe"); - expect(skill.content).toContain("troubleshoot"); - }); - - test("description includes prompt optimization routing keywords", () => { - const description = skill.metadata.description; - expect(description).toContain("improve prompt"); - expect(description).toContain("prompt optimizer"); - expect(description).toContain("improve agent instructions"); - expect(description).toContain("optimize system prompt"); - }); - - test("contains common project context resolution", () => { - expect(skill.content).toContain("azure.yaml"); - expect(skill.content).toContain("azd env get-values"); - }); - - test("documents azd variable mapping", () => { - expect(skill.content).toContain("AZURE_AI_PROJECT_ENDPOINT"); - expect(skill.content).toContain("AZURE_CONTAINER_REGISTRY_NAME"); - }); - - test("documents .foundry workspace standard and metadata sidecars", () => { - expect(skill.content).toContain(".foundry/agent-metadata.yaml"); - expect(skill.content).toContain("agent-metadata.prod.yaml"); - expect(skill.content).toContain("selected metadata file"); - expect(skill.content).toContain("defaultEnvironment"); - expect(skill.content).toContain("Agent Metadata Contract"); - }); - - test("documents testSuites/testCases migration to evaluationSuites", () => { - expect(skill.content).toContain("older `testSuites[]`"); - expect(skill.content).toContain("legacy `testCases[]`"); - expect(skill.content).toContain("normalize each entry in memory"); - expect(skill.content).toContain("persist only `evaluationSuites[]`"); - }); - - }); - - describe("Sub-Skills Reference", () => { - test("has Sub-Skills table", () => { - expect(skill.content).toContain("## Sub-Skills"); - }); - - test("references agent sub-skills in table", () => { - expect(skill.content).toContain("foundry-agent/deploy/deploy.md"); - expect(skill.content).toContain("foundry-agent/invoke/invoke.md"); - expect(skill.content).toContain("foundry-agent/observe/observe.md"); - expect(skill.content).toContain("foundry-agent/troubleshoot/troubleshoot.md"); - }); - - test("observe sub-skill row routes prompt optimization scenarios", () => { - expect(skill.content).toMatch(/observe.*optimize prompts/i); - expect(skill.content).toMatch(/observe.*improve agent instructions/i); - expect(skill.content).toMatch(/observe.*CI\/CD monitoring/i); - }); - - test("references quota sub-skill", () => { - expect(skill.content).toContain("quota"); - expect(skill.content).toContain("quota/quota.md"); - }); - - test("references rbac sub-skill", () => { - expect(skill.content).toContain("rbac"); - expect(skill.content).toContain("rbac/rbac.md"); - }); - }); - - describe("Quota Sub-Skill Content", () => { - let quotaContent: string; - - beforeAll(async () => { - const fs = await import("fs/promises"); - const path = await import("path"); - const quotaPath = path.join( - SKILLS_PATH, - "microsoft-foundry/quota/quota.md" - ); - quotaContent = await fs.readFile(quotaPath, "utf-8"); - }); - - test("has quota reference file", () => { - expect(quotaContent).toBeDefined(); - expect(quotaContent.length).toBeGreaterThan(100); - }); - - test("contains quota management workflows", () => { - expect(quotaContent).toContain("### 1. Check Regional Quota"); - expect(quotaContent).toContain("### 2. Find Best Region for Deployment"); - expect(quotaContent).toContain("### 3. Check Quota Before Deployment"); - expect(quotaContent).toContain("### 5. Delete Deployment (Free Quota)"); - }); - - test("contains command patterns for each workflow", () => { - expect(quotaContent).toContain("View quota usage"); - expect(quotaContent).toContain("Request quota increase"); - }); - - test("contains az cognitiveservices commands", () => { - expect(quotaContent).toContain("az rest"); - expect(quotaContent).toContain("az cognitiveservices account deployment"); - }); - - test("contains error troubleshooting", () => { - expect(quotaContent).toContain("QuotaExceeded"); - expect(quotaContent).toContain("InsufficientQuota"); - expect(quotaContent).toContain("DeploymentLimitReached"); - }); - - test("includes quota management guidance", () => { - expect(quotaContent).toContain("## Core Workflows"); - }); - - test("contains bash command examples", () => { - expect(quotaContent).toContain("```bash"); - expect(quotaContent).toContain("az rest"); - }); - }); - - describe("Agent Development Lifecycle Routing", () => { - test("routes prompt optimization intents to observe", () => { - expect(skill.content).toContain( - "Optimize / improve agent prompt or instructions" - ); - expect(skill.content).toContain("observe (Step 4: Optimize)"); - expect(skill.content).toContain("Evaluate and optimize agent (full loop)"); - }); - - test("mentions prompt_optimize at the top level", () => { - expect(skill.content).toContain("prompt_optimize"); - expect(skill.content).toMatch(/Prompt Optimization:/i); - }); - }); - - describe("RBAC Sub-Skill Content", () => { - let rbacContent: string; - - beforeAll(async () => { - const fs = await import("fs/promises"); - const path = await import("path"); - const rbacPath = path.join( - SKILLS_PATH, - "microsoft-foundry/rbac/rbac.md" - ); - rbacContent = await fs.readFile(rbacPath, "utf-8"); - }); - - test("has RBAC reference file", () => { - expect(rbacContent).toBeDefined(); - expect(rbacContent.length).toBeGreaterThan(100); - }); - - test("contains Azure AI Foundry roles table", () => { - expect(rbacContent).toContain("Azure AI User"); - expect(rbacContent).toContain("Azure AI Project Manager"); - expect(rbacContent).toContain("Azure AI Account Owner"); - expect(rbacContent).toContain("Azure AI Owner"); - }); - - test("contains roles capability matrix", () => { - expect(rbacContent).toContain("Create Projects"); - expect(rbacContent).toContain("Data Actions"); - expect(rbacContent).toContain("Role Assignments"); - }); - - test("contains Portal vs SDK/CLI warning", () => { - expect(rbacContent).toMatch(/portal.*but.*not.*sdk|cli/i); - }); - - test("contains all 6 RBAC workflows", () => { - expect(rbacContent).toContain("### 1. Assign User Permissions"); - expect(rbacContent).toContain("### 2. Assign Developer Permissions"); - expect(rbacContent).toContain("### 3. Audit Role Assignments"); - expect(rbacContent).toContain("### 4. Validate Permissions"); - expect(rbacContent).toContain("### 5. Configure Managed Identity Roles"); - expect(rbacContent).toContain("### 6. Create Service Principal"); - }); - - test("contains az role assignment commands", () => { - expect(rbacContent).toContain("az role assignment create"); - expect(rbacContent).toContain("az role assignment list"); - }); - - test("contains az ad sp commands for service principal", () => { - expect(rbacContent).toContain("az ad sp create-for-rbac"); - }); - - test("contains managed identity roles for connected resources", () => { - expect(rbacContent).toContain("Storage Blob Data Reader"); - expect(rbacContent).toContain("Storage Blob Data Contributor"); - expect(rbacContent).toContain("Key Vault Secrets User"); - expect(rbacContent).toContain("Search Index Data Reader"); - expect(rbacContent).toContain("Search Index Data Contributor"); - }); - - test("uses correct Foundry resource type", () => { - expect(rbacContent).toContain("Microsoft.CognitiveServices/accounts"); - }); - - test("contains permission requirements table", () => { - expect(rbacContent).toContain("Permission Requirements by Action"); - expect(rbacContent).toContain("Deploy models"); - expect(rbacContent).toContain("Create projects"); - }); - - test("contains error handling section", () => { - expect(rbacContent).toContain("Error Handling"); - expect(rbacContent).toContain("Authorization failed"); - }); - - test("contains bash command examples", () => { - expect(rbacContent).toContain("```bash"); - }); - }); - - describe("Private Network Sub-Skill References", () => { - let intakeContent: string; - let templateIndexContent: string; - let scaffoldContent: string; - let adaptationContent: string; - - beforeAll(async () => { - const fs = await import("fs/promises"); - const path = await import("path"); - const refBase = path.join( - SKILLS_PATH, - "microsoft-foundry/resource/private-network/references" - ); - intakeContent = await fs.readFile(path.join(refBase, "intake.md"), "utf-8"); - templateIndexContent = await fs.readFile(path.join(refBase, "template-index.md"), "utf-8"); - scaffoldContent = await fs.readFile(path.join(refBase, "scaffold.md"), "utf-8"); - adaptationContent = await fs.readFile(path.join(refBase, "custom-template-adaptation.md"), "utf-8"); - }); - - test("intake.md exists and has tiered structure", () => { - expect(intakeContent).toContain("Tier 1"); - expect(intakeContent).toContain("Tier 2"); - expect(intakeContent).toContain("Tier 3"); - }); - - test("intake.md contains approach determination", () => { - expect(intakeContent).toContain("OFFICIAL"); - expect(intakeContent).toContain("ADAPT"); - expect(intakeContent).toContain("EXTEND"); - }); - - test("intake.md contains Learn validation section", () => { - expect(intakeContent).toContain("Validate Against Learn"); - expect(intakeContent).toContain("microsoft_docs_fetch"); - }); - - test("template-index.md contains GitHub repo URLs", () => { - expect(templateIndexContent).toContain("github.com/microsoft-foundry/foundry-samples"); - expect(templateIndexContent).toContain("infrastructure-setup-bicep"); - expect(templateIndexContent).toContain("infrastructure-setup-terraform"); - }); - - test("template-index.md instructs to fetch directory listing", () => { - expect(templateIndexContent).toContain("directory listing"); - }); - - test("scaffold.md references all three paths", () => { - expect(scaffoldContent).toContain("OFFICIAL"); - expect(scaffoldContent).toContain("ADAPT"); - expect(scaffoldContent).toContain("EXTEND"); - }); - - test("scaffold.md links to template-index and custom-template-adaptation", () => { - expect(scaffoldContent).toContain("template-index.md"); - expect(scaffoldContent).toContain("custom-template-adaptation.md"); - }); - - test("custom-template-adaptation.md contains gap analysis instructions", () => { - expect(adaptationContent).toContain("Read"); - expect(adaptationContent).toContain("Analyze"); - expect(adaptationContent).toContain("Present"); - expect(adaptationContent).toContain("Wait"); - }); - - test("custom-template-adaptation.md contains retry safety warning", () => { - expect(adaptationContent).toContain("legionservicelink"); - }); - }); - -}); diff --git a/tests/package.json b/tests/package.json index b8c8d5884..b4809baba 100644 --- a/tests/package.json +++ b/tests/package.json @@ -13,10 +13,8 @@ "test:ci": "node scripts/run-tests.js ci", "test:watch": "node scripts/run-tests.js watch", "test:skill": "node scripts/run-tests.js skill", - "coverage:grid": "node scripts/generate-coverage-grid.js", "report": "npx tsx scripts/generate-test-reports.ts", "results": "node scripts/show-test-results.js", - "quality-report": "node scripts/generate-quality-report.js", "update:mcp-tool-snapshot": "node scripts/update-mcp-tool-snapshot.js", "update:snapshots": "node scripts/update-snapshots.js", "typecheck": "tsc --noEmit", diff --git a/tests/scripts/generate-quality-report.js b/tests/scripts/generate-quality-report.js deleted file mode 100644 index 6586bd777..000000000 --- a/tests/scripts/generate-quality-report.js +++ /dev/null @@ -1,1013 +0,0 @@ -#!/usr/bin/env node - -/** - * Quality Report Generator - * - * Post-processes raw test outputs (JUnit XML, token-usage.json, agent-metadata) - * into a single skill-quality-report.json contract file. - * - * This is the "Layer 2" processor — sits between raw pipeline outputs and the - * reporting dashboard. The dashboard reads only this JSON, doing zero processing. - * - * Usage: - * node generate-quality-report.js # Process most recent test run - * node generate-quality-report.js --run # Process a specific test run - * node generate-quality-report.js --junit # Use a specific JUnit XML - */ - -import fs from "fs"; -import path from "path"; -import { fileURLToPath } from "url"; - -const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); - -const REPORTS_PATH = path.resolve(__dirname, "../reports"); -const TEST_RUN_PREFIX = "test-run-"; -const CONTRACT_VERSION = "1.0"; - -// ─── CLI Argument Parsing ──────────────────────────────────────────────────── - -function parseArgs(argv) { - const args = argv.slice(2); - let runDir = null; - let junitPath = null; - - for (let i = 0; i < args.length; i++) { - if (args[i] === "--run" && i + 1 < args.length) runDir = args[++i]; - if (args[i] === "--junit" && i + 1 < args.length) junitPath = args[++i]; - } - - return { runDir, junitPath }; -} - -// ─── Find Most Recent Test Run ─────────────────────────────────────────────── - -function getMostRecentTestRun() { - if (!fs.existsSync(REPORTS_PATH)) return null; - const entries = fs.readdirSync(REPORTS_PATH, { withFileTypes: true }); - const testRuns = entries - .filter(e => e.isDirectory() && e.name.startsWith(TEST_RUN_PREFIX)) - .map(e => e.name) - .sort() - .reverse(); - return testRuns.length > 0 ? testRuns[0] : null; -} - -// ─── JUnit XML Parser (regex-based, matches existing show-test-results.js) ── - -function extractAttr(tag, name) { - const match = tag.match(new RegExp(`${name}="([^"]*)"`)); - return match ? match[1] : null; -} - -function parseJunitXml(xmlPath) { - if (!fs.existsSync(xmlPath)) return null; - const xml = fs.readFileSync(xmlPath, "utf-8"); - - const result = { - totalTests: 0, - failures: 0, - errors: 0, - time: 0, - suites: [], - }; - - const headerMatch = xml.match(/]*>/); - if (headerMatch) { - const attrs = headerMatch[0]; - result.totalTests = parseInt(extractAttr(attrs, "tests") || "0", 10); - result.failures = parseInt(extractAttr(attrs, "failures") || "0", 10); - result.errors = parseInt(extractAttr(attrs, "errors") || "0", 10); - result.time = parseFloat(extractAttr(attrs, "time") || "0"); - } - - const suiteRegex = /]*>[\s\S]*?<\/testsuite>/g; - let suiteMatch; - while ((suiteMatch = suiteRegex.exec(xml)) !== null) { - const suiteXml = suiteMatch[0]; - const suiteAttrsMatch = suiteXml.match(/]*>/); - if (!suiteAttrsMatch) continue; - - const sa = suiteAttrsMatch[0]; - const suite = { - name: extractAttr(sa, "name") || "Unknown", - tests: parseInt(extractAttr(sa, "tests") || "0", 10), - failures: parseInt(extractAttr(sa, "failures") || "0", 10), - time: parseFloat(extractAttr(sa, "time") || "0"), - testcases: [], - }; - - const tcRegex = /]*>[\s\S]*?<\/testcase>|/g; - let tcMatch; - while ((tcMatch = tcRegex.exec(suiteXml)) !== null) { - const tcXml = tcMatch[0]; - const tcAttrs = tcXml.match(/]*>/)?.[0]; - if (!tcAttrs) continue; - - const tc = { - classname: extractAttr(tcAttrs, "classname") || "", - name: extractAttr(tcAttrs, "name") || "Unknown", - time: parseFloat(extractAttr(tcAttrs, "time") || "0"), - status: "passed", - failure: null, - }; - - const failMatch = tcXml.match(/]*>([\s\S]*?)<\/failure>/); - if (failMatch) { - tc.status = "failed"; - tc.failure = failMatch[1].trim().substring(0, 500); - } - const errMatch = tcXml.match(/]*>([\s\S]*?)<\/error>/); - if (errMatch) { - tc.status = "error"; - tc.failure = errMatch[1].trim().substring(0, 500); - } - if (tcXml.includes(" { - try { return JSON.parse(line); } catch { return null; } - }).filter(Boolean); - } catch { - // Fall back to legacy JSON format - } - const summaryPath = path.join(testRunPath, "token-summary.json"); - try { - const raw = fs.readFileSync(summaryPath, "utf-8"); - if (!raw.trim()) return []; - return JSON.parse(raw); - } catch (err) { - console.warn( - `Warning: Failed to load token summary at ${testRunPath}: ${ - err && err.message ? err.message : err - }` - ); - return []; - } -} - -// ─── Skill Area Extraction ─────────────────────────────────────────────────── - -/** - * Extract skill area from test directory name or JUnit classname. - * Handles both formats: - * Directory: "microsoft-foundry_deploy-model_-_Integration_Tests_..." → "microsoft-foundry/deploy-model" - * JUnit: "microsoft-foundry_deploy-model - Integration Tests ›..." → "microsoft-foundry/deploy-model" - * JUnit: "microsoft-foundry_ - Integration Tests" → "microsoft-foundry" - */ -function extractSkillArea(name) { - // Try JUnit classname format first: "skill_sub - Integration Tests..." - const match = name.match(/^(.+?)\s*-\s*Integration Tests/); - if (match) { - let skillPart = match[1].trim(); - // "microsoft-foundry_deploy-model" → "microsoft-foundry/deploy-model" - // "microsoft-foundry_" → "microsoft-foundry" - skillPart = skillPart.replace(/_$/, ""); // trailing underscore - return skillPart.replace(/_/g, "/"); - } - - // Try directory name format: "skill_sub_-_Integration_Tests_..." - const integrationIdx = name.indexOf("_-_Integration_Tests"); - if (integrationIdx !== -1) { - const skillPart = name.substring(0, integrationIdx); - return skillPart.replace(/_/g, "/"); - } - - // Fallback - const parts = name.split("_"); - return parts[0]; -} - -// ─── Tool Call Extraction ───────────────────────────────────────────────────── - -/** - * Extract tool calls from structured agent-metadata.json. - */ -function extractToolCalls(testRunPath, dirName) { - const dirPath = path.join(testRunPath, dirName); - if (!fs.existsSync(dirPath)) return []; - - const jsonPath = path.join(dirPath, "agent-metadata.json"); - if (!fs.existsSync(jsonPath)) return []; - - try { - const data = JSON.parse(fs.readFileSync(jsonPath, "utf-8")); - const toolCalls = []; - - for (const event of (data.events || [])) { - // SDK events use tool.execution_start with data.toolName - if (event.type === "tool.execution_start") { - const toolName = event.data?.toolName || ""; - if (toolName === "skill") { - const args = event.data?.arguments; - const skillName = typeof args === "string" ? args : JSON.stringify(args || ""); - toolCalls.push({ tool: "skill", args: skillName, source: "agent-metadata.json" }); - } else { - const args = event.data?.arguments || ""; - let argsStr = ""; - if (typeof args === "string") { - try { - const parsed = JSON.parse(args); - argsStr = Object.entries(parsed).map(([k,v]) => `${k}: ${typeof v === "string" ? v.substring(0, 80) : v}`).join(", "); - } catch { argsStr = args.substring(0, 100); } - } else if (typeof args === "object") { - argsStr = Object.entries(args).map(([k,v]) => `${k}: ${typeof v === "string" ? v.substring(0, 80) : v}`).join(", "); - } - toolCalls.push({ tool: toolName, args: argsStr, source: "agent-metadata.json" }); - } - } else if (event.type === "assistant.reasoning") { - const reasoning = event.data?.content || event.data?.text || ""; - const lastCall = toolCalls[toolCalls.length - 1]; - if (lastCall && reasoning) { - lastCall.reasoning = reasoning.substring(0, 500); - } - } - } - - return toolCalls; - } catch (err) { - console.warn(`Warning: Failed to parse ${jsonPath}: ${err.message}`); - return []; - } -} - -// ─── Build Area Summaries ──────────────────────────────────────────────────── - -function buildAreaSummaries(junit, tokenEntries) { - const areaMap = new Map(); - - // Process JUnit test cases - if (junit) { - for (const suite of junit.suites) { - for (const tc of suite.testcases) { - const area = extractSkillArea(tc.classname.split(" › ")[0] || suite.name); - if (!areaMap.has(area)) { - areaMap.set(area, { - name: area, - tests: 0, - passed: 0, - failed: 0, - skipped: 0, - totalInputTokens: 0, - totalOutputTokens: 0, - totalLLMCalls: 0, - totalDurationMs: 0, - testDetails: [], - }); - } - const entry = areaMap.get(area); - entry.tests++; - if (tc.status === "passed") entry.passed++; - else if (tc.status === "failed" || tc.status === "error") entry.failed++; - else if (tc.status === "skipped") entry.skipped++; - - entry.testDetails.push({ - name: tc.name, - status: tc.status, - time: tc.time, - failure: tc.failure, - }); - } - } - } - - // Merge token data - for (const tokenEntry of tokenEntries) { - const area = extractSkillArea(tokenEntry.testName); - if (!areaMap.has(area)) { - areaMap.set(area, { - name: area, - tests: 0, - passed: 0, - failed: 0, - skipped: 0, - totalInputTokens: 0, - totalOutputTokens: 0, - totalLLMCalls: 0, - totalDurationMs: 0, - testDetails: [], - }); - } - const entry = areaMap.get(area); - entry.totalInputTokens += tokenEntry.inputTokens || 0; - entry.totalOutputTokens += tokenEntry.outputTokens || 0; - entry.totalLLMCalls += tokenEntry.apiCallCount || 0; - entry.totalDurationMs += tokenEntry.totalApiDurationMs || 0; - } - - // Compute derived metrics - const areas = []; - for (const [, entry] of areaMap) { - const totalTests = entry.tests || 1; - areas.push({ - name: entry.name, - tests: entry.tests, - passed: entry.passed, - failed: entry.failed, - skipped: entry.skipped, - passRate: totalTests > 0 ? Math.round((entry.passed / totalTests) * 1000) / 10 : null, - avgInputTokens: Math.round(entry.totalInputTokens / totalTests), - avgOutputTokens: Math.round(entry.totalOutputTokens / totalTests), - totalInputTokens: entry.totalInputTokens, - totalOutputTokens: entry.totalOutputTokens, - avgLLMCalls: Math.round((entry.totalLLMCalls / totalTests) * 10) / 10, - totalLLMCalls: entry.totalLLMCalls, - avgDurationMs: Math.round(entry.totalDurationMs / totalTests), - testDetails: entry.testDetails, - }); - } - - return areas.sort((a, b) => a.passRate - b.passRate); -} - -// ─── Build Token Usage Per Test ────────────────────────────────────────────── - -function buildTokenUsage(tokenEntries) { - return tokenEntries.map(entry => ({ - testName: entry.testName, - prompt: entry.prompt, - timestamp: entry.timestamp, - model: entry.model, - inputTokens: entry.inputTokens, - outputTokens: entry.outputTokens, - cacheReadTokens: entry.cacheReadTokens || 0, - cacheWriteTokens: entry.cacheWriteTokens || 0, - llmCalls: entry.apiCallCount, - durationMs: entry.totalApiDurationMs, - perCall: (entry.perCallUsage || []).map((call, i) => ({ - call: i + 1, - model: call.model, - inputTokens: call.inputTokens, - outputTokens: call.outputTokens, - durationMs: call.durationMs, - initiator: call.initiator, - })), - })); -} - -// ─── Expected Path Definitions ─────────────────────────────────────────────── - -/** - * Expected execution paths per skill area. - * Each entry defines the ideal sequence of tool calls the agent should make. - * Used for side-by-side comparison in the trace viewer. - */ -const EXPECTED_PATHS = { - // Derived from passing test traces — each path reflects the common - // tool-call sequence observed across successful test executions. - "microsoft-foundry/deploy-model": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog deployment intent" }, - { tool: "view", label: "📖 Read skill doc\ndeploy-model SKILL.md" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nList/check resources" }, - { tool: "powershell", label: "💻 powershell\naz cognitiveservices deploy" }, - ], - endLabel: "⬜ EXPECTED END\nModel deployed", - }, - "microsoft-foundry/quota": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog quota check" }, - { tool: "view", label: "📖 Read skill doc\nquota SKILL.md" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nQuery Foundry resources" }, - { tool: "azure-quota", label: "📊 azure-quota\nRetrieve quota data" }, - ], - endLabel: "⬜ EXPECTED END\nQuota info returned", - }, - "microsoft-foundry/resource/create": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog resource creation" }, - { tool: "view", label: "📖 Read skill doc\nresource/create SKILL.md" }, - { tool: "powershell", label: "💻 powershell\naz group create" }, - { tool: "powershell", label: "💻 powershell\naz cognitiveservices create" }, - ], - endLabel: "⬜ EXPECTED END\nResource created", - }, - "microsoft-foundry/capacity": { - steps: [ - { tool: "report_intent", label: "📋 report_intent\nLog capacity check" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nQuery Foundry resources" }, - { tool: "azure-quota", label: "📊 azure-quota\nCheck capacity/quota" }, - { tool: "azure-extension_cli_generate", label: "🔧 cli_generate\nGenerate CLI command" }, - ], - endLabel: "⬜ EXPECTED END\nCapacity report shown", - }, - "microsoft-foundry/customize-deployment": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog customization" }, - { tool: "view", label: "📖 Read skill doc\ncustomize-deployment SKILL.md" }, - { tool: "view", label: "📖 Read references\nDeployment config docs" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nApply custom settings" }, - ], - endLabel: "⬜ EXPECTED END\nDeployment customized", - }, - "microsoft-foundry/deploy-model-optimal-region": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog region search" }, - { tool: "view", label: "📖 Read skill doc\noptimal-region SKILL.md" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nCheck region availability" }, - { tool: "powershell", label: "💻 powershell\nDeploy to optimal region" }, - { tool: "azure-quota", label: "📊 azure-quota\nVerify region capacity" }, - ], - endLabel: "⬜ EXPECTED END\nDeployed to best region", - }, - "microsoft-foundry/create": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog agent creation" }, - { tool: "view", label: "📖 Read skill doc\ncreate agent SKILL.md" }, - { tool: "powershell", label: "💻 powershell\nSetup agent resources" }, - ], - endLabel: "⬜ EXPECTED END\nAgent created", - }, - "microsoft-foundry/foundry-agent": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog agent task" }, - { tool: "view", label: "📖 Read skill doc\nfoundry-agent SKILL.md" }, - { tool: "azure-foundry", label: "🏗️ azure-foundry\nManage agent resources" }, - { tool: "powershell", label: "💻 powershell\nExecute agent operation" }, - ], - endLabel: "⬜ EXPECTED END\nAgent task complete", - }, - "microsoft-foundry/observe": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog agent observe" }, - { tool: "view", label: "📖 Read skill doc\nobserve SKILL.md" }, - { tool: "powershell", label: "💻 powershell\nQuery agent metrics" }, - ], - endLabel: "⬜ EXPECTED END\nAgent observability shown", - }, - "microsoft-foundry": { - steps: [ - { tool: "skill", args: "microsoft-foundry", label: "🔧 skill:\nmicrosoft-foundry" }, - { tool: "report_intent", label: "📋 report_intent\nLog skill routing" }, - { tool: "view", label: "📖 Read skill doc\nmicrosoft-foundry SKILL.md" }, - { tool: "view", label: "📖 Read sub-skill doc\nSub-skill reference" }, - { tool: "powershell", label: "💻 powershell\nExecute Azure operation" }, - ], - endLabel: "⬜ EXPECTED END\nOperation complete", - }, -}; - -/** - * Build expected path nodes and edges for a skill area. - * Returns { nodes: [], edges: [] } with Cytoscape-ready data. - */ -function buildExpectedPath(skillArea) { - const expectedDef = EXPECTED_PATHS[skillArea]; - if (!expectedDef) return null; - - const nodes = []; - const edges = []; - let prevId = "start"; // share start node with actual - - for (let i = 0; i < expectedDef.steps.length; i++) { - const step = expectedDef.steps[i]; - const llmId = `el${i + 1}`; - const stepId = `e${i + 1}`; - - // LLM call node - nodes.push({ - id: llmId, - label: `🤖 LLM Call #${i + 1}\nAnalyze & route`, - type: "expected_llm", - detail: { desc: `Expected: LLM decides to call ${step.tool}` }, - }); - edges.push({ source: prevId, target: llmId, type: "expected" }); - - // Tool call node - nodes.push({ - id: stepId, - label: step.label, - type: "expected", - step: i + 1, - detail: { tool: step.tool, args: step.args || "", desc: step.label.replace(/\n/g, " ") }, - }); - edges.push({ source: llmId, target: stepId, type: "expected" }); - - prevId = stepId; - } - - // End expected - const endLlmId = `el${expectedDef.steps.length + 1}`; - nodes.push({ - id: endLlmId, - label: `🤖 LLM Call #${expectedDef.steps.length + 1}\nGenerate response`, - type: "expected_llm", - detail: { desc: "Expected: LLM generates final response to user" }, - }); - edges.push({ source: prevId, target: endLlmId, type: "expected" }); - - nodes.push({ - id: "end_exp", - label: expectedDef.endLabel, - type: "endOk", - shape: "ellipse", - detail: { desc: "Expected outcome" }, - }); - edges.push({ source: endLlmId, target: "end_exp", type: "expected" }); - - return { nodes, edges, stepCount: expectedDef.steps.length }; -} - -// ─── Build Execution Traces ────────────────────────────────────────────────── - -/** - * Build execution trace graphs for the dashboard trace viewer. - * - * For each test that has agent-metadata output (from agent-runner), this function: - * 1. Extracts the ordered sequence of tool/skill calls from agent-metadata.json. - * 2. Pairs each call with token usage data from the per-call breakdown. - * 3. Builds a Cytoscape-compatible node/edge graph: Start → LLM Call → Tool → ... → End. - * 4. Overlays the expected execution path (if defined) and computes path adherence - * metrics (matched/deviated/extra/skipped steps and adherence percentage). - * - * @param {string} testRunPath - Path to the test-run reports directory. - * @param {Array} tokenEntries - Token usage entries from token-summary.jsonl. - * @returns {Object} Map of testName → { prompt, model, summary, nodes, edges, - * expectedNodes?, expectedEdges?, pathAdherence? } - */ -function buildTraces(testRunPath, tokenEntries) { - const traces = {}; - - // Group token entries by testName - const tokenByTest = new Map(); - for (const entry of tokenEntries) { - if (!tokenByTest.has(entry.testName)) { - tokenByTest.set(entry.testName, []); - } - tokenByTest.get(entry.testName).push(entry); - } - - for (const [testName, entries] of tokenByTest) { - // Use the last entry (most recent run) for this test - const latestEntry = entries[entries.length - 1]; - const toolCalls = extractToolCalls(testRunPath, testName); - - // Build Cytoscape-ready nodes - const nodes = []; - const edges = []; - - // Start node - nodes.push({ - id: "start", - label: `🟢 START\n${(latestEntry.prompt || "").substring(0, 40)}`, - type: "start", - shape: "ellipse", - detail: { prompt: latestEntry.prompt }, - }); - - // Build actual path nodes from perCallUsage + tool calls - const perCall = latestEntry.perCallUsage || []; - let prevNodeId = "start"; - - for (let i = 0; i < perCall.length; i++) { - const call = perCall[i]; - const llmNodeId = `llm${i + 1}`; - const toolCall = toolCalls[i]; // May not align 1:1 - - // LLM call node - nodes.push({ - id: llmNodeId, - label: `🤖 LLM Call #${i + 1}\n${call.inputTokens.toLocaleString()}↓ ${call.outputTokens.toLocaleString()}↑`, - type: "llmcall", - detail: { - desc: `LLM inference call #${i + 1}`, - tokens: { in: call.inputTokens, out: call.outputTokens }, - durationMs: call.durationMs, - model: call.model, - }, - }); - - edges.push({ - source: prevNodeId, - target: llmNodeId, - type: "llmcall", - label: `${call.inputTokens.toLocaleString()} in`, - }); - - // If there's a corresponding tool call, add it - if (toolCall) { - const toolNodeId = `t${i + 1}`; - nodes.push({ - id: toolNodeId, - label: `🔧 ${toolCall.tool}\n${toolCall.args.substring(0, 30)}`, - type: "matched", - detail: { - tool: toolCall.tool, - args: toolCall.args, - reasoning: toolCall.reasoning, - }, - }); - edges.push({ - source: llmNodeId, - target: toolNodeId, - type: "matched", - label: `${call.outputTokens.toLocaleString()} out`, - }); - prevNodeId = toolNodeId; - } else { - prevNodeId = llmNodeId; - } - } - - // End node - const endNodeId = "end_act"; - nodes.push({ - id: endNodeId, - label: `⬜ END\n${perCall.length} LLM calls`, - type: "endOk", - shape: "ellipse", - detail: { - totalTokens: { - in: latestEntry.inputTokens, - out: latestEntry.outputTokens, - llmCalls: latestEntry.apiCallCount, - }, - }, - }); - edges.push({ source: prevNodeId, target: endNodeId, type: "matched" }); - - traces[testName] = { - prompt: latestEntry.prompt, - model: latestEntry.model, - summary: { - inputTokens: latestEntry.inputTokens, - outputTokens: latestEntry.outputTokens, - llmCalls: latestEntry.apiCallCount, - durationMs: latestEntry.totalApiDurationMs, - }, - nodes, - edges, - }; - - // Add expected path and compute path adherence if available - const skillArea = extractSkillArea(testName); - const expectedPath = buildExpectedPath(skillArea); - if (expectedPath) { - traces[testName].expectedNodes = expectedPath.nodes; - traces[testName].expectedEdges = expectedPath.edges; - traces[testName].expectedSteps = expectedPath.stepCount; - - // Compute path adherence: compare actual tool calls vs expected steps - const actualTools = nodes.filter(n => n.type === "matched").map(n => ({ - tool: n.detail.tool, - args: n.detail.args || "", - })); - const expectedSteps = expectedPath.nodes - .filter(n => n.type === "expected") - .map(n => ({ tool: n.detail.tool, args: n.detail.args || "" })); - - let matched = 0, deviated = 0, extra = 0; - const matchedExpectedIdx = new Set(); - - for (let ai = 0; ai < actualTools.length; ai++) { - const at = actualTools[ai]; - // Find matching expected step (in order, not yet matched) - let found = false; - for (let ei = 0; ei < expectedSteps.length; ei++) { - if (matchedExpectedIdx.has(ei)) continue; - const et = expectedSteps[ei]; - if (at.tool === et.tool && (et.args === "" || at.args.includes(et.args))) { - matched++; - matchedExpectedIdx.add(ei); - found = true; - // Update node type to 'matched' - const matchIdx = nodes.findIndex(n => n.id === `t${ai + 1}`); - if (matchIdx >= 0) nodes[matchIdx].type = "matched"; - break; - } - } - if (!found) { - // Check if tool exists in expected but wrong order/args → deviated - const anyMatch = expectedSteps.some(et => at.tool === et.tool); - if (anyMatch) { - deviated++; - const nodeIdx = nodes.findIndex(n => n.id === `t${ai + 1}`); - if (nodeIdx >= 0) { - nodes[nodeIdx].type = "deviated"; - nodes[nodeIdx].label = nodes[nodeIdx].label.replace("🔧", "❌"); - } - // Update edge type too - const edgeIdx = edges.findIndex(e => e.target === `t${ai + 1}`); - if (edgeIdx >= 0) edges[edgeIdx].type = "deviated"; - } else { - extra++; - const nodeIdx = nodes.findIndex(n => n.id === `t${ai + 1}`); - if (nodeIdx >= 0) { - nodes[nodeIdx].type = "extra"; - nodes[nodeIdx].label = nodes[nodeIdx].label.replace("🔧", "⚡"); - } - const edgeIdx = edges.findIndex(e => e.target === `t${ai + 1}`); - if (edgeIdx >= 0) edges[edgeIdx].type = "extra"; - } - } - } - - const skipped = expectedSteps.length - matchedExpectedIdx.size; - const adherence = expectedSteps.length > 0 - ? Math.round((matched / expectedSteps.length) * 100) - : 0; - - // Update end node based on adherence - const endNode = nodes.find(n => n.id === "end_act"); - if (endNode) { - if (adherence >= 70) { - endNode.type = "endOk"; - endNode.label = `🟢 END\n${adherence}% adherence`; - } else if (adherence >= 40) { - endNode.type = "endWarn"; - endNode.label = `🟡 END\n${adherence}% adherence`; - } else { - endNode.type = "endBad"; - endNode.label = `🔴 END\n${adherence}% adherence`; - } - } - - traces[testName].pathAdherence = { - expected: expectedSteps.length, - actual: actualTools.length, - matched, - deviated, - extra, - skipped, - adherence, - }; - } - } - - return traces; -} - -// ─── Skill Coverage Analysis ───────────────────────────────────────────────── - -const SKILLS_PATH = path.resolve(__dirname, "../../plugin/skills"); -const TESTS_PATH = path.resolve(__dirname, ".."); - -/** - * Recursively find sub-skills within a skill directory. - * A sub-skill is a directory containing a SKILL.md or a primary .md file - * (excluding references/ directories). - */ -function findSubSkills(skillDir, prefix = "") { - const subSkills = []; - if (!fs.existsSync(skillDir)) return subSkills; - - const entries = fs.readdirSync(skillDir, { withFileTypes: true }); - for (const entry of entries) { - if (!entry.isDirectory()) continue; - if (["references", "templates", "assets", "scripts", "examples"].includes(entry.name)) continue; - - const subPath = path.join(skillDir, entry.name); - const relPath = prefix ? `${prefix}/${entry.name}` : entry.name; - - // Check if this directory is a sub-skill (has SKILL.md or a primary .md) - const hasSkillMd = fs.existsSync(path.join(subPath, "SKILL.md")); - const mdFiles = fs.readdirSync(subPath).filter(f => - f.endsWith(".md") && f !== "SKILL.md" && !f.startsWith("TEST_") && !f.startsWith("EXAMPLES") - ); - const hasPrimaryMd = mdFiles.length > 0; - - if (hasSkillMd || hasPrimaryMd) { - subSkills.push({ - name: entry.name, - path: relPath, - hasSkillMd, - primaryMd: hasSkillMd ? "SKILL.md" : mdFiles[0] || null, - }); - } - - // Recurse into subdirectories - const nested = findSubSkills(subPath, relPath); - subSkills.push(...nested); - } - return subSkills; -} - -/** - * Check if a test directory exists for a given skill/sub-skill path. - */ -function hasIntegrationTest(skillPath) { - // Direct match: tests//integration.test.ts - const testDir = path.join(TESTS_PATH, skillPath); - if (fs.existsSync(path.join(testDir, "integration.test.ts"))) return true; - - const parts = skillPath.split("/"); - const lastPart = parts[parts.length - 1]; - const parentSkill = parts[0]; - - // Check alternative test layout patterns - const testPaths = [ - // Nested under parent: tests/microsoft-foundry/quota/integration.test.ts - path.join(TESTS_PATH, parentSkill, lastPart, "integration.test.ts"), - // Skill path with models/deploy nesting: tests/microsoft-foundry/models/deploy/deploy-model - path.join(TESTS_PATH, skillPath, "integration.test.ts"), - ]; - - // Also try finding a test dir anywhere under parent that contains lastPart - if (fs.existsSync(path.join(TESTS_PATH, parentSkill))) { - const findTest = (dir) => { - if (!fs.existsSync(dir)) return false; - const entries = fs.readdirSync(dir, { withFileTypes: true }); - for (const e of entries) { - if (!e.isDirectory()) continue; - const sub = path.join(dir, e.name); - if (e.name === lastPart && fs.existsSync(path.join(sub, "integration.test.ts"))) return true; - if (findTest(sub)) return true; - } - return false; - }; - if (findTest(path.join(TESTS_PATH, parentSkill))) return true; - } - - return testPaths.some(p => fs.existsSync(p)); -} - -/** - * Build coverage analysis for all skills. - * Compares skill definitions (SKILL.md) against test coverage. - */ -function buildCoverage(areas) { - const coverage = {}; - - if (!fs.existsSync(SKILLS_PATH)) { - console.warn("⚠️ Skills directory not found, skipping coverage analysis"); - return coverage; - } - - const topLevelSkills = fs.readdirSync(SKILLS_PATH, { withFileTypes: true }) - .filter(e => e.isDirectory()) - .map(e => e.name); - - // Set of area keys that have test results - const testedAreas = new Set(areas.map(a => a.name)); - - for (const skillName of topLevelSkills) { - const skillDir = path.join(SKILLS_PATH, skillName); - const subSkills = findSubSkills(skillDir); - - // Check top-level integration test - const hasTopLevelTest = fs.existsSync( - path.join(TESTS_PATH, skillName, "integration.test.ts") - ); - const hasUnitTest = fs.existsSync( - path.join(TESTS_PATH, skillName, "unit.test.ts") - ); - const hasTriggersTest = fs.existsSync( - path.join(TESTS_PATH, skillName, "triggers.test.ts") - ); - - // Check sub-skill test coverage - const subSkillCoverage = subSkills.map(ss => { - const fullPath = `${skillName}/${ss.path}`; - const hasTest = hasIntegrationTest(fullPath); - // Also check if area appears in test results - const hasResults = testedAreas.has(fullPath) || - testedAreas.has(`${skillName}/${ss.name}`); - return { - name: ss.name, - path: ss.path, - hasTest, - hasResults, - }; - }); - - const coveredSubSkills = subSkillCoverage.filter(s => s.hasTest || s.hasResults).length; - const totalSubSkills = subSkillCoverage.length; - - coverage[skillName] = { - hasIntegrationTest: hasTopLevelTest, - hasUnitTest, - hasTriggersTest, - subSkills: { - total: totalSubSkills, - covered: coveredSubSkills, - percentage: totalSubSkills > 0 - ? Math.round((coveredSubSkills / totalSubSkills) * 100) - : (hasTopLevelTest ? 100 : 0), - missing: subSkillCoverage - .filter(s => !s.hasTest && !s.hasResults) - .map(s => s.path), - details: subSkillCoverage, - }, - }; - } - - return coverage; -} - -// ─── Main ──────────────────────────────────────────────────────────────────── - -function main() { - const { runDir, junitPath } = parseArgs(process.argv); - - // Resolve test run directory - let testRunName = runDir; - if (!testRunName) { - testRunName = getMostRecentTestRun(); - if (!testRunName) { - console.error("❌ No test runs found in", REPORTS_PATH); - process.exit(1); - } - } - const testRunPath = path.join(REPORTS_PATH, testRunName); - console.log(`📂 Processing test run: ${testRunName}`); - - // Parse JUnit XML - const resolvedJunit = junitPath || path.join(REPORTS_PATH, "junit.xml"); - console.log(`📋 Reading JUnit XML: ${resolvedJunit}`); - const junit = parseJunitXml(resolvedJunit); - - // Load token data - console.log("🔢 Loading token data..."); - const tokenEntries = loadTokenSummary(testRunPath); - console.log(` Found ${tokenEntries.length} token entries`); - - // Build report sections - console.log("📊 Building area summaries..."); - const areas = buildAreaSummaries(junit, tokenEntries); - - console.log("💬 Building token usage details..."); - const tokenUsage = buildTokenUsage(tokenEntries); - - console.log("🔀 Building execution traces..."); - const traces = buildTraces(testRunPath, tokenEntries); - - console.log("📋 Building skill coverage analysis..."); - const coverage = buildCoverage(areas); - - // Compute global summary - const totalTests = junit ? junit.totalTests : areas.reduce((s, a) => s + a.tests, 0); - const totalPassed = areas.reduce((s, a) => s + a.passed, 0); - const totalFailed = areas.reduce((s, a) => s + a.failed, 0); - const totalInputTokens = areas.reduce((s, a) => s + a.totalInputTokens, 0); - const totalOutputTokens = areas.reduce((s, a) => s + a.totalOutputTokens, 0); - const totalLLMCalls = areas.reduce((s, a) => s + a.totalLLMCalls, 0); - - // Build the contract - const report = { - version: CONTRACT_VERSION, - generatedAt: new Date().toISOString(), - testRun: testRunName, - model: tokenEntries.length > 0 ? tokenEntries[0].model : "unknown", - - summary: { - totalTests, - passed: totalPassed, - failed: totalFailed, - passRate: totalTests > 0 ? Math.round((totalPassed / totalTests) * 1000) / 10 : null, - totalInputTokens, - totalOutputTokens, - totalLLMCalls, - avgTokensPerTest: totalTests > 0 - ? Math.round((totalInputTokens + totalOutputTokens) / totalTests) - : 0, - totalDurationSec: junit ? Math.round(junit.time) : 0, - }, - - areas, - tokenUsage, - traces, - coverage, - }; - - // Write output - const outputPath = path.join(testRunPath, "skill-quality-report.json"); - fs.writeFileSync(outputPath, JSON.stringify(report, null, 2), "utf-8"); - console.log(`\n✅ Report generated: ${outputPath}`); - console.log(` Version: ${CONTRACT_VERSION}`); - console.log(` Tests: ${totalTests} (${totalPassed} passed, ${totalFailed} failed)`); - console.log(` Areas: ${areas.length}`); - console.log(` Traces: ${Object.keys(traces).length}`); - console.log(` Total tokens: ${(totalInputTokens + totalOutputTokens).toLocaleString()}`); - console.log(` Total LLM calls: ${totalLLMCalls}`); - - // Coverage summary - const covKeys = Object.keys(coverage); - const withTests = covKeys.filter(k => coverage[k].hasIntegrationTest).length; - const totalMissingSubs = covKeys.reduce((s, k) => s + coverage[k].subSkills.missing.length, 0); - console.log(` Coverage: ${withTests}/${covKeys.length} skills have integration tests, ${totalMissingSubs} sub-skills missing tests`); -} - -main(); diff --git a/tests/scripts/run-tests.js b/tests/scripts/run-tests.js index fcdba4fad..73e223435 100644 --- a/tests/scripts/run-tests.js +++ b/tests/scripts/run-tests.js @@ -6,7 +6,6 @@ * * Types: * all - Run all tests (default) - * unit - Run unit tests only * integration - Run integration tests only * verbose - Run all tests with verbose output * coverage - Run tests with coverage report @@ -16,12 +15,10 @@ * * Examples: * node run-tests.js # Run all tests - * node run-tests.js unit # Run unit tests * node run-tests.js integration # Run integration tests * node run-tests.js integration azure-deploy # Run integration tests for azure-deploy * node run-tests.js integration azure-deploy static-web-apps-deploy # Run integration tests for a sub group * node run-tests.js skill azure-ai # Run tests for azure-ai skill - * node run-tests.js unit --verbose # Run unit tests with verbose flag */ import { spawn } from "child_process"; @@ -46,10 +43,6 @@ const testConfigs = { description: "all tests", jestArgs: [] }, - unit: { - description: "unit tests", - jestArgs: ["--testPathIgnorePatterns=\"node_modules|_template|integration\""] - }, integration: { description: "integration tests", jestArgs: [ diff --git a/tests/utils/__tests__/unit.test.ts b/tests/utils/__tests__/unit.test.ts deleted file mode 100644 index 73717cc42..000000000 --- a/tests/utils/__tests__/unit.test.ts +++ /dev/null @@ -1,87 +0,0 @@ -import { redactSecrets } from "../redact"; - -describe("redactSecrets", () => { - test("redacts JWT tokens", () => { - const input = [ - "start", - "Authorization payload: eyJabcdefghijklmnopqrstuvwxyz.abcdefghijklmnopqrstuvwxyz.abcdefghijklmnopqrstuvwxyz", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "Authorization payload: [REDACTED]", - "end" - ].join("\n")); - }); - - test("redacts bearer tokens", () => { - const input = [ - "start", - "Authorization: Bearer abcdefghijklmnopqrstuvwxyz1234567890._-~+/", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "Authorization: [REDACTED]", - "end" - ].join("\n")); - }); - - test("redacts GitHub tokens", () => { - const input = [ - "start", - "token=ghp_abcdefghijklmnopqrstuvwxyzABCDEFGHIJ0123456789", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "token=[REDACTED]", - "end" - ].join("\n")); - }); - - test("redacts Azure SAS signatures", () => { - const input = [ - "start", - "url=https://example.blob.core.windows.net/c?sv=1&sig = AbCdEfGhIjKlMnOpQrStUvWxYz0123456789%2F%2B", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "url=https://example.blob.core.windows.net/c?sv=1&[REDACTED]", - "end" - ].join("\n")); - }); - - test("redacts key-value style secret values while preserving keys", () => { - const input = [ - "start", - "password=myVerySecret123 token: anotherSecret456 api-key=superSecret789 connection-string=Endpoint=sb://foo;SharedAccessKey=bar", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "password=[REDACTED] token: [REDACTED] api-key=[REDACTED] connection-string=[REDACTED]", - "end" - ].join("\n")); - }); - - test("does not redact non-secret or short values", () => { - const input = [ - "start", - "status=ok password=short token=tiny", - "end" - ].join("\n"); - - expect(redactSecrets(input)).toBe([ - "start", - "status=ok password=short token=tiny", - "end" - ].join("\n")); - }); -}); \ No newline at end of file From 61ae753f6e1e6c142ae0c5663228dcf599fb03f8 Mon Sep 17 00:00:00 2001 From: Chunan Ye Date: Mon, 18 May 2026 13:19:09 -0700 Subject: [PATCH 2/2] Remove more remaining mentioning of unit test --- .github/skills/sensei/references/LOOP.md | 1 - .github/skills/sensei/scripts/gepa/auto_evaluator.py | 2 +- docs/spec/azure-hosted-copilot-sdk.md | 1 - tests/AGENTS.md | 5 ++--- tests/README.md | 4 ++-- tests/airunway-aks-setup/README.md | 2 -- 6 files changed, 5 insertions(+), 10 deletions(-) diff --git a/.github/skills/sensei/references/LOOP.md b/.github/skills/sensei/references/LOOP.md index 7127659a0..d86ed41d2 100644 --- a/.github/skills/sensei/references/LOOP.md +++ b/.github/skills/sensei/references/LOOP.md @@ -108,7 +108,6 @@ The Ralph loop is an iterative improvement cycle inspired by the ["Ralph Wiggum" **Files to read:** ``` plugin/skills/{skill-name}/SKILL.md # Required -tests/{skill-name}/unit.test.ts # If exists tests/{skill-name}/triggers.test.ts # If exists tests/{skill-name}/integration.test.ts # If exists ``` diff --git a/.github/skills/sensei/scripts/gepa/auto_evaluator.py b/.github/skills/sensei/scripts/gepa/auto_evaluator.py index d85e3fccc..8e0822aa7 100644 --- a/.github/skills/sensei/scripts/gepa/auto_evaluator.py +++ b/.github/skills/sensei/scripts/gepa/auto_evaluator.py @@ -7,7 +7,7 @@ Currently, the evaluator: - parses triggers.test.ts files to extract trigger prompt arrays - - detects the presence of unit.test.ts and integration.test.ts files + - detects the presence of integration.test.ts files - uses this structural information plus content/keyword heuristics to construct a fitness function diff --git a/docs/spec/azure-hosted-copilot-sdk.md b/docs/spec/azure-hosted-copilot-sdk.md index de5c68944..9d1cb6bb3 100644 --- a/docs/spec/azure-hosted-copilot-sdk.md +++ b/docs/spec/azure-hosted-copilot-sdk.md @@ -43,7 +43,6 @@ The `azure-hosted-copilot-sdk` skill enables users to build, deploy, and configu | Test File | Tests | Status | |-----------|-------|--------| | `triggers.test.ts` | 5 test blocks (10 trigger, 8 negative, 5 edge cases) | ✅ Complete | -| `unit.test.ts` | 17 tests (metadata, content, BYOM, frontmatter) | ✅ Complete | | `integration.test.ts` | 8 tests (invocation rate + content quality) | ✅ Complete | | `__snapshots__/triggers.test.ts.snap` | Keyword snapshots | ✅ Regenerated | diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 1f4fbfbb1..9a27af4df 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -27,7 +27,7 @@ Load the file at `plugin/skills/{skill-name}/SKILL.md` to understand: > **Frontmatter Format Rule:** Descriptions over 200 characters MUST use folded YAML format (`>-`) for maintainability. The `>-` format keeps descriptions readable in source while parsing to a flat string compatible with skills.sh and other registries. Do NOT use `|` (literal block) as it preserves newlines. ### Step 3: Update test files -In each test file (`unit.test.ts`, `triggers.test.ts`), change: +In each test file `triggers.test.ts`, change: ```typescript const SKILL_NAME = '{skill-name}'; // Must match the folder name exactly ``` @@ -70,7 +70,7 @@ npm run coverage:grid ## Overview -This testing framework uses **Jest** to validate skill behavior across three test categories: +This testing framework uses **Jest** to validate skill behavior across these test categories: - **Trigger Tests** - Skill activation validation - **Integration Tests** - MCP tool interaction testing @@ -84,7 +84,6 @@ This testing framework uses **Jest** to validate skill behavior across three tes | File | Purpose | |------|---------| -| `unit.test.ts` | Tests isolated skill logic and metadata | | `triggers.test.ts` | Tests skill activation on prompts | | `integration.test.ts` | Tests real Copilot agent behavior (optional) | | `fixtures/*.json` | Test data and mock responses | diff --git a/tests/README.md b/tests/README.md index 06cf158e2..3476a7182 100644 --- a/tests/README.md +++ b/tests/README.md @@ -41,7 +41,7 @@ Each skill in `/plugin/skills/{skill-name}/` can have a corresponding test suite 3. **Each test file loads its skill** via `utils/skill-loader.ts` 4. **Tests execute** - execute test code and generate output -There are 3 types of tests. +There are 2 types of tests. - trigger test: tests that validate if the description of a skill can trigger or not trigger a given prompt using a heuristic. - integration test: tests that validate if the skill can lead to successful completion of a task by running a given prompt against a Copilot SDK agent. @@ -287,7 +287,7 @@ cp -r _template {skill-name} Edit each test file and change the `SKILL_NAME` constant: ```typescript -// In unit.test.ts, triggers.test.ts, integration.test.ts +// In triggers.test.ts, integration.test.ts const SKILL_NAME = 'azure-redis'; // ← Change this to match your skill folder ``` diff --git a/tests/airunway-aks-setup/README.md b/tests/airunway-aks-setup/README.md index 5783e03b3..7515b8135 100644 --- a/tests/airunway-aks-setup/README.md +++ b/tests/airunway-aks-setup/README.md @@ -19,7 +19,6 @@ npm run update:snapshots -- --testPathPatterns=airunway-aks-setup ``` airunway-aks-setup/ -├── unit.test.ts # Skill metadata, structure, and content tests ├── triggers.test.ts # Skill activation / non-activation tests ├── integration.test.ts # Real Copilot agent tests (requires SDK auth) ├── __snapshots__/ @@ -32,7 +31,6 @@ airunway-aks-setup/ | File | What it covers | |------|----------------| -| `unit.test.ts` | Frontmatter validity, six-phase workflow presence, step file references, kubectl/make tool references, Error Handling section | | `triggers.test.ts` | Direct AI Runway prompts, intent-based GPU/LLM prompts, negative cases (wrong cloud, wrong domain), edge cases | | `integration.test.ts` | Skill invocation rate, controller install guidance, GPU assessment response, model recommendation, ModelDeployment CR, kubeconfig workspace setup |