Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/agents/SkillCreator.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,12 @@ REQUIREMENTS.md should _always_ include the following:

When creating a new skill, tests must be created following the patterns documented in `/tests/AGENTS.md`. The test suite should include:

1. **Unit Tests** (`tests/{skill-name}/unit.test.js`):
- Validate SKILL.md metadata
- Test any validation or processing logic

2. **Trigger Tests** (`tests/{skill-name}/triggers.test.js`):
1. **Trigger Tests** (`tests/{skill-name}/triggers.test.js`):
- At least 5 prompts that SHOULD trigger the skill
- At least 5 prompts that should NOT trigger the skill
- Snapshot tests for keyword changes

3. **Integration Tests** (`tests/{skill-name}/integration.test.js`) - if applicable:
2. **Integration Tests** (`tests/{skill-name}/integration.test.js`) - if applicable:
- Mock MCP tool interactions
- Test error handling

Expand Down
1 change: 0 additions & 1 deletion .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@ cd tests
npm install
npm test # Run all tests
npm test -- --testPathPatterns=<skill-name> # Run tests for a single skill
npm run test:unit # Unit tests only
npm run typecheck # TypeScript type checking
npm run lint # ESLint
```
Expand Down
1 change: 0 additions & 1 deletion .github/skills/sensei/references/LOOP.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,6 @@ The Ralph loop is an iterative improvement cycle inspired by the ["Ralph Wiggum"
**Files to read:**
```
plugin/skills/{skill-name}/SKILL.md # Required
tests/{skill-name}/unit.test.ts # If exists
tests/{skill-name}/triggers.test.ts # If exists
tests/{skill-name}/integration.test.ts # If exists
```
Expand Down
2 changes: 1 addition & 1 deletion .github/skills/sensei/scripts/gepa/auto_evaluator.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

Currently, the evaluator:
- parses triggers.test.ts files to extract trigger prompt arrays
- detects the presence of unit.test.ts and integration.test.ts files
- detects the presence of integration.test.ts files
- uses this structural information plus content/keyword heuristics to
construct a fitness function

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ See [routing-analysis.md](routing-analysis.md) for detailed checks.
|-------|------|
| Registered | Skill listed in `tests/skills.json` |
| Trigger tests | `tests/{skill-name}/triggers.test.ts` exists with shouldTrigger/shouldNotTrigger arrays |
| Unit tests | `tests/{skill-name}/unit.test.ts` exists |
| Snapshots | Updated if description changed |

## 10. Positive Acknowledgment
Expand Down
3 changes: 0 additions & 3 deletions docs/spec/azure-hosted-copilot-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ The `azure-hosted-copilot-sdk` skill enables users to build, deploy, and configu
| Test File | Tests | Status |
|-----------|-------|--------|
| `triggers.test.ts` | 5 test blocks (10 trigger, 8 negative, 5 edge cases) | ✅ Complete |
| `unit.test.ts` | 17 tests (metadata, content, BYOM, frontmatter) | ✅ Complete |
| `integration.test.ts` | 8 tests (invocation rate + content quality) | ✅ Complete |
| `__snapshots__/triggers.test.ts.snap` | Keyword snapshots | ✅ Regenerated |

Expand All @@ -54,7 +53,6 @@ The `azure-hosted-copilot-sdk` skill enables users to build, deploy, and configu
| `tests/utils/agent-runner.ts` | Enhanced with `runConversation()` for multi-turn scenarios | ✅ |
| `tests/utils/evaluate.ts` | Shared evaluation helpers (`matchesCommand`, `getAllAssistantMessages`, etc.) | ✅ |
| `tests/utils/regression-detectors.ts` | Regression detectors (secrets, ACR spirals, port confusion, etc.) | ✅ |
| `tests/_template/unit.test.ts` | Unit test template for new skills | ✅ |

### Local Dev Tooling — 🔄 Replaced

Expand Down Expand Up @@ -167,7 +165,6 @@ docs/spec/
tests/azure-hosted-copilot-sdk/
├── integration.test.ts # 8 integration tests
├── triggers.test.ts # 5 test blocks (23 parameterized cases)
├── unit.test.ts # 17 unit tests
└── __snapshots__/triggers.test.ts.snap # Trigger keyword snapshots

tests/utils/
Expand Down
34 changes: 4 additions & 30 deletions tests/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Load the file at `plugin/skills/{skill-name}/SKILL.md` to understand:
> **Frontmatter Format Rule:** Descriptions over 200 characters MUST use folded YAML format (`>-`) for maintainability. The `>-` format keeps descriptions readable in source while parsing to a flat string compatible with skills.sh and other registries. Do NOT use `|` (literal block) as it preserves newlines.

### Step 3: Update test files
In each test file (`unit.test.ts`, `triggers.test.ts`), change:
In each test file `triggers.test.ts`, change:
```typescript
const SKILL_NAME = '{skill-name}'; // Must match the folder name exactly
```
Expand All @@ -45,15 +45,12 @@ Based on the skill's description and content, add to `triggers.test.ts`:
- Different cloud providers ("AWS", "GCP")
- Different Azure services not covered by this skill

### Step 5: Add skill-specific unit tests
In `unit.test.ts`, add tests that verify the skill's content contains expected sections, commands, or patterns documented in its SKILL.md.

### Step 6: Configure integration tests (optional)
### Step 5: Configure integration tests (optional)
In `integration.test.ts`, customize the prompts to test real agent behavior.

Follow existing `integration.test.ts` files for how to implement such tests.

### Step 7: Run and verify
### Step 6: Run and verify
```bash
cd tests

Expand All @@ -73,8 +70,7 @@ npm run coverage:grid

## Overview

This testing framework uses **Jest** to validate skill behavior across three test categories:
- **Unit Tests** - Isolated logic testing
This testing framework uses **Jest** to validate skill behavior across these test categories:
- **Trigger Tests** - Skill activation validation
- **Integration Tests** - MCP tool interaction testing

Expand All @@ -88,7 +84,6 @@ This testing framework uses **Jest** to validate skill behavior across three tes

| File | Purpose |
|------|---------|
| `unit.test.ts` | Tests isolated skill logic and metadata |
| `triggers.test.ts` | Tests skill activation on prompts |
| `integration.test.ts` | Tests real Copilot agent behavior (optional) |
| `fixtures/*.json` | Test data and mock responses |
Expand All @@ -108,26 +103,6 @@ tests/{skill-name}/

---

## Writing Unit Tests

Unit tests validate skill metadata and any extractable logic.

### Required Tests

Every skill should have these basic unit tests:

```typescript
describe('Skill Metadata', () => {
test('has valid SKILL.md with required fields', () => {
expect(skill.metadata.name).toBe(SKILL_NAME);
expect(skill.metadata.description).toBeDefined();
expect(skill.metadata.description.length).toBeGreaterThan(10);
});
});
```

---

## Writing Trigger Tests

Trigger tests verify that prompts correctly activate (or don't activate) your skill.
Expand Down Expand Up @@ -392,7 +367,6 @@ When adding tests for a new skill:
- [ ] Update `SKILL_NAME` in all test files
- [ ] Add 5+ prompts that should trigger
- [ ] Add 5+ prompts that should NOT trigger
- [ ] Add unit tests for any validation logic
- [ ] Run tests locally and verify passing
- [ ] Update coverage grid if significant changes

Expand Down
75 changes: 5 additions & 70 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,8 @@ Each skill in `/plugin/skills/{skill-name}/` can have a corresponding test suite
3. **Each test file loads its skill** via `utils/skill-loader.ts`
4. **Tests execute** - execute test code and generate output

There are 3 types of tests.
There are 2 types of tests.

- unit test: tests that validate the file structure of a skill and the content.
- trigger test: tests that validate if the description of a skill can trigger or not trigger a given prompt using a heuristic.
- integration test: tests that validate if the skill can lead to successful completion of a task by running a given prompt against a Copilot SDK agent.

Expand Down Expand Up @@ -71,17 +70,7 @@ Run tests manually anytime during development (see [Running Tests Locally](#runn

## What Tests Validate

### 1. Unit Tests (`unit.test.ts`)

**Purpose:** Validate skill metadata and any embedded logic.

**What it checks:**
- `SKILL.md` exists and has valid frontmatter (`name`, `description`)
- Description is meaningful (not empty, appropriate length)
- Content contains expected sections
- Any validation rules documented in the skill work correctly

### 2. Trigger Tests (`triggers.test.ts`)
### 1. Trigger Tests (`triggers.test.ts`)

**Purpose:** Verify the skill activates on correct prompts and ignores unrelated ones.

Expand All @@ -96,7 +85,7 @@ Run tests manually anytime during development (see [Running Tests Locally](#runn
npm run update:snapshots -- --testPathPatterns={skill-name}
```

### 3. Integration Tests (`integration.test.ts`)
### 2. Integration Tests (`integration.test.ts`)

**Purpose:** Test skill behavior with a real Copilot agent session.

Expand Down Expand Up @@ -298,7 +287,7 @@ cp -r _template {skill-name}
Edit each test file and change the `SKILL_NAME` constant:

```typescript
// In unit.test.ts, triggers.test.ts, integration.test.ts
// In triggers.test.ts, integration.test.ts
const SKILL_NAME = 'azure-redis'; // ← Change this to match your skill folder
```

Expand All @@ -322,66 +311,12 @@ const shouldNotTriggerPrompts = [
];
```

#### Step 4: Customize Unit Tests

In `unit.test.ts`, add tests specific to your skill's content:

```typescript
test('documents cache tiers', () => {
expect(skill.content).toContain('Basic');
expect(skill.content).toContain('Standard');
expect(skill.content).toContain('Premium');
});
```

#### Step 5: Run and Verify
#### Step 4: Run and Verify

```bash
npm run test:skill -- {skill-name}
```

#### Step 6: Update Coverage Grid

```bash
npm run coverage:grid
```

This updates the Skills Coverage Grid in this README.

---

## Skills Coverage Grid

<!-- COVERAGE_GRID_START -->
| Skill | Tests | Unit | Triggers | Integration | Coverage |
|-------|-------|------|----------|-------------|----------|
| appinsights-instrumentation | ✅ | ✅ | ✅ | ✅ | - |
| azure-ai | ✅ | ✅ | ✅ | ✅ | - |
| azure-aigateway | ✅ | ✅ | ✅ | ✅ | - |
| azure-cloud-migrate | ✅ | ✅ | ✅ | ✅ | - |
| azure-compliance | ✅ | ✅ | ✅ | ✅ | - |
| azure-compute | ✅ | ✅ | ✅ | ✅ | - |
| azure-cost | ✅ | ✅ | ✅ | ✅ | - |
| azure-deploy | ✅ | ✅ | ✅ | ✅ | - |
| azure-diagnostics | ✅ | ✅ | ✅ | ✅ | - |
| azure-enterprise-infra-planner | ✅ | ✅ | ✅ | ✅ | - |
| azure-hosted-copilot-sdk | ✅ | ✅ | ✅ | ✅ | - |
| azure-kusto | ✅ | - | - | ✅ | - |
| azure-messaging | ✅ | ✅ | ✅ | ✅ | - |
| azure-prepare | ✅ | ✅ | ✅ | ✅ | - |
| azure-quotas | ✅ | ✅ | ✅ | ✅ | - |
| azure-rbac | ✅ | ✅ | ✅ | ✅ | - |
| azure-resource-lookup | ✅ | ✅ | ✅ | ✅ | - |
| azure-resource-visualizer | ✅ | ✅ | ✅ | ✅ | - |
| azure-storage | ✅ | - | - | ✅ | - |
| azure-upgrade | ✅ | ✅ | ✅ | ✅ | - |
| azure-validate | ✅ | ✅ | ✅ | ✅ | - |
| entra-app-registration | ✅ | ✅ | ✅ | ✅ | - |
| microsoft-foundry | ✅ | ✅ | ✅ | ✅ | - |

**Legend:** ✅ Exists | ❌ Missing | Coverage shown as percentage
<!-- COVERAGE_GRID_END -->

---

## Troubleshooting
Expand Down
5 changes: 0 additions & 5 deletions tests/_template/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,6 @@ your-skill-name/

## Test Types

### Unit Tests (`unit.test.js`)
- Test skill metadata (SKILL.md parsing)
- Test validation logic
- Test utility functions

### Trigger Tests (`triggers.test.js`)
- Verify correct prompts activate the skill
- Verify unrelated prompts don't activate
Expand Down
82 changes: 0 additions & 82 deletions tests/_template/unit.test.ts

This file was deleted.

2 changes: 0 additions & 2 deletions tests/airunway-aks-setup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ npm run update:snapshots -- --testPathPatterns=airunway-aks-setup

```
airunway-aks-setup/
├── unit.test.ts # Skill metadata, structure, and content tests
├── triggers.test.ts # Skill activation / non-activation tests
├── integration.test.ts # Real Copilot agent tests (requires SDK auth)
├── __snapshots__/
Expand All @@ -32,7 +31,6 @@ airunway-aks-setup/

| File | What it covers |
|------|----------------|
| `unit.test.ts` | Frontmatter validity, six-phase workflow presence, step file references, kubectl/make tool references, Error Handling section |
| `triggers.test.ts` | Direct AI Runway prompts, intent-based GPU/LLM prompts, negative cases (wrong cloud, wrong domain), edge cases |
| `integration.test.ts` | Skill invocation rate, controller install guidance, GPU assessment response, model recommendation, ModelDeployment CR, kubeconfig workspace setup |

Expand Down
Loading
Loading