fix: restore skill routing disambiguation for azure-hosted-copilot-sdk by jongio · Pull Request #1644 · microsoft/GitHub-Copilot-for-Azure

Jon Gallant (jongio) · 2026-04-01T18:06:40Z

Summary

Restores the DO NOT USE FOR disambiguation clause and adds codebase detection to azure-hosted-copilot-sdk, fixing the routing regression where azure-prepare incorrectly captured generic deploy prompts (e.g., "Deploy this app to Azure") when the codebase contains @github/copilot-sdk. Also updates the sensei and skill-authoring guidelines that caused this regression.

Root Cause Analysis

The Regression (3 Layers)

Layer 1 -- Primary Cause: Commit a1ba84c (PR #1210)

PR #1210 ("Reduce char count of existing skills", merged 2026-03-10) removed the DO NOT USE FOR: general web apps without copilot SDK clause from azure-hosted-copilot-sdk's frontmatter description. This clause was the critical negative discriminator that told the LLM routing engine: "do not select this skill for generic deploy prompts -- prefer azure-prepare instead." Without it, azure-prepare wins on prompts like "Deploy this app to Azure" because it has "deploy to Azure" as a direct trigger phrase.

Layer 2 -- Contributing: azure-prepare only checked prompt keywords

azure-prepare's Step 0 routing logic only examined prompt keywords (e.g., "copilot SDK" in the user's message), not codebase markers (e.g., @github/copilot-sdk in package.json). The failing test prompt "Deploy this app to Azure" contains no copilot keywords -- they exist only in the codebase fixture (copilot-sdk-app-package.json).

Layer 3 -- Contributing: Detection timing

The regression was introduced at commit a1ba84c (2026-03-10) but only detected when the nightly integration test run happened to tag commit f47b744 (2026-03-30) as failing. The actual causal change is 20 days older than the tagged failing commit.

Commit Trace

Date	Commit	Description	Impact
2026-03-10	`a1ba84c`	PR #1210: "Reduce char count of existing skills"	ROOT CAUSE -- Removed `DO NOT USE FOR` from description
2026-03-30	`f47b744`	PR #1574: Updated azure-prepare aspire reference	Tagged as failing commit by nightly test, but NOT the cause

Engineering Process Gap Analysis

Why This Regression Shipped

6 specific gaps allowed this regression to merge undetected:

Gap 1: Integration tests do not run on PRs

test-all-integration.yml only runs on nightly schedule (Mon-Fri cron) or manual trigger
PR CI only runs test-all-skills.yml (non-integration: trigger tests + unit tests)
The integration test for Scenario 3 (integration.test.ts:89-102) would have caught this immediately if it ran on the PR

Gap 2: Unit test was a no-op for removed clauses

unit.test.ts:118-126 used a conditional check: if (description.includes("DO NOT USE FOR")) -- this passes silently when the clause is removed entirely
The test validated format consistency (colon after the phrase) but not presence
Fixed in this PR: Added 2 new regression guard tests that REQUIRE DO NOT USE FOR and PREFER OVER azure-prepare clauses

Gap 3: Copilot PR reviewer gave incorrect assessment

The automated Copilot reviewer on PR Reduce char count of existing skills #1210 stated that DO NOT USE FOR clauses were "adding length without routing benefit"
This was factually wrong -- the clause had critical routing benefit for disambiguation against azure-prepare

Gap 4: Sensei/skill-authoring guidelines actively DISCOURAGE `DO NOT USE FOR`

.github/skills/sensei/SKILL.md warned against DO NOT USE FOR as "risky in multi-skill environments"
.github/skills/skill-authoring/references/guidelines/frontmatter.md said "Do NOT add DO NOT USE FOR clauses"
This means PR Reduce char count of existing skills #1210 was following the established guidance when it removed the clause
The guidance was wrong for skills with competing routing against broader skills like azure-prepare
Fixed in this PR: Updated all 3 guideline files to carve out a disambiguation exception

Gap 5: No automated rule validates disambiguation for overlapping skills

No sensei rule checks that skills whose triggers overlap with azure-prepare maintain disambiguation clauses
No pre-merge gate validates skill routing correctness beyond keyword extraction

Gap 6: Waza evals run separately from PR CI

eval.yml workflow runs waza evaluations on a separate schedule
Waza is not a description/frontmatter validator -- it runs integration-style evals
Even if it had caught the regression, it wouldn't have blocked the PR

What the Team Could Have Done Differently

Run integration tests before merging PR Reduce char count of existing skills #1210 -- Even a manual workflow_dispatch of test-all-integration.yml would have caught the failure
Question the Copilot reviewer's assessment -- The human reviewer could have tested whether removing DO NOT USE FOR affected routing for the known integration test scenarios
Recognize that DO NOT USE FOR is routing-critical -- The clause wasn't decorative; it was a routing instruction to the LLM. Removing it without running routing tests was the equivalent of removing an API permission check without running auth tests.

Changes Made

Skill Fixes

1. `plugin/skills/azure-hosted-copilot-sdk/SKILL.md` (v1.0.1 -> v1.0.2)

Restored DO NOT USE FOR: general web apps without copilot SDK, standard App Service deployments, Azure Functions without SDK
Added PREFER OVER azure-prepare when codebase contains @github/copilot-sdk
Added new "Codebase Detection -- MANDATORY FIRST CHECK" section with marker file table
Added warning about generic deploy prompts requiring codebase inspection

2. `plugin/skills/azure-prepare/SKILL.md` (v1.1.1 -> v1.1.2)

Expanded Step 0 from prompt-only check to "prompt OR codebase" check
Split into "Check 1: Prompt keywords" and "Check 2: Codebase markers"
Added marker table with detection -> action mapping (e.g., @github/copilot-sdk in package.json -> hand off to azure-hosted-copilot-sdk)

Test Fixes

3. `tests/azure-hosted-copilot-sdk/unit.test.ts`

Added regression guard: description contains DO NOT USE FOR clause to disambiguate from azure-prepare
Added regression guard: description contains PREFER OVER clause for codebase-based routing
These tests will fail immediately if someone removes these clauses again

4. `tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap`

Updated snapshots to match new description keywords

Guideline Fixes (Addresses Gap 4)

5. `.github/skills/sensei/SKILL.md`

Added disambiguation exception to the DO NOT USE FOR warning: skills whose triggers overlap with broader skills MUST retain the clause
Updated both warning blocks to reference Integration test failure: azure-hosted-copilot-sdk – existing SDK app deploy codebase scan [Skill not invoked] #1599

6. `.github/skills/sensei/references/SCORING.md`

Added new row to risk assessment table: "Specialized skill with trigger overlap" -> REQUIRED
Updated scoring section: anti-triggers for disambiguation emit NO WARNING
Updated pseudocode: scoreSkill() now checks for overlapping triggers before warning
Updated collectSuggestions() to not suggest removal for disambiguation-critical skills

7. `.github/skills/skill-authoring/references/guidelines/frontmatter.md`

Changed blanket "Do NOT add DO NOT USE FOR" to conditional guidance with disambiguation exception
Added reference to Integration test failure: azure-hosted-copilot-sdk – existing SDK app deploy codebase scan [Skill not invoked] #1599

Process Improvements (Recommendations #4-6)

8. `.github/PULL_REQUEST_TEMPLATE.md` (NEW)

PR template with skill routing verification checklist

9. `.github/workflows/test-all-integration.yml`

Changed continue-on-error: true to false on integration test step

10. `tests/azure-quotas/unit.test.ts`

Added warning comment about fragile conditional pattern (Integration test failure: azure-hosted-copilot-sdk – existing SDK app deploy codebase scan [Skill not invoked] #1599)

Test Results

Unit tests:   19/19 passed (includes 2 new regression guards)
Trigger tests: 23/23 passed

Recommendations to Prevent Future Regressions

#	Recommendation	Priority	Status
1	Add targeted integration test subset to PR CI	Critical	Follow-up: #1645
2	~~Update sensei/skill-authoring guidance~~	~~Critical~~	Done in this PR
3	Add sensei rule for disambiguation validation	High	Follow-up: #1646
4	~~Audit conditional unit tests~~	~~High~~	Done in this PR
5	~~Add PR template checkbox~~	~~Medium~~	Done in this PR
6	~~Set `continue-on-error: false`~~	~~Medium~~	Done in this PR

Copilot

Pull request overview

Restores routing disambiguation for azure-hosted-copilot-sdk to prevent azure-prepare from incorrectly capturing generic Azure deploy prompts when the repo contains Copilot SDK markers (Fixes #1599).

Changes:

Reintroduced DO NOT USE FOR and added PREFER OVER azure-prepare disambiguation in azure-hosted-copilot-sdk skill metadata.
Updated azure-prepare Step 0 to route based on prompt or codebase markers (e.g., @github/copilot-sdk).
Added regression-guard unit tests and updated trigger snapshots for the new description content.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
plugin/skills/azure-hosted-copilot-sdk/SKILL.md	Restores negative/priority routing clauses and documents mandatory codebase detection markers.
plugin/skills/azure-prepare/SKILL.md	Expands specialized-tech routing to include codebase-marker checks, not just prompt keywords.
tests/azure-hosted-copilot-sdk/unit.test.ts	Adds regression guard tests requiring routing-disambiguation phrases in the description.
tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap	Updates expected extracted keywords/triggers to match the new description text.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Wallace Breza (wbreza)

Code Review Summary

I've performed a thorough review of PR #1644. The PR addresses a critical routing regression and makes important improvements to prevent future issues. However, I found 2 medium-severity issues that should be addressed:

Issue 1: Description exceeds recommended word count guideline

File: plugin/skills/azure-hosted-copilot-sdk/SKILL.md:3
Severity: Medium

Problem: The new description contains 107 words, exceeding the recommended 60-word limit for cross-model reliability stated in .github/skills/sensei/references/SCORING.md. The guideline explicitly states: "Trim description to ≤60 words for cross-model reliability"

Evidence:

Character count: 815 (under 1024 limit ✓)
Word count: 107 (exceeds 60 recommendation ✗)

Suggested fix: Consider consolidating the description by:

Removing redundant phrasing in trigger phrases
Moving some technical details to the skill body rather than frontmatter
Keeping only the most distinctive trigger phrases

Note: I recognize this is a deliberate trade-off for disambiguation clarity, and the updated sensei guidelines now allow exceptions for disambiguation-critical skills. However, it's worth considering if the same disambiguation can be achieved more concisely.

Issue 2: Inconsistent marker detection tables between skills

Files:

plugin/skills/azure-hosted-copilot-sdk/SKILL.md:205-210
plugin/skills/azure-prepare/SKILL.md:247-250

Severity: Medium

Problem: The codebase marker detection tables are inconsistent between the two skills that need to coordinate routing:

azure-hosted-copilot-sdk lists 4 markers:

@github/copilot-sdk in package.json dependencies
copilot-sdk in package.json name or dependencies
CopilotClient in source files
createSession + sendAndWait in source files

azure-prepare lists only 2 markers:

@github/copilot-sdk in dependencies in package.json
CopilotClient import in .ts/.js source files

Impact: If azure-prepare doesn't check for all the same markers that azure-hosted-copilot-sdk expects, routing could still fail in edge cases (e.g., a project with copilot-sdk in package.json name but not as @github/copilot-sdk).

Suggested fix: Synchronize these tables so both skills use identical detection logic. Either:

Add the missing markers to azure-prepare's table (recommended for completeness), OR
Document why the markers differ if there's a deliberate reason

Positive Observations

✅ Core regression fix is sound: Restoring DO NOT USE FOR with proper context
✅ Test coverage improved: New regression guard tests will prevent this from happening again
✅ Process improvements: PR template, guideline updates, and CI changes address root causes
✅ Comprehensive documentation: Excellent root cause analysis in PR description
✅ All CI checks passing: 10 successful checks including ESLint, CodeQL, and all skill tests

Recommendation

The core logic changes are solid and address a real production issue. The two issues I've identified are related to consistency and guidelines compliance rather than critical bugs. I recommend:

Fix Issue #2 (marker inconsistency) before merging - this could affect routing correctness
Consider Issue #1 (word count) for a follow-up - not blocking if the team accepts the trade-off

Great work on the thorough root cause analysis and process improvements! 👍

Jon Gallant (jongio) · 2026-04-02T17:14:41Z

Thanks for the thorough review Wallace Breza (@wbreza) - both issues are fixed in the latest push (4e2c024):

Issue 1 (word count): Trimmed the description from 107 to 55 words (under the 60-word guideline). Removed redundant phrases and implementation details while keeping all routing-critical triggers (copilot SDK, BYOM, CopilotClient, DO NOT USE FOR clause). The removed phrases like DefaultAzureCredential and azure openai model are in the skill body/steps and aren't needed for initial routing.

Issue 2 (marker inconsistency): Added the 2 missing markers to azure-prepare's Check 2 table so both skills use identical detection logic:

copilot-sdk in name or dependencies (package.json)
createSession + sendAndWait calls (.ts/.js source files)

Version bumps: azure-hosted-copilot-sdk 1.0.2->1.0.3, azure-prepare 1.1.2->1.1.3. All tests pass (19 unit + 18 trigger).

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

microsoft#1599) Restore DO NOT USE FOR clause and add codebase detection to fix routing regression where azure-prepare incorrectly won on generic deploy prompts (e.g. 'Deploy this app to Azure') when the codebase contains @github/copilot-sdk. Root cause: PR microsoft#1210 (commit a1ba84c) removed the DO NOT USE FOR clause from azure-hosted-copilot-sdk description, eliminating the critical negative discriminator that prevented azure-prepare from capturing Copilot SDK deployment prompts. Changes: - azure-hosted-copilot-sdk/SKILL.md: Restore DO NOT USE FOR clause, add PREFER OVER azure-prepare directive, add Codebase Detection section with marker table (v1.0.1 -> v1.0.2) - azure-prepare/SKILL.md: Expand Step 0 to check codebase markers (not just prompt keywords) for Copilot SDK detection (v1.1.1 -> v1.1.2) - unit.test.ts: Add 2 regression guard tests requiring DO NOT USE FOR and PREFER OVER clauses to prevent silent removal - triggers.test.ts.snap: Updated snapshots for new description Fixes microsoft#1599 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…R for disambiguation Add exception to the blanket 'do not use DO NOT USE FOR' guidance: skills whose triggers overlap with broader skills (e.g., azure-prepare) MUST retain DO NOT USE FOR clauses for routing disambiguation. The previous guidance actively caused the microsoft#1599 regression — PR microsoft#1210 removed the clause while following sensei's recommendation, breaking routing for azure-hosted-copilot-sdk. Updated files: - sensei/SKILL.md: Add exception note to warning blocks - sensei/references/SCORING.md: Add disambiguation row to risk table, update pseudocode to skip warning for overlapping skills - skill-authoring frontmatter.md: Add exception for disambiguation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- #4: Audit conditional unit tests — added warning comments to azure-quotas and azure-hosted-copilot-sdk about the fragile if(description.includes('DO NOT USE FOR')) pattern - #5: Add PR template with skill routing verification checkbox - microsoft#6: Set continue-on-error: false in test-all-integration.yml so integration test failures actually fail the workflow Part of microsoft#1599 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…comments, genericize refs - Bump sensei version 1.0.1 -> 1.0.2 - Bump skill-authoring version 1.0.0 -> 1.0.1 - Update snapshot header to jestjs.io (CI Jest version compatibility) - Fix PR template: correct integration test command, remove issue link - Update stale workflow comments to match continue-on-error: false - Remove explicit issue number references from skill and test files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ion exception - skill-authoring Constraints: changed blanket prohibition to conditional guidance - sensei help banner: added '(unless disambiguation-critical)' qualifier - test-all-integration.yml: clarified continue-on-error comment wording Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Main branch already has sensei at version 1.0.2 from another merged PR. Our PR also modified sensei, so we need a higher version to pass the Skill Structure CI check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@wbreza

Address @wbreza review feedback: - Issue 1: Trim azure-hosted-copilot-sdk description from 107 to 55 words (under 60-word guideline). Removed redundant/implementation-detail phrases while preserving all routing-critical triggers. - Issue 2: Sync codebase marker tables - added 2 missing markers to azure-prepare Check 2 (copilot-sdk in name/deps, createSession+sendAndWait) so both skills use identical detection logic. - Version bumps: azure-hosted-copilot-sdk 1.0.2->1.0.3, azure-prepare 1.1.2->1.1.3 - Updated unit test to match trimmed description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix snapshot header to jestjs.io URL (CI requirement) - Bump azure-prepare 1.1.3->1.1.4 (main already at 1.1.3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@wbreza

microsoft#1644) * fix: restore skill routing disambiguation for azure-hosted-copilot-sdk (microsoft#1599) Restore DO NOT USE FOR clause and add codebase detection to fix routing regression where azure-prepare incorrectly won on generic deploy prompts (e.g. 'Deploy this app to Azure') when the codebase contains @github/copilot-sdk. Root cause: PR microsoft#1210 (commit f41a5fa) removed the DO NOT USE FOR clause from azure-hosted-copilot-sdk description, eliminating the critical negative discriminator that prevented azure-prepare from capturing Copilot SDK deployment prompts. Changes: - azure-hosted-copilot-sdk/SKILL.md: Restore DO NOT USE FOR clause, add PREFER OVER azure-prepare directive, add Codebase Detection section with marker table (v1.0.1 -> v1.0.2) - azure-prepare/SKILL.md: Expand Step 0 to check codebase markers (not just prompt keywords) for Copilot SDK detection (v1.1.1 -> v1.1.2) - unit.test.ts: Add 2 regression guard tests requiring DO NOT USE FOR and PREFER OVER clauses to prevent silent removal - triggers.test.ts.snap: Updated snapshots for new description Fixes microsoft#1599 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update sensei/skill-authoring guidelines to allow DO NOT USE FOR for disambiguation Add exception to the blanket 'do not use DO NOT USE FOR' guidance: skills whose triggers overlap with broader skills (e.g., azure-prepare) MUST retain DO NOT USE FOR clauses for routing disambiguation. The previous guidance actively caused the microsoft#1599 regression — PR microsoft#1210 removed the clause while following sensei's recommendation, breaking routing for azure-hosted-copilot-sdk. Updated files: - sensei/SKILL.md: Add exception note to warning blocks - sensei/references/SCORING.md: Add disambiguation row to risk table, update pseudocode to skip warning for overlapping skills - skill-authoring frontmatter.md: Add exception for disambiguation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: implement recommendations #4 #5 microsoft#6 from RCA - #4: Audit conditional unit tests — added warning comments to azure-quotas and azure-hosted-copilot-sdk about the fragile if(description.includes('DO NOT USE FOR')) pattern - #5: Add PR template with skill routing verification checkbox - microsoft#6: Set continue-on-error: false in test-all-integration.yml so integration test failures actually fail the workflow Part of microsoft#1599 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address review feedback - version bumps, snapshot header, stale comments, genericize refs - Bump sensei version 1.0.1 -> 1.0.2 - Bump skill-authoring version 1.0.0 -> 1.0.1 - Update snapshot header to jestjs.io (CI Jest version compatibility) - Fix PR template: correct integration test command, remove issue link - Update stale workflow comments to match continue-on-error: false - Remove explicit issue number references from skill and test files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: update blanket DO NOT USE FOR prohibition to reflect disambiguation exception - skill-authoring Constraints: changed blanket prohibition to conditional guidance - sensei help banner: added '(unless disambiguation-critical)' qualifier - test-all-integration.yml: clarified continue-on-error comment wording Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump sensei version to 1.0.3 (main already at 1.0.2) Main branch already has sensei at version 1.0.2 from another merged PR. Our PR also modified sensei, so we need a higher version to pass the Skill Structure CI check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: trim description to 55 words, sync marker tables between skills Address @wbreza review feedback: - Issue 1: Trim azure-hosted-copilot-sdk description from 107 to 55 words (under 60-word guideline). Removed redundant/implementation-detail phrases while preserving all routing-critical triggers. - Issue 2: Sync codebase marker tables - added 2 missing markers to azure-prepare Check 2 (copilot-sdk in name/deps, createSession+sendAndWait) so both skills use identical detection logic. - Version bumps: azure-hosted-copilot-sdk 1.0.2->1.0.3, azure-prepare 1.1.2->1.1.3 - Updated unit test to match trimmed description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: snapshot header and azure-prepare version bump - Fix snapshot header to jestjs.io URL (CI requirement) - Bump azure-prepare 1.1.3->1.1.4 (main already at 1.1.3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 1, 2026 18:06

Jon Gallant (jongio) requested review from kvenkatrajan, Tom Meschter (tmeschter) and Wallace Breza (wbreza) as code owners April 1, 2026 18:06

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Comment thread plugin/skills/azure-hosted-copilot-sdk/SKILL.md Outdated

Comment thread plugin/skills/azure-prepare/SKILL.md

Copilot started reviewing on behalf of Jon Gallant (jongio) April 1, 2026 18:19 View session

Copilot AI review requested due to automatic review settings April 1, 2026 18:39

This was referenced Apr 1, 2026

Add skill-routing integration tests to PR CI pipeline #1645

Open

Add sensei rule to detect missing disambiguation for competing skills #1646

Closed

Copilot started reviewing on behalf of Jon Gallant (jongio) April 1, 2026 18:42 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Comment thread tests/azure-hosted-copilot-sdk/unit.test.ts

Comment thread .github/PULL_REQUEST_TEMPLATE.md Outdated

Comment thread .github/workflows/test-all-integration.yml

Jon Gallant (jongio) requested a review from Shayne Boyer (spboyer) April 1, 2026 19:32

kvenkatrajan reviewed Apr 1, 2026

View reviewed changes

Comment thread .github/skills/sensei/SKILL.md

Jon Gallant (jongio) mentioned this pull request Apr 1, 2026

Run integration tests against all supported models (multi-model matrix) #1647

Closed

Jon Gallant (jongio) requested review from Copilot and kvenkatrajan April 1, 2026 20:18

Copilot started reviewing on behalf of Jon Gallant (jongio) April 1, 2026 20:19 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Comment thread plugin/skills/azure-hosted-copilot-sdk/SKILL.md

Comment thread .github/skills/skill-authoring/SKILL.md

Comment thread .github/skills/sensei/SKILL.md

Comment thread .github/skills/sensei/references/SCORING.md

Comment thread .github/workflows/test-all-integration.yml Outdated

kvenkatrajan previously approved these changes Apr 1, 2026

View reviewed changes

Jon Gallant (jongio) dismissed kvenkatrajan’s stale review via 56616ed April 1, 2026 22:50

Copilot AI review requested due to automatic review settings April 1, 2026 23:07

Copilot started reviewing on behalf of Jon Gallant (jongio) April 1, 2026 23:08 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Comment thread plugin/skills/azure-prepare/SKILL.md

Comment thread plugin/skills/azure-hosted-copilot-sdk/SKILL.md

Jon Gallant (jongio) force-pushed the fix/azure-hosted-copilot-sdk-skill-routing-1599 branch from 48176e8 to de97ed7 Compare April 1, 2026 23:17

Wallace Breza (wbreza) reviewed Apr 2, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 2, 2026 17:14

Copilot started reviewing on behalf of Jon Gallant (jongio) April 2, 2026 17:15 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

Comment thread plugin/skills/azure-prepare/SKILL.md

Comment thread plugin/skills/azure-hosted-copilot-sdk/SKILL.md

Comment thread .github/skills/sensei/references/SCORING.md

Jon Gallant (jongio) and others added 7 commits April 2, 2026 10:24

fix: snapshot header and azure-prepare version bump

aaee68c

- Fix snapshot header to jestjs.io URL (CI requirement) - Bump azure-prepare 1.1.3->1.1.4 (main already at 1.1.3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jon Gallant (jongio) force-pushed the fix/azure-hosted-copilot-sdk-skill-routing-1599 branch from 5da5ed9 to aaee68c Compare April 2, 2026 17:26

Jon Gallant (jongio) requested a review from Wallace Breza (wbreza) April 2, 2026 17:48

kvenkatrajan approved these changes Apr 2, 2026

View reviewed changes

kvenkatrajan merged commit 70c03ac into microsoft:main Apr 2, 2026
12 checks passed

Jon Gallant (jongio) mentioned this pull request Apr 2, 2026

fix: add modify/feature routing to azure-hosted-copilot-sdk #1679

Merged

This was referenced Apr 2, 2026

[repo-status] Weekly Repo Status — Mar 27 – Apr 2, 2026 #1681

Closed

[repo-status] Weekly Repo Status — Mar 27 – Apr 3, 2026 #1703

Closed

Copilot AI mentioned this pull request Apr 8, 2026

Add Sensei/frontmatter guardrails for trigger-overlap disambiguation regressions #1778

Merged

Uh oh!

Conversation

Jon Gallant (jongio) commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause Analysis

The Regression (3 Layers)

Commit Trace

Engineering Process Gap Analysis

Why This Regression Shipped

Gap 1: Integration tests do not run on PRs

Gap 2: Unit test was a no-op for removed clauses

Gap 3: Copilot PR reviewer gave incorrect assessment

Gap 4: Sensei/skill-authoring guidelines actively DISCOURAGE DO NOT USE FOR

Gap 5: No automated rule validates disambiguation for overlapping skills

Gap 6: Waza evals run separately from PR CI

What the Team Could Have Done Differently

Changes Made

Skill Fixes

1. plugin/skills/azure-hosted-copilot-sdk/SKILL.md (v1.0.1 -> v1.0.2)

2. plugin/skills/azure-prepare/SKILL.md (v1.1.1 -> v1.1.2)

Test Fixes

3. tests/azure-hosted-copilot-sdk/unit.test.ts

4. tests/azure-hosted-copilot-sdk/__snapshots__/triggers.test.ts.snap

Guideline Fixes (Addresses Gap 4)

5. .github/skills/sensei/SKILL.md

6. .github/skills/sensei/references/SCORING.md

7. .github/skills/skill-authoring/references/guidelines/frontmatter.md

Process Improvements (Recommendations #4-6)

8. .github/PULL_REQUEST_TEMPLATE.md (NEW)

9. .github/workflows/test-all-integration.yml

10. tests/azure-quotas/unit.test.ts

Test Results

Recommendations to Prevent Future Regressions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Wallace Breza (wbreza) left a comment

Choose a reason for hiding this comment

Code Review Summary

Issue 1: Description exceeds recommended word count guideline

Issue 2: Inconsistent marker detection tables between skills

Positive Observations

Recommendation

Uh oh!

Jon Gallant (jongio) commented Apr 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Jon Gallant (jongio) commented Apr 1, 2026 •

edited

Loading

Gap 4: Sensei/skill-authoring guidelines actively DISCOURAGE `DO NOT USE FOR`

1. `plugin/skills/azure-hosted-copilot-sdk/SKILL.md` (v1.0.1 -> v1.0.2)

2. `plugin/skills/azure-prepare/SKILL.md` (v1.1.1 -> v1.1.2)

3. `tests/azure-hosted-copilot-sdk/unit.test.ts`

4. `tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap`

5. `.github/skills/sensei/SKILL.md`

6. `.github/skills/sensei/references/SCORING.md`

7. `.github/skills/skill-authoring/references/guidelines/frontmatter.md`

8. `.github/PULL_REQUEST_TEMPLATE.md` (NEW)

9. `.github/workflows/test-all-integration.yml`

10. `tests/azure-quotas/unit.test.ts`