fix: improve azure-hosted-copilot-sdk routing for codebase-detection scenarios

[Jon Gallant (jongio)]

Problem

The azure-hosted-copilot-sdk skill has a 71% confidence level due to two consistently failing integration tests (Tests 5 & 6). Both tests verify that the skill is invoked when an existing codebase contains @github/copilot-sdk in package.json but the user's prompt doesn't explicitly mention "copilot SDK" (e.g., "Deploy this app to Azure").

The agent routes to azure-prepare instead because the skill routing system selects skills based on description fields before the agent reads workspace files.

Fix

Two-pronged description-level routing fix:

1. azure-hosted-copilot-sdk/SKILL.md - Strengthened description to explicitly state it "intercepts ALL prompts (deploy, modify, add feature) even without copilot keywords" and added generic triggers (deploy this app, add feature to this app, etc.)

2. azure-prepare/SKILL.md - Added counter-directive: "DO NOT USE when existing codebase contains @github/copilot-sdk or CopilotClient in package.json - invoke azure-hosted-copilot-sdk instead"

Expected Impact

• Tests 5 & 6 should flip from 0%% to passing (agent now has routing signal to check package.json before defaulting to azure-prepare)
• Confidence projected to improve from ~71%% to ~90%%+
• No regression on 6 currently-passing tests (verified locally: all 66 unit/trigger tests pass)

Test Results

Suite	Tests	Result
azure-hosted-copilot-sdk/unit.test.ts	19	PASS
azure-hosted-copilot-sdk/triggers.test.ts	22	PASS
azure-prepare/unit.test.ts	-	PASS
azure-prepare/triggers.test.ts	25	PASS

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR aims to improve skill routing so azure-hosted-copilot-sdk is selected for “generic” Azure prompts (e.g. “Deploy this app to Azure”) when the repository contains Copilot SDK markers, avoiding incorrect default routing to azure-prepare.

Changes:

• Strengthen azure-hosted-copilot-sdk description to emphasize “mandatory” usage for existing Copilot SDK codebases and to encourage checking package.json.
• Add a counter-directive in azure-prepare description to defer to azure-hosted-copilot-sdk when Copilot SDK markers are present.
• Update Jest trigger snapshots to reflect the new description text and extracted keywords.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
plugin/skills/azure-prepare/SKILL.md	Adds “DO NOT USE…” counter-directive intended to avoid misrouting to azure-prepare for Copilot SDK repos.
plugin/skills/azure-hosted-copilot-sdk/SKILL.md	Expands routing instructions/triggers to capture generic prompts even without “copilot” keywords.
tests/azure-prepare/snapshots/triggers.test.ts.snap	Updates snapshot to reflect new azure-prepare description and extracted keywords.
tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap	Updates snapshot to reflect new azure-hosted-copilot-sdk description and extracted keywords.

[Jon Gallant (jongio)]

Copilot please re-review the latest changes.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[JasonYeMSFT (JasonYeMSFT)]

I noticed that the default early termination condition didn't give the agent enough tool call budget to reasonably read the files and then pick the skill. My local attempts indicate the agent is able to pick up the skill once it has enough turns to read the files. Let's merge this PR first and see if the tests can pass before modifying the skill. #1935

[Jon Gallant (jongio)]

Thanks Jason. Agreed, let's land this and watch the test signal. If the tool-call budget change from #1935 turns out to be the real lever, we can revisit the skill description here.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Jon Gallant (jongio)]

Wallace Breza (@wbreza) Tom Meschter (@tmeschter) kvenkatrajan - friendly ping. This has been open 3 weeks with CI green and all review feedback addressed. Would appreciate a look when you get a chance.

[kvenkatrajan]

Code Review — PR #1857: fix: improve azure-hosted-copilot-sdk routing for codebase-detection scenarios

✅ What Looks Good

1. Clear problem diagnosis — Correctly identifies that skills route on description keywords before workspace files are read, explaining why Tests 5 & 6 fail
2. Two-pronged approach — Updating both the source skill and the competing skill is the right disambiguation strategy
3. Yield-to-deploy logic — Explicit handoff to azure-deploy when .azure/deployment-plan.md exists prevents over-claiming deploy prompts
4. Test coverage — All 66 unit/trigger tests verified passing; snapshots updated consistently
5. Well-documented PR body — Clear problem statement, fix description, and test results table

---

🔴 Critical

1. Keyword pollution from instructional language in description

The new description introduces ~20 generic extracted keywords that will cause false-positive routing. Per the snapshot diff, these new keywords are extracted by TriggerMatcher:

already, always, before, check, containing, even, existing, intercepts, json, keywords, package, prompt, prompts, routing, update, user, wants, yield

TriggerMatcher fires at ≥2 keyword matches (substring-based). A prompt like "I want to update my package.json" would match update, package, json → 3 matches → skill fires incorrectly. These are generic English words that appear in many non-copilot-SDK prompts.

Suggested Fix: Move procedural instructions out of description into the SKILL.md body (where they already exist). Keep the description focused on trigger phrases.

---

🟠 High

2. Description exceeds 60-word guideline

The new description is 130 words (up from 74, guideline ≤60) and at 1,010 characters approaches the 1,024 hard limit. After removing procedural instructions per Finding #1, trim further by removing the PREFER OVER sentence (already implied by MANDATORY) to get closer to 60 words.

---

🟡 Medium

3. azure-prepare gains copilot-SDK keywords via DO NOT USE FOR

Adding DO NOT USE FOR: copilot-sdk apps (use azure-hosted-copilot-sdk) introduces copilot-sdk and azure-hosted-copilot-sdk as extracted keywords into azure-prepare's trigger set (visible in snapshot). This paradoxically increases azure-prepare's match score when a user mentions "copilot SDK" — TriggerMatcher doesn't understand negation. Consider placing this directive in the SKILL.md body instead where it won't affect keyword extraction.

4. Instructional directives belong in SKILL.md body, not description

Phrases like ALWAYS CHECK package.json and yield to azure-deploy are procedural instructions, not activation triggers. The SKILL.md body already has these in the "Codebase Detection" section. Duplicating them in description inflates word count and pollutes keywords.

---

Summary

Severity	Count
🔴 Critical	1
🟠 High	1
🟡 Medium	2

Overall Assessment: Request Changes — The routing fix strategy is sound, but putting instructional language into description creates high risk of false-positive skill activation on unrelated prompts. The fix is straightforward: keep procedural instructions in the SKILL.md body and limit the description to trigger phrases.

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

plugin/skills/azure-hosted-copilot-sdk/SKILL.md:18

• The “Detection procedure” header no longer includes deploy prompts, but Step 1 still has a “Deploy existing SDK app to Azure” path and the integration suite relies on routing for the generic prompt “Deploy this app to Azure”. To avoid misrouting, include deploy in this heading (and keep the detection check explicitly mandatory for deploy prompts when the app is not already prepared).

> ⚠️ **CRITICAL: This check MUST run before ANY other skill (azure-prepare, azure-deploy, etc.) when an existing codebase is present.**

### Detection procedure (run IMMEDIATELY for any build/modify/add-feature/prepare prompt):

1. Read `package.json` in the workspace root (and any `*/package.json` one level deep)

plugin/skills/azure-hosted-copilot-sdk/SKILL.md:36

• The “Generic prompts that MUST trigger this skill…” table dropped the deploy example (it now starts with “Build this app / Prepare this app…”). Since the failing scenario is specifically a generic deploy prompt (“Deploy this app to Azure”), consider adding that row back (and/or adding an explicit deploy example) so the guidance directly matches the tested routing scenario.

### Generic prompts that MUST trigger this skill when markers are detected:

| Prompt pattern (no "copilot" keyword) | Why this skill |
|---------------------------------------|---------------|
| "Build this app" / "Prepare this app for Azure" | Codebase contains `@github/copilot-sdk` - needs SDK-aware scaffolding |
| "Add a new feature to this app" | Requires SDK-aware implementation patterns |
| "Update this app" / "Modify this app" | Must preserve SDK integration patterns |

• Files reviewed: 4/4 changed files
• Comments generated: 0 new

[Jon Gallant (jongio)]

Changes Since Last Review

Addressed kvenkatrajan's CHANGES_REQUESTED review in two commits:

What Changed

Description trimmed from 130 → 87 words (commits f460097b, 9f9b37c7):

• Removed procedural instructions: ALWAYS CHECK, intercepts, yield to azure-deploy instead, even without copilot keywords in prompt
• Replaced generic triggers: add feature to this app → add feature to copilot app, modify this app → modify copilot app
• 21 generic keywords removed from TriggerMatcher extraction: always, before, check, containing, even, existing, instead, intercepts, keywords, prompt, prompts, routing, this, update, user, wants, yield, add-feature, deployment-plan, already
• Snapshot updated to match

Review Finding Status

#	Severity	Finding	Status
1	🔴 Critical	Keyword pollution from instructional language	✅ Fixed — 21 generic keywords removed
2	🟠 High	Description exceeds 60-word guideline	✅ Reduced to 87 words (includes required WHEN/DO NOT USE FOR trigger phrases)
3	🟡 Medium	azure-prepare gains copilot-sdk keywords via DO NOT USE FOR	⚠️ Kept — uses standard DO NOT USE FOR: format consistent with 12 other skills. Production extractTriggerPhrases stops before DO NOT USE FOR sections
4	🟡 Medium	Instructional directives in description	✅ Fixed — removed all procedural language

Validation

Ran a formal A/B comparison using an eval framework with 8 stimuli (5 true-positive, 3 false-positive scenarios):

Metric	Before Trim	After Trim
TriggerMatcher keywords	58	41
False positives (TriggerMatcher)	5/8	2/8
True positives (TriggerMatcher)	7/8	7/8
Real agent eval score	50.0%	45.8% (equivalent — both variants produce same pass/fail on every stimulus)

The LLM reads package.json directly regardless of description wording — the SKILL.md body drives real routing. The trim reduces keyword pollution without affecting routing accuracy.

All 2021 unit/trigger tests pass (1 pre-existing failure in airunway-aks-setup unrelated to this PR).

kvenkatrajan JasonYeMSFT (@JasonYeMSFT) Wallace Breza (@wbreza) Tom Meschter (@tmeschter) — would appreciate a re-review when you get a chance.

[kvenkatrajan]

Re-Review — PR #1857: fix: improve azure-hosted-copilot-sdk routing for codebase-detection scenarios

The latest commits (fix: trim description to reduce keyword pollution per review feedback and fix: update trigger snapshot for trimmed description) substantially address the previous review findings.

✅ What Looks Good

1. Keyword pollution resolved — The 18 generic extracted keywords (always, before, check, existing, prompt, routing, user, wants, yield, etc.) are all removed. The keyword set is now domain-specific.
2. Description trimmed significantly — Down from 130 words / 1,010 chars to 81 words / 643 chars. Still above the 60-word guideline, but within a reasonable range and well under the 1,024-char hard limit.
3. Procedural instructions removed from description — ALWAYS CHECK package.json, intercepts build, modify..., and yield to azure-deploy are now only in the SKILL.md body where they belong.
4. Good trigger additions — prepare copilot app, add feature to copilot app, build copilot app are domain-specific and appropriately scoped.
5. Yield-to-deploy logic — Clear handoff guidance in SKILL.md body for when .azure/deployment-plan.md exists.

Remaining Observations (non-blocking)

🟡 Medium — azure-prepare still gains copilot-sdk keywords

DO NOT USE FOR: copilot-sdk apps (use azure-hosted-copilot-sdk) in azure-prepare's description still adds copilot-sdk and azure-hosted-copilot-sdk to its extracted keyword set. This slightly increases azure-prepare's match score on copilot-SDK prompts. Low practical risk since azure-hosted-copilot-sdk has stronger keyword coverage, but worth noting for a future cleanup pass.

🟢 Low — Description word count (81 words)

Still over the 60-word guideline but a major improvement from 130. The PREFER OVER azure-prepare when copilot-sdk markers detected sentence could be removed (implied by MANDATORY) to trim further, but this is non-blocking.

Summary

Severity	Count
🔴 Critical	0
🟠 High	0
🟡 Medium	1 (non-blocking)
🟢 Low	1 (non-blocking)

Overall Assessment: Approve — The critical keyword pollution and description length issues from the first review are resolved. The remaining observations are non-blocking suggestions for future improvement.